Dance Controller: Your Body Is the Joystick

How Pose Detection Works

Pose detection AI looks at video from your webcam and identifies where your body parts are: your head, shoulders, elbows, wrists, hips, knees, and ankles. It maps these as a set of points called keypoints, and then it classifies the overall shape of those points as a named pose.

You have probably seen pose detection in action without realizing it. Every time a dance game follows your movements, or a fitness app counts your push-ups, or a video filter tracks your face — that is pose detection at work. In 2026, it runs fast enough to work in real time on a regular laptop webcam with no special hardware.

Today you are going to train a pose model with your own custom poses and then hook it up to a Scratch game so that your body movements control a character on screen.

Train Your Poses

Go to teachablemachine.withgoogle.com, click Get Started, and choose Pose Project. Think about what two or three body positions could control a game:

• Arms Left — lean or stretch your arms to the left → character moves left
• Arms Right — lean or stretch your arms to the right → character moves right
• Jump Pose (optional) — raise both arms above your head → character jumps

For each pose class, enable your webcam and click Hold to Record. Hold the pose and let Teachable Machine take 30 to 50 photos. Move around slightly — step a little left or right — so the AI learns the general shape, not just one exact frozen position.

Tip: Add a Neutral class too. Record photos of yourself standing still with arms at your sides. This tells the AI what your resting position looks like, so it does not trigger game actions when you are just sitting there watching.

Click Train Model. Test each pose and check the confidence bars to make sure each one is recognized reliably.

Connect to Scratch

Once your pose model is trained, you need to connect it to Scratch. Here is how:

• In Teachable Machine, click Export Model. Choose Upload (shareable link) and copy the URL it gives you.
• Open scratch.mit.edu and start a new project. You can use Scratch without creating an account.
• Click the Extensions button (the puzzle piece icon in the bottom left corner) and look for the Teachable Machine extension.
• Paste your model URL into the extension setup. Now your Scratch project can read your pose predictions in real time.

In your Scratch project, use blocks like: if [pose = Arms Left] then [move sprite -10 steps]. The AI continuously reads your webcam and updates the pose prediction, and your Scratch code reacts to each change.

Play and Improve

Test your game. Does your character move when you move? Try adjusting the sensitivity. If the character moves when you did not mean it to, add more Neutral pose photos and retrain. If it misses your movements, add more examples of each action pose.

Here are some game ideas to build around your pose controller:

• Dodge falling objects by leaning left or right
• Collect coins by jumping with arms raised
• Guide a maze character through a level using body lean

Once it works, show someone. Body-controlled games are always more impressive than keyboard-controlled ones — and you built the AI part yourself from scratch.