Build-a-Tower Robot Arm: A robot arm project Claude Code checks with a webcam
Teach the arm three spots. Claude reads the photo, picks the order, and checks the tower after every move.
A curious adult, or a parent and kid.
The plan is up. The kit is coming.
We do not sell the parts for this one yet.
You need: a robot arm that remembers poses you teach it (we have not found one to sell yet); a Mac with Claude Code; Homebrew, the free Mac tool installer (Claude can walk you through installing it); a USB webcam on a fixed overhead mount; a steady desk and good light.
Last updated
The plan in 3 steps (planned)
This is how we expect setup to go. We have not run it yet.
- Step 1
Teach the arm its spots.
- Step 2
Take a photo with your webcam.
- Step 3
Ask Claude which block is where, then build the tower.
What you need
You need
- A robot arm that remembers poses you teach it (we have not found one to sell yet)
- A Mac with Claude Code
- Homebrew, the free Mac tool installer (Claude can walk you through installing it)
- A USB webcam on a fixed overhead mount
- A steady desk and good light
Related products
- ESP32-CAM camera board: For the robot arm plan we recommend a USB webcam instead of this board. ESP32-CAM boards can reset when power dips, and flashing one means moving a wire.
Your first evening
We test this before we promise it
The first evening, as we have written it
We have not run these steps ourselves yet, so read them as a preview. We change anything that does not work before we sell the kit.
Put the arm together
Assemble the arm. If your model has the camera, try its built-in color sorting first; Hiwonder lists color recognition as a feature of that camera.
Fix everything in place
Print the mat. Tape the mat and the arm to the desk so neither moves.
Teach the poses
Teach and save the poses: above A, grab A, lift, the same for B and C, then place level 1, 2 and 3.
Mount the webcam
Fix the webcam overhead, looking straight down at the mat.
Take the first photo
Paste the first prompt below into Claude Code. It installs imagesnap and takes desk.jpg.
Ask Claude what it sees
Paste the second prompt. Claude says which circle holds each color.
Prompts to paste into Claude
Copy one, paste it, and fill in anything in square brackets. Claude takes it one step at a time.
The first two prompts need a webcam, Claude Code and Homebrew. Claude can walk you through installing Homebrew, and macOS asks you to allow camera access the first time. The arm prompts wait for our own build.
1. Take a photo
Paste into Claude Code
Install imagesnap and take a photo from my webcam called desk.jpg. Show me the photo.
2. Which block is where
Paste into Claude Code
Look at desk.jpg. Tell me which circle (A, B, C) holds each color. Answer in JSON.
3. Write the arm commands
Paste into Claude Code
Write a small arm.sh with the commands pick A/B/C and place 1/2/3, using my saved poses. Don't add any other moves.
4. Build the tower
Paste into Claude Code
Build a tower: yellow on the bottom, then blue, then red. Use only the commands in arm.sh. After each placement, take a photo and confirm before the next step.
5. It fell over
Paste into Claude Code
The tower fell. Look at the photo, say what went wrong, and try once more.
When you have a weekend
We test this before we promise it
Ideas we are still testing
These are written from our research. We run each one before we promise it.
Give Claude arm commands
Paste the third prompt: Claude writes arm.sh with pick A, B or C and place 1, 2 or 3, using your saved poses. Whether the arm can be driven from a Mac script is not verified yet.
Build the tower
Paste the fourth prompt and watch a three-block tower go up, with a photo check after each move.
Knock it over
Knock the tower over and paste the fifth prompt.
What Claude really does with a robot arm
Claude does not look through the camera and steer the arm. The arm repeats moves you teach it. Claude looks at a photo, decides the order and checks the result. That split is what makes it work.
What you teach the arm
- Where each pickup circle is: above A, grab A, lift, and the same for B and C.
- Where the tower goes, one pose per level: place level 1, 2 and 3.
- The arm saves each pose and repeats it exactly.
What Claude does
- Looks at a photo and says which circle holds which color.
- Picks the stacking order.
- Runs the small pick and place commands it wrote for your saved poses.
- Takes a new photo after each move, checks the tower is standing, and tries again if not.
Why Claude does not steer by sight
Anthropic's own vision docs say Claude's positions in an image are approximate and warn against tasks that need perfect precision. Turning a spot in a photo into an arm angle also needs camera-to-arm calibration, which a first project should not need.
Tape everything down
The taught poses only work if nothing moves. Tape the printed mat and the arm to the desk. If either one shifts, teach the poses again.
What each photo costs
Every photo Claude looks at uses your Claude usage. A 1000 by 1000 pixel photo is about 1,296 tokens, so a long session with many checks uses more.
If you want real AI training next
Teaching an arm a task without saved poses is called imitation learning. The LeRobot project recommends at least 50 recorded demonstrations and several hours of training. That is a big step up from this plan, and a good one once the tower works.
What it can't do
- No pick up whatever you see. It only uses the spots you taught.
- Move the mat or the arm and you teach the poses again.
- It needs a steady desk and good light.
- Every photo uses Claude usage. A 1000 by 1000 pixel photo is about 1,296 tokens.
- Learning a task without hand teaching (LeRobot) needs 50 or more demonstrations and hours of training. That is not in this plan.
Questions
Before we sell this kit
The plan is yours today. The buy button stays off until each of these is done.
- Find a source for the arm at a fair price.
- Test driving the arm from a small Mac script.
- Build it ourselves and film one uncut run, retries included.
Guides that go with this plan
New to this? Start here, or see every kit plan.