Lesson reading
live
1 hr
Lesson 26 of 281 hour
Task-and-Motion Sequencing
Start with the lesson question, connect the representations, and test the model with evidence.
task motion couplingpayload geometrycollision checkingexecution evidencescene freshness
Learning objectives
- Explain grasping, reachability, collision checking, and contact.
- Sequence perception and motion for a pick-and-place task.
- Design clear intent signals and human override points.
Lesson flowHook, model, explanationShow guidanceHide guidance
Inspect the opening phenomenon
Predict what changes, then name the evidence.
Apply in the lab
Name the evidence before reading the answer.
Read only what helps
Then use the lab and recall check.
More when needed
Transcript and resources stay available below.
Course progress
AI & Robotics Foundations · Manipulation and Human–Robot Interaction · Lesson 26
Task-and-Motion Sequencing
In progress
Decision challenge
Observe the phenomenon. Then connect the representations.
Use the opening example to make a prediction, identify evidence, and explain which model supports it.
Why Can a Robot Reach but Fail? Task-and-Motion Planning
Why Can a Robot Reach but Fail? Task-and-Motion Planning
Why Can a Robot Reach but Fail? Task-and-Motion Planning
Reference drawerTranscript, source notes, scripts, and package status stay tucked away until you need them.7 files
Video script
approved
Transcript fallback
available
courses/ai-robotics/modules/09-manipulation-and-human-robot-interaction/lessons/02-task-and-motion-sequencing/video-transcript.md
Make the Whole Sequence Work
approved
30 min
Mastery check
live
7 questions / 8 min
Book section:courses/ai-robotics/modules/09-manipulation-and-human-robot-interaction/lessons/02-task-and-motion-sequencing/book-section.md
Transcript for accessibility and fallback
Descriptive transcript: Task-and-Motion Sequencing Complete narration and original visual descriptions. Untimed non-video learning path. 00:00 — It fits. Until it picks. Visual: Top-down overhead gantry moves empty across opening, then a separate planned loaded path stops at first model collision. Do not depict crashing hardware. Narration: This robot can reach both sides of the opening. Its empty gripper fits. But pick up the box, and the same route fails. The goal is reachable. The sequence is not. 00:11 — What next? How exactly? Visual: Pick/carry/place nodes pulse in order; geometric failure at carry sends feedback to earlier choice. Label candidate route, not execution. Narration: Task planning chooses what happens next: pick, carry, then place. Motion planning checks how each action can happen. They must talk to each other. A grasp that works here can make the next movement impossible. 00:27 — The robot got wider. Visual: Loaded box beside the opening; dimension brackets show 60 + 5 + 5 = 70 cm versus 40 cm. The top-down rectangle is the collision geometry. Narration: In our simplified top-down model, the opening is forty centimeters wide. The box presents sixty centimeters across it. Add five centimeters of clearance on each side: seventy centimeters will not fit. Checking just the gripper misses the problem. 00:44 — Turn before crossing. Visual: The box rotates in the left bay, moves through the opening, then rotates in the right bay. Each stage highlights during motion. The 30 cm demand is compared with the 40 cm opening only after the first turn. Narration: Try a different sequence. Rotate in the open bay first. The box now presents twenty centimeters across the opening. With the same margins, it needs thirty. Plan the rotation, the loaded crossing, and the final placement together. In this model, that candidate clears the sampled checks. 01:04 — Endpoints can fool you. Visual: At the opening's center, ghost poses at 0 and 180 degrees both fit horizontally. The attempted rotation stops at the first colliding sample. A separate translucent 90-degree pose is labeled blocked. Narration: Do not postpone the turn until the narrow opening. These two end orientations both fit, but the box sweeps into the wall halfway through. Collision checking has to cover the motion, not just the start and finish. Sampling still leaves gaps between samples. 01:23 — Did it actually grasp? Visual: An event timeline connects close, result plus object evidence, attached geometry, and carry. Missing evidence suppresses carry. A scene-version change prevents dispatch of the old plan. This is the lesson's illustrative model, not the ROS action-state machine. Narration: Now execution. Sending close is not proof that the box is held. Wait for the action result and evidence of the object. Only then add the carried geometry and permit the transfer. If the scene changes before that transfer, stop dispatch and check the plan again. 01:42 — Which plan survives? Visual: The transfer problem shows 35 + 5 + 5 = ? against a 40 cm opening. A separate accepted-command token is distinguished from object-held evidence. There is a three-second silent pause before the answer. Narration: Your turn. A wider box needs thirty-five centimeters across the opening, before margins. The opening is still forty. Would our five-centimeter margin on each side pass? And would an accepted grasp command let the robot carry immediately? Pause and explain both decisions. 02:07 — A sequence needs evidence. Visual: The 45 cm demand exceeds the 40 cm opening, so the candidate is rejected. Accepted is not held. A moving marker connects task choices, motion checks and evidence in a feedback loop. The closing invitation points to the course simulation. Narration: No to both. Thirty-five plus ten is forty-five, so that clearance test fails. Accepted is not completed, and completed is not automatically proof of a secure grasp. Choose compatible actions, check their whole motions, verify the results, and revise when evidence changes. Try the simulation in our AI Robotics course. A three-second silent retrieval pause precedes the answer. Pause longer as needed. This is a simplified simulation, not hardware guidance.
Reading lab
Core explanation
Connect the lesson's words, diagrams, graphs, evidence, and equations.
A robot can reach the pickup and the destination, yet fail between them. Why? Picking an object changes what the robot must carry through the world. A task is more than a list of reachable endpoints.
Learn to
- Separate an action's preconditions, intended effects and observed results.
- Explain how a later motion can invalidate an earlier task choice.
- Check carried geometry and intermediate configurations, not only endpoints.
- Use evidence and scene freshness to decide whether the next action may start.
Prerequisites: lesson 9.1's reach, clearance and contact checks; lesson 8.3's failure and timeout reasoning. Core work needs no robot, paid software or video.
Before / During / After the video
Before: An empty gripper fits through a 40 cm opening. It picks up a 60 cm by 20 cm box. Predict whether the original path still works. Which orientation and geometry did you assume?
During: At 00:27 compare the payload with the gap. At 00:44 explain why rotation happens in the bay. At 01:05 predict the intermediate rotated shape. At 01:23 distinguish a command from evidence. At 01:43 attempt both questions before the answer around 02:07.
After: Explain why changing a grasp or inserting an action can be necessary even if the destination is reachable. Give one geometric failure and one evidence failure. Then test your explanation in the lab.
Watch the full original Humanoid Hub tutorial. The tutorial and descriptive transcript teach the same content. Without video, read the eight scene descriptions in order and stop before the answer. Write, sketch or speak your prediction; the pause is not a timed assessment.
Two questions that constrain each other
Task reasoning asks what action and in what order? Motion reasoning asks which feasible movement realizes it? The answer to the second can force a revision of the first. MIT's TAMP research overview describes this dependence between task choices and low-level feasibility.
Our original example uses an overhead gantry in a top-down plane. The rail is above the modeled walls; the diagram is not an articulated robot arm. A centered gripper carries a rectangle, and a wrist can turn it in the plane. No lifting dynamics, grasp forces or rail-height collision is simulated. This isolates one sequencing failure; lesson 9.1's contact requirements still apply.
Worked example: the payload changes the route
Invented model dimensions, not recommended hardware settings:
| Quantity | Value |
|---|---|
| Workspace | 2.8 m by 2.0 m |
| Opening between two walls | 0.40 m |
| Box | 0.60 m by 0.20 m |
| Empty tool envelope | 0.10 m square |
| Illustrative clearance margin | 0.05 m each side |
| Start / destination center | (0.6, 1.0) / (2.2, 1.0) m |
The box starts with its long side across the opening. Required vertical span plus margins is 0.60 + 2 * 0.05 = 0.70 m, exceeding 0.40 m. Empty-tool success cannot establish loaded-route success.
Try turn in left bay → cross → turn in right bay. During crossing, the short side faces across the opening: 0.20 + 2 * 0.05 = 0.30 m. That fits the width test. It does not by itself prove that the turns or approach fit. The supplied rectangle-collision model checks those connected motions at 201 poses per segment. This particular candidate passes those samples.
The model tests exactly two candidates, not every possible strategy. A failure to find either is not proof that no physical solution exists. A full task-and-motion planner may explore different grasps, placements, intermediate supports or action orders. MoveIt Task Constructor's stage concepts illustrate how subproblem solutions must connect into a complete plan; our small independent program is not an MTC implementation.
Reasonableness check: In a 0.80 m opening the original 0.70 m width demand can fit. In a 0.25 m opening even the 0.30 m demand fails. Test both; do not memorize “always turn.”
Endpoints are not the swept shape
Place the horizontal box at the opening's center. Orientations 0 and 180 degrees have the same narrow vertical span. Yet the intermediate 90-degree orientation presents 0.60 m vertically and collides.
For angle a measured from horizontal, vertical span is abs(0.60 * sin(a)) + abs(0.20 * cos(a)). At 0 degrees it is 0.20 m; at 90 it is 0.60 m. The full model additionally checks the finite wall thickness and workspace boundary, using the separating-axis test for rectangles.
Walls are padded by 0.05 m along each coordinate axis. This is a conservative square-corner margin, not an exact rounded Euclidean offset. Touching a padded boundary is rejected. Sampled clearance is evidence about those checked poses, not a continuous certificate. Higher sample density can discover a missed collision; it does not model unmeasured objects, forces or real hardware uncertainty.
Preconditions are promises to check
Use three distinct columns when designing an action:
| Action | Must be true before starting | Evidence required afterward |
|---|---|---|
| Grasp | Current object pose, compatible plan and empty tool | Successful result plus evidence the object is held |
| Carry | Held object, correct attached geometry and current scene/plan | Transfer result and observed destination state |
| Place | Supported placement plan and appropriate contact | Support confirmed and object released |
A planning effect such as “holding = true” predicts the result of a successful action. It is not a sensor reading. The example's grasp-result requires both succeeded and objectDetected. A false value suppresses carry. In practice, the evidence must suit the mechanism and task; a binary flag here is only a teaching abstraction.
MoveIt's pick-and-place tutorial explicitly updates attached-object geometry. Such a model update changes planning; it does not physically secure an object. ROS 2's action design distinguishes accepted, executing and terminal results. Acceptance alone is not successful completion.
A plan is tied to assumptions
Suppose the scene version changes after planning but before carry dispatch. The program refuses to send the transfer and enters replan. It keeps its belief that it holds the object; it does not pretend the object vanished.
This teaching model does not execute a loaded recovery from an arbitrary pose: a subsequent attempt stops and requests a separately checked recovery. If a change occurs during real motion, cancellation, stopping distance, load retention and human oversight need a designed controller. “Stop dispatch” means do not send a new motion; it is not an instant physical emergency stop.
Do not replan forever. Use a bounded recovery policy, suitable support or supervised assistance. Never derive physical-robot commands from this tutorial.
Transfer check
A different payload presents 0.35 m across a 0.40 m opening. Keep the invented margin of 0.05 m per side. Is it eligible for this crossing? May a grasp accepted message authorize carry?
Attempt both before checking: 0.35 + 0.10 = 0.45 m, so the margin test rejects the crossing. Acceptance does not prove completion or holding. Neither faster motion nor renaming an action fixes these missing conditions.
Complete the companion lab, then the seven-question quiz. Submit an original counterexample with a prediction, model output, explanation and a limitation.
Takeaway: Choose compatible actions, check the motions and geometry they create, then verify the results before advancing.
Practice labMake the Whole Sequence WorkOpen this when you are ready to apply the model, collect evidence, and check your explanation.30 min
Lab: Make the whole sequence work
Status: Approved lesson 9.2. Time: 25–35 minutes. Required equipment: browser or paper. Optional: Node.js 22+ for the supplied JavaScript model. No hardware, ROS installation, account or paid service is needed.
Objective
Produce evidence that separates geometric feasibility from execution readiness. This is a simulation-only overhead-gantry model. Do not connect it to actuators or use its dimensions/margins as hardware settings.
Materials
Paper, a ruler and a calculator are sufficient. Optional: Node.js 22+ and the standalone model download. No robot, account or paid service is required.
Steps
Copy a results table with columns: case, predicted result, observed result, first failed condition, justified next action, limitation.
- Empty tool travels straight through a 0.40 m opening.
- Held box travels broadside through the same opening.
- Turn in the left bay, cross, then turn at destination.
- Rotate 180 degrees while centered inside the opening.
- Repeat case 3 with gap 0.25 m and 0.80 m.
- Grasp succeeds but object detection is false.
- Scene changes after grasp confirmation but before transfer dispatch.
Run the model
Download model.mjs, or use the equivalent paper activity below. From the folder containing it:
// Save as your-own-trace.mjs beside model.mjs; run: node your-own-trace.mjs
import {routes, pathCheck, choosePlan, initial, event} from './model.mjs';
console.log(pathCheck(routes.empty, {held:false}));
console.log(pathCheck(routes.broadside));
console.log(pathCheck(routes.turnFirst));
console.log(pathCheck(routes.turnInGate));
console.log(choosePlan(0.25), choosePlan(0.80));
let s = event(event(initial(), 'observe'), 'plan');
s = event(s, 'grasp-result', {succeeded:true, objectDetected:false});
console.log(s.stage, s.trace);
The renderer and lab use the same model. Scene coordinates are metres; angles are radians. The broadside box is 0.60 m across the opening; turning makes its span 0.20 m during crossing. Padding is 0.05 m per side. The top-down rail is outside the collision plane.
Expected Result
After your attempt: 1 passes; 2 fails; 3 passes sampled geometry; 4 fails during rotation despite clear endpoints; narrow-gap plan search finds neither candidate, wide-gap search chooses broadside; 6 stops without carry; 7 withholds dispatch and enters replan while retaining the held-object state.
To run case 7, change detection to true, then apply event(s,'scene-change') followed by event(s,'carry-result',{succeeded:true}). The latter event is a combined dispatch guard/completion abstraction: stale plans are rejected before simulated movement. It is not a real asynchronous controller. The model deliberately stops rather than improvising loaded recovery.
Paper alternative: same outcomes
Draw a 28 by 20 cm workspace. Use 1 cm = 0.10 m. Draw a wall strip from x=13 to 15 cm, leaving an opening between y=8 and 12 cm. Pad the wall edges by 0.5 cm. Cut a 6 by 2 cm rectangle and a 1 cm-square empty tool.
Start the box center at (6,10) cm with the long axis vertical; goal (22,10) cm. Test the four paths slowly, including intermediate turns. Treat contact with a padded edge as failure. Your paper trace replaces the software output, but it is still approximate.
For feedback, make cards: observe, plan, grasp, carry, place, done, stop, replan. Advance only when the evidence listed in the lesson is present. An “accepted” card alone cannot advance grasp to carry. Change the scene-version card between grasp and carry and explain why dispatch is withheld. Keep the object card with the gripper during recovery.
Reflection Questions
- Explain the numerical demands 0.70 m and 0.30 m, with units.
- Find a pair of clear endpoints with a blocked middle configuration.
- Distinguish “none of two candidates passed” from “no solution exists.”
- Explain why the attached-object flag is not proof of grasp success.
- Submit your completed seven-case table and one new test with a prediction.
- State two omitted real-robot requirements; at least one must be non-geometric.
Rubric: 2 points correct geometry, 2 correct sequencing/evidence, 2 reproducible trace, 2 counterexample explanation, 2 explicit limitations. Correct an error after feedback; fast execution is not a mastery measure.
Troubleshooting
If Node cannot import, keep the two files together and use the .mjs extension. A browser download opening as text is normal; save the file locally or use the paper method. If a changed gap rejects both routes, do not force success: record the candidate set and revise the problem. A sampled pass does not imply continuous clearance or safe physical operation.
Extension Challenge
Choose a new opening width, predict which of the two candidates passes, and record its first failed condition if any. Explain why a sampled geometric pass still needs execution evidence and does not certify a physical robot.