EEduQuest
AI & Robotics Foundations
PreviousNext
Lesson 17 of 221 hour

Detection, Segmentation, Pose, and Tracking

Start with the lesson question, connect the representations, and test the model with evidence.

classificationdetectionsegmentationposetracking

Learning objectives

  • Explain images, depth, filtering, and visual features as robot data.
  • Compare classification, detection, segmentation, pose, and tracking.
  • Evaluate a perception pipeline using errors and changing conditions.
Lesson flowHook, model, explanationShow guidance

Inspect the opening phenomenon

Predict what changes, then name the evidence.

Apply in the lab

Name the evidence before reading the answer.

Read only what helps

Then use the lab and recall check.

More when needed

Transcript and resources stay available below.

Course progress

AI & Robotics Foundations · Robot Perception and Computer Vision · Lesson 17

Detection, Segmentation, Pose, and Tracking

In progress

Decision challenge

Observe the phenomenon. Then connect the representations.

Use the opening example to make a prediction, identify evidence, and explain which model supports it.

What Should Robot Vision Return? Detection to Tracking

Choose the minimum sufficient perception output and verify geometry and time.

Choose the minimum sufficient perception output and verify geometry and time.

Reference drawerTranscript, source notes, scripts, and package status stay tucked away until you need them.6 files

Lesson reading

live

1 hr

Video script

draft

Transcript fallback

available

courses/ai-robotics/modules/06-robot-perception-and-computer-vision/lessons/02-detection-segmentation-pose-and-tracking/video-transcript.md

Choose and Audit a Perception Output

draft

30 min

Mastery check

live

6 questions / 10 min

Transcript for accessibility and fallback

# Transcript A robot sees a cup. What should vision return? Classification answers what, but not where. Detection adds a bounding box: rough location, not an exact boundary. Segmentation labels pixels, useful for object shape or drivable floor. Pose estimates rotation and translation in named frames when geometry and correspondences support it. Tracking associates a target across time, but IDs can switch after occlusion. For a grasp, a class score alone is not enough. Verify depth, pose error, timestamp age, and track continuity. Choose the minimum sufficient output for the action, then define what evidence must stop it. Which output labels every pixel, and which preserves identity over time? Continue the lab on EduQuest.

Reading lab

Core explanation

Connect the lesson's words, diagrams, graphs, evidence, and equations.

One image, five different questions

A warehouse robot sees a cup. “Cup” alone may be enough to count inventory, but not to grasp it. Robot perception must return the type of evidence the next component actually needs.

Five computer-vision output contracts

The output contract

TaskOutputAnswersDoes not guarantee
Classificationclass scoresWhat is prominent?location
Detectionclass + bounding boxWhat and roughly where?exact boundary or depth
Segmentationpixel labels/masksWhich pixels belong where?3D orientation
Pose estimationrotation + translation or keypointsHow is it positioned?persistent identity
Trackingidentity + state over timeIs this the same target now?correct detection forever

These are contracts, not a ladder where the “largest” output is always best. Choose the smallest output that safely supports the action.

Worked example: a pick task

Goal: pick the blue cup, not the blue tray.

  1. A classifier says cup: 0.83, but gives no location.
  2. A detector returns box (220, 140, 180, 240) for a cup.
  3. A segmentation mask separates the cup silhouette from the tray.
  4. Calibrated correspondences and solvePnP can estimate rotation and translation when the object geometry is known.
  5. A tracker associates the target across frames while the arm approaches.

Before grasping, verify depth, pose reprojection error, fresh timestamps, and whether the track survived occlusion. A confidence score is model output—not a probability that the action is safe.

Retrieval pause: Which output do you need to paint only drivable-floor pixels? Segmentation.

Tracking is a time problem

A detector can produce independent boxes in each frame. A tracker predicts a target state, associates new measurements, updates the state, and records missed observations. Identity switches can occur when targets cross or disappear. Use track age, missed-frame count, motion consistency, and re-detection evidence—not only the displayed ID.

Misconception checks

  • A tight box is not an exact object boundary.
  • A mask is not automatically metric 3D geometry.
  • Pose always belongs to named object and camera frames.
  • A stable track ID can still be attached to the wrong object.

Decision rule

State the downstream question first. Then choose the output, define measurable acceptance checks, test changing light/viewpoint/occlusion, and specify a safe response to stale or conflicting evidence.

Summary

Classification, detection, segmentation, pose, and tracking answer different questions. Reliable robotics preserves those distinctions and tests spatial, geometric, and temporal evidence before acting.

Further learning

Practice labChoose and Audit a Perception OutputOpen this when you are ready to apply the model, collect evidence, and check your explanation.30 min

Lab: Choose and audit a perception output

Objective

Select the minimum sufficient perception output and identify evidence that must block action.

Materials

  • lab-worksheet.csv
  • Spreadsheet, text editor, or paper; no hardware required

Steps

  1. For each mission, write the downstream question.
  2. Choose classification, detection, segmentation, pose, or tracking.
  3. Name the output fields the controller receives.
  4. Add one spatial or geometric acceptance check.
  5. Add one temporal check such as timestamp age, track age, or missed frames.
  6. Specify a safe response when the check fails.

Expected Result

Counting can use classification or detection; grasping normally needs localization plus depth/pose; drivable-area reasoning needs segmentation; following a person needs tracking with re-identification and stale-data rules.

Reflection Questions

  1. When is a bounding box insufficient?
  2. Why is a track ID not proof of identity?
  3. What is the minimum sufficient output for your mission?

Extension Challenge

Add an occlusion scenario and define the maximum number of missed frames before the robot pauses.

Accessibility fallback

All scenarios are textual. Describe masks as included/excluded pixel sets and pose as named-frame rotation plus translation.