EEduQuest
AI & Robotics Foundations
PreviousNext
Lesson 16 of 2255 minutes

Images, Depth, and Features

Start with the lesson question, connect the representations, and test the model with evidence.

imagedepthcamera intrinsicsvisual features

Learning objectives

  • Explain images, depth, filtering, and visual features as robot data.
  • Compare classification, detection, segmentation, pose, and tracking.
  • Evaluate a perception pipeline using errors and changing conditions.
Lesson flowHook, model, explanationShow guidance

Inspect the opening phenomenon

Predict what changes, then name the evidence.

Apply in the lab

Name the evidence before reading the answer.

Read only what helps

Then use the lab and recall check.

More when needed

Transcript and resources stay available below.

Course progress

AI & Robotics Foundations · Robot Perception and Computer Vision · Lesson 16

Images, Depth, and Features

In progress

Decision challenge

Observe the phenomenon. Then connect the representations.

Use the opening example to make a prediction, identify evidence, and explain which model supports it.

Does a Robot Camera See Objects? Pixels, Depth & Features

A camera measures arrays; calibrated depth and features become testable evidence.

A camera measures arrays; calibrated depth and features become testable evidence.

Reference drawerTranscript, source notes, scripts, and package status stay tucked away until you need them.6 files

Lesson reading

live

55 min

Video script

draft

Transcript fallback

available

courses/ai-robotics/modules/06-robot-perception-and-computer-vision/lessons/01-images-depth-and-features/video-transcript.md

From Pixel to 3D Evidence

draft

25 min

Mastery check

live

6 questions / 10 min

Transcript for accessibility and fallback

# Transcript Does a robot camera see objects? No. It receives arrays of measurements. An RGB pixel stores channel values at row v and column u. A depth pixel can add distance Z, but zero or missing depth is not geometry. Calibration supplies focal lengths and the principal point. Together, pixel and valid depth back-project to X, Y, Z in the camera frame. Edges react to change. Corners change in two directions, making useful anchors across frames. But a feature is evidence, not an object. Check encoding, time, frame, alignment, and validity before the robot acts. Which input is missing before a pixel can become a metric 3D point? Continue the worked example on EduQuest.

Reading lab

Core explanation

Connect the lesson's words, diagrams, graphs, evidence, and equations.

The question

Does a robot camera see a chair? No. It receives measurements: an image grid, perhaps a depth value for each pixel, and metadata describing when and where those measurements were taken. Software must turn that evidence into useful estimates.

Mental model: measurement, geometry, evidence

  1. Image: a rectangular array. Pixel (u, v) stores encoded channel values such as red, green, and blue. Resolution adds samples, not guaranteed understanding.
  2. Depth: a distance measurement or estimate associated with a pixel. Invalid, missing, noisy, or misaligned depth must not silently become geometry.
  3. Feature: a repeatable local pattern—often an edge, corner, or keypoint—that can help match observations. A feature is evidence, not an object label.

ROS sensor_msgs/Image records height, width, encoding, row step, data, timestamp, and camera frame. The encoding matters: the same bytes mean different things under RGB, grayscale, or depth encodings.

From pixels to geometric evidence

Worked example: one pixel becomes a 3D ray

Assume an aligned depth image reports Z = 2.0 m at pixel (u, v) = (420, 290). Camera calibration gives fx = fy = 500 px and principal point (cx, cy) = (320, 240).

For a pinhole camera:

[ X=(u-c_x)Z/f_x, \qquad Y=(v-c_y)Z/f_y ]

So X = 0.40 m, Y = 0.20 m, and Z = 2.0 m in the camera optical frame. This is meaningful only when the depth and color pixels are aligned, the intrinsics match the stream, units are known, and the reading is valid.

Retrieval pause: If Z doubles while the pixel and intrinsics stay fixed, what happens to X and Y? They double too.

Why features help

A flat patch may look nearly identical after a small camera motion. A corner changes strongly in two image directions, so it can be localized and matched more reliably. Edges help describe boundaries but can slide along their own direction. Blur, lighting, viewpoint, and repeated texture can still break matches.

EvidenceUseful forCommon failure
RGB valuesappearance and color contrastillumination and exposure change
Depthscale and obstacle geometryholes, range limits, reflective surfaces
Edgeboundary evidenceweak contrast or blur
Corner/keypointmatching across framesrepeated texture or occlusion

Misconception checks

  • “More pixels means correct perception.” No—resolution cannot repair bad calibration, blur, or distribution shift.
  • “Depth is ground truth.” No—inspect invalid values, range limits, alignment, and uncertainty.
  • “A feature is an object.” No—a corner can support tracking, but semantics require additional inference.

Evidence-first checklist

Before a robot acts, ask: What is the encoding? Which timestamp and frame apply? Is depth valid and aligned? Are features repeatable under expected lighting and motion? What fallback handles missing evidence?

Summary

Images are encoded grids; depth adds metric evidence; calibration connects pixels to rays; features provide repeatable anchors. Reliable perception preserves the measurement’s frame, time, units, and uncertainty.

Further learning

Practice labFrom Pixel to 3D EvidenceOpen this when you are ready to apply the model, collect evidence, and check your explanation.25 min

Lab: From pixel to 3D evidence

Objective

Back-project valid depth pixels and distinguish measurements from inferred features.

Materials

  • lab-worksheet.csv
  • Spreadsheet, calculator, or paper
  • No camera or paid software required

Steps

  1. Use fx = fy = 500, cx = 320, and cy = 240.
  2. For each valid row compute X=(u-cx)Z/fx and Y=(v-cy)Z/fy.
  3. Mark zero or missing depth as invalid; do not invent a 3D point.
  4. Compare the two observations of feature A. Decide whether their geometry is consistent enough to investigate as a match.
  5. Record one extra check needed before motion, such as timestamp, frame, or depth alignment.

Expected Result

The center pixel projects to X=0, Y=0; off-center points shift laterally with depth. Invalid depth produces no trustworthy 3D point.

Reflection Questions

  1. Why does doubling depth double lateral coordinates for a fixed pixel?
  2. Why can two similar corners still be a false match?
  3. Which metadata would you log for debugging?

Extension Challenge

Change fx from 500 to 700 and explain why the same pixel projects closer to the optical axis.

Accessibility fallback

All values and formulas are provided as text. Learners may describe direction and validity without drawing the image.