Skip to content

Level 2 capstone: stitch a panorama and measure depth

Level 2 · Capstone · ⏱️ 8–12 h · Prerequisites: all of Level 2, especially Matching and RANSAC, Cameras and calibration, and Stereo and multi-view geometry

Features, RANSAC, homographies, and stereo. This capstone puts the geometric half of classical vision to work on two problems with known answers. In Part A you'll stitch three overlapping views into a panorama with homographies you estimate yourself, and measure how close they are to the true ones. In Part B you'll compute disparity on a real stereo pair, convert it to metric depth, and check it against ground truth. Finish it without notes before moving on to Level 3.

The scenario

You've joined a small team building a site-survey tool for construction managers. A tablet app does two things:

  1. Panoramas. The user turns on the spot, taking a few overlapping photos, and the app stitches them into one wide image for the daily log.
  2. Measurements. A stereo camera clipped to the tablet measures distances to objects (the team lead's example: "how far is that scaffold from the wall?").

The team lead, Sam, is skeptical of both features:

"The panorama library we tried drops photos without telling anyone, and nobody knows how accurate the stereo depth really is. Before we ship, I want our own homography pipeline with numbers that show it's right, a fair comparison with OpenCV's stitcher, and an honest error budget for depth: how wrong is it at 2 meters, and at 5?"

You don't have the tablet yet, so you'll test on data where the right answers are known exactly.

The data

Part A: three synthetic views. Use skimage's rocket image (427 × 640) as "the world". Generate three views as a camera rotating about its center would see them: treat the photo as the image of a camera with intrinsics \(\mathbf{K}_s\) (focal length 600 px, principal point at the image center), and make each view with the homography \(\mathbf{H}_i = \mathbf{K}_v\mathbf{R}_i\mathbf{K}_s^{-1}\), where \(\mathbf{K}_v\) has the same focal length and a 300 × 427 sensor, and \(\mathbf{R}_i\) is a rotation with yaw −10°, 0°, and +10° (add a degree or two of pitch and roll). Give each view a different exposure gain (for example 0.88, 1.0, 1.15) and a little Gaussian noise. Because you built the views, you know the true homography between any two of them: \(\mathbf{H}_j\mathbf{H}_i^{-1}\).

Part B: the Middlebury motorcycle pair. skimage.data.stereo_motorcycle() returns a rectified left image, right image, and ground-truth disparity (non-finite where unknown). Its documentation gives the calibration at this resolution: focal length \(f = 994.978\) px, baseline \(B = 193.001\) mm, and a principal-point offset \(d_{\text{offs}} = 31.086\) px between the two cameras, so depth is \(Z = fB/(d + d_{\text{offs}})\).

from skimage import data
scene = data.rocket()                              # Part A
left, right, disp_gt = data.stereo_motorcycle()    # Part B

Tasks

Part A: panorama

  1. Generate the views and the true homographies from each outer view to the middle (reference) view. Display the three views.
  2. Match features between each outer view and the middle view: SIFT (or ORB, with the right distance), plus the ratio test. Report the number of matches.
  3. Estimate each homography with your own RANSAC and normalized DLT. No cv2.findHomography in your pipeline. Seed every random choice. Report the inlier count and the number of iterations, and justify your threshold.
  4. Measure accuracy against the truth: map a dense grid of points through your estimate and through the true homography, and report the mean and maximum distance, separately for the region that overlaps the reference view and for the whole view. Run cv2.findHomography on the same matches as a reference.
  5. Composite the panorama: compute the canvas from the warped corners, warp all views with a translation so nothing is cut off, compensate the exposure differences, and blend (feathering at minimum).
  6. Measure the panorama's quality: map it back into the original scene's coordinates and report the mean absolute error and PSNR against the original photo, with and without exposure compensation.
  7. Compare with cv2.Stitcher in PANORAMA mode: status, which views it used (component()), its estimated focal lengths and relative rotations versus the truth, runtime, and a side-by-side image. Explain any difference in what the two methods assume.

Part B: stereo depth

  1. Compute disparity with cv2.StereoBM and cv2.StereoSGBM. Justify numDisparities from the data, and remember the fixed-point output.
  2. Add a left-right consistency check to SGBM (your own implementation).
  3. Evaluate disparity against the ground truth on pixels that have it: coverage, bad-1.0 and bad-2.0 rates, MAE, and RMSE.
  4. Convert to depth with the stated \(f\), \(B\), and \(d_{\text{offs}}\), and evaluate: median absolute depth error in millimeters, mean relative error, and the share of pixels within 1% and 5%.
  5. Build an error budget by depth: in bands of depth (for example every 0.5 m), report the median disparity error and the median depth error, and compare the latter with the prediction \(\Delta Z \approx Z^2\Delta d/(fB)\).
  6. Show the left image, ground-truth depth, your depth, and a relative-error map, and explain where the large errors are.

Deliverables

  • A notebook or script (or two) that runs end to end from a fresh environment in a couple of minutes on a laptop CPU, with all randomness seeded.
  • A table of homography errors (yours and OpenCV's) and a panorama image beside the cv2.Stitcher result.
  • A table of disparity and depth metrics for BM, SGBM, and SGBM with the left-right check, plus the depth-band error budget.
  • A short write-up for Sam (half a page): how accurate the homographies are and why; whether to use your stitcher or OpenCV's, and the risks of each; and the depth error to expect at 2 m and at 5 m with this camera, with the formula that predicts it.

Acceptance checklist

  • Every homography comes from your own RANSAC + normalized DLT, and reruns give identical numbers.
  • Mean homography error in the overlap is below 1 pixel for both outer views, and you can explain why it's larger outside the overlap.
  • Your errors are within about 0.1 px of cv2.findHomography's on the same matches.
  • The panorama has no visible exposure seams, and its PSNR against the original improves with exposure compensation.
  • You checked which views cv2.Stitcher actually used, and compared its focal length and rotations with the truth.
  • Disparities were divided by 16, numDisparities covers the ground-truth range, and every metric is reported with its coverage.
  • Depth uses \(Z = fB/(d + d_{\text{offs}})\) with the documented values, and the depth-band errors agree with \(Z^2\Delta d/(fB)\) to within a few millimeters.
  • The write-up states concrete numbers and the conditions under which they hold.

Hints

Hint 1: getting the true homographies right

Write down what each matrix maps from and to. If \(\mathbf{H}_i\) maps scene pixels to view i pixels, then a view-\(i\) pixel goes back to the scene with \(\mathbf{H}_i^{-1}\) and on to view \(j\) with \(\mathbf{H}_j\), so \(\mathbf{H}_{j \leftarrow i} = \mathbf{H}_j\mathbf{H}_i^{-1}\). Test it before anything else: map a few view-\(i\) pixels through it and check that the pixel values in view \(j\) match.

Hint 2: source and destination in matching

In knnMatch(d_src, d_dst, k=2), queryIdx indexes the first descriptor set and trainIdx the second. Build your source points from queryIdx and destination points from trainIdx, and estimate \(\mathbf{H}\) with \(\text{dst} \sim \mathbf{H}\,\text{src}\). If your warped view lands in the wrong place, you probably have \(\mathbf{H}^{-1}\).

Hint 3: RANSAC threshold and iterations

SIFT keypoints are localized to roughly a pixel or better, and the views were made with interpolation and noise, so a threshold of 1.5–3 px is reasonable. Use the adaptive iteration count \(N = \lceil\log(1-p)/\log(1-w^4)\rceil\), updating \(w\) whenever you find a larger consensus set, and refit on all inliers at the end.

Hint 4: black borders and the blend

The views have black corners where the rotation pulled in pixels from outside the original photo. Warp a validity mask along with each view and use it (eroded by a pixel) in the blending weights and in the exposure estimate. Otherwise black pixels get averaged into the panorama and drag the gain estimates down.

Hint 5: measuring the panorama against the original

Your canvas is the reference view's frame shifted by \(\mathbf{T}\). The reference view maps to the scene by \(\mathbf{H}_1^{-1}\). So the canvas maps to the scene by \(\mathbf{H}_1^{-1}\mathbf{T}^{-1}\): warp the panorama (and its coverage mask) with that, and compare only where the mask is set, eroded a little to stay away from borders.

Hint 6: the right disparity map for the left-right check

StereoSGBM only computes the left view's disparity. Flip both images horizontally, swap them, run the same matcher, and flip the result back: that's the right view's disparity. Then keep a left pixel at column \(x\) with disparity \(d\) only if the right disparity at column \(x - d\) agrees within a pixel.

Hint 7: medians versus means

Stereo errors are heavy-tailed: most pixels are within a fraction of a pixel, and a few are off by tens of pixels. Report medians for the "typical" error and the bad-pixel rate or RMSE for the tail. They answer different questions, and Sam needs both.

Solution

A complete worked solution, with code, outputs, and discussion, is in Level 2 capstone solution. The same code is in the repository as scripts/capstones/level-2/part_a_panorama.py and scripts/capstones/level-2/part_b_stereo_depth.py. Try the capstone yourself first.