Level 2: Classical Computer Vision¶

Level 2 · Overview · ⏱️ ~4–5 weeks at about an hour a day · Prerequisites: Level 1: Image processing, especially Edges and gradients, The frequency domain (pyramids), and Geometric transformations (homogeneous coordinates and homographies)

Level 1 taught you to process images: filter them, find edges, segment them, and warp them. Level 2 teaches you to make sense of them the classical way, with geometry and hand-designed features: find the same point in two photos, estimate how the photos are related, model the camera that took them, recover depth from two views, follow things that move, and recognize what's in an image. These methods predate deep learning, and they're still running inside nearly every phone camera, AR app, robot, and 3D reconstruction system. Every idea is derived, implemented from scratch in NumPy, and checked against OpenCV or scikit-image.

What this level covers¶

The first two chapters are about correspondence. Features and keypoints derives the Harris corner detector from the structure tensor, builds scale space and blob detection, and walks through SIFT and ORB, measuring their invariance to rotation, scale, and lighting. Matching and RANSAC turns descriptors into reliable matches with the ratio test, derives RANSAC's iteration formula, implements the Direct Linear Transform for homographies, and stitches a panorama.

The next two are about geometry in 3D. Cameras and calibration builds the pinhole model, the projection matrix, and lens distortion, then calibrates a synthetic camera with known ground truth, undistorts images, and uses PnP to draw an augmented-reality cube. Stereo and multi-view geometry derives the epipolar constraint and the essential and fundamental matrices, implements the normalized eight-point algorithm, triangulation, and semi-global matching, and evaluates depth against the Middlebury ground truth.

The last two are about time and meaning. Motion and tracking derives Lucas-Kanade optical flow from brightness constancy, compares it with Farnebäck flow, separates moving objects from the background, and builds a Kalman filter tracker. Classical recognition covers HOG, LBP, bags of visual words, sliding-window detection with hard negative mining, and the Viola-Jones face detector, and ends with why deep learning replaced these methods and where they still win.

What you'll be able to do¶

Derive the Harris response from the structure tensor, explain scale selection, and choose between SIFT and ORB for a task, with measured evidence.
Match features robustly and estimate homographies with your own RANSAC and DLT, knowing how many iterations you need and why.
Write the full camera model, calibrate a camera, judge whether a calibration is trustworthy, and estimate an object's pose with PnP.
Estimate the fundamental and essential matrices, recover relative pose, triangulate points, and compute dense depth with a clear error budget (\(\Delta Z \approx Z^2\Delta d/(fB)\)).
Compute sparse and dense optical flow, choose a background model, and track noisy detections with a Kalman filter.
Build a hand-crafted recognition pipeline (feature, classifier, sliding window, NMS) and explain what each invariance costs.
Evaluate every one of these against ground truth, with the right metric, instead of trusting a picture that "looks right".

Chapters¶

#	Chapter	What it covers	Time
1	Features and keypoints	What makes a good feature, Harris derived, scale space, LoG and DoG blob detection, SIFT step by step, ORB and binary descriptors, measured invariance	~75 min read + 4–5 h practice
2	Matching and RANSAC	Descriptor distances, brute-force and approximate matching, the ratio test, RANSAC derived, the DLT, homography estimation, panorama stitching and blending	~75 min read + 4–5 h practice
3	Cameras and calibration	The pinhole camera, intrinsics and extrinsics, the projection matrix, lens distortion, Zhang's calibration, undistortion, PnP, augmented reality	~80 min read + 5–6 h practice
4	Stereo and multi-view geometry	Epipolar geometry, essential and fundamental matrices, the eight-point algorithm, triangulation, rectification, disparity and depth, block matching and SGM, structure from motion	~85 min read + 5–6 h practice
5	Motion and tracking	Brightness constancy, the aperture problem, Lucas-Kanade and KLT, Farnebäck flow, background subtraction, mean-shift and CamShift, Kalman filters	~80 min read + 4–5 h practice
6	Classical recognition	Color histograms, HOG, LBP, bags of visual words, sliding windows and NMS, HOG + SVM with hard negative mining, Viola-Jones, why deep learning took over	~80 min read + 4–5 h practice

The geometry and cameras cheat sheet collects the transforms, camera equations, epipolar relations, stereo formulas, and OpenCV argument gotchas from this level on one page.

How to study this level¶

flowchart LR
    P["Picture it:<br/>what should happen?"] --> D["Derive it<br/>on paper"]
    D --> S["Build it<br/>from scratch"]
    S --> V["Verify against<br/>ground truth and OpenCV"]
    V --> B["Break it:<br/>find the failure mode"]
    B --> P

Picture it first. Before the math, say what should happen: a corner changes in every direction, a match must lie on a line, depth error grows with distance. Most of the derivations in this level formalize one picture.
Derive the key result on paper. The Harris response, the RANSAC iteration count, the DLT equations, the projection matrix, Zhang's constraints, \(\mathbf{E} = [\mathbf{t}]_\times\mathbf{R}\), \(Z = fB/d\), the Lucas-Kanade normal equations, and the Kalman gain each fit on half a page.
Build it, then match the library. Every chapter implements the core algorithm in NumPy and checks it against OpenCV or scikit-image, often to machine precision. When yours doesn't match, the difference teaches you a convention: (x, y) versus (row, col), a derivative scale factor, a fixed-point disparity, which point set is the source.
Test on ground truth. Synthetic data with known answers (warped images with known homographies, rendered checkerboards with known intrinsics, sequences with known flow, stereo with ground-truth disparity) is the most valuable habit in this level. "It looks right" is not a measurement.
Break it on purpose. Blur the image until SIFT fails, push RANSAC past 90% outliers, calibrate from fronto-parallel views only, put all points on a plane, let an object stop in front of a background model. Knowing the failure modes is what makes these tools safe to use.
Revisit the "Check yourself" questions a few days later, and again a week later.

Geometry before learning

Even in a deep-learning project, the geometry in this level decides whether the system works: camera calibration for any measurement, homographies for document and plane rectification, epipolar geometry and triangulation for 3D, and Kalman filters for tracking. Learned models replace the features; they rarely replace the geometry.

OpenCV 4 versus 5

The code targets OpenCV 5 (opencv-python-headless) and also runs on OpenCV 4.10+, except where noted. The main differences you'll meet: SIFT is in the main package (since 4.4); the old calib3d module is split into calib, geometry, and stereo in C++ but keeps the same Python names; and cv2.CascadeClassifier and cv2.HOGDescriptor aren't available in the OpenCV 5 main package, which Classical recognition works around.

Runtime

Every example runs on a laptop CPU, mostly in seconds. The slowest pieces (the from-scratch sliding-window detector and semi-global matching) take a few seconds each. No GPU is needed in this level.

Capstone¶

The level ends with Stitch a panorama and measure depth: features, RANSAC, homographies, and stereo. In Part A you stitch three overlapping views of a scene with your own RANSAC homographies, measure their error against the known true homographies, and compare your panorama with OpenCV's Stitcher. In Part B you compute disparity on the Middlebury motorcycle pair, convert it to metric depth, and build an error budget against the ground truth. Complete it without notes before moving on to Level 3: CNNs for vision.