Skip to content

Geometry and cameras cheat sheet

The formulas, workflows, and OpenCV gotchas from Level 2 on one page. Conventions: pixels are \((u, v)\) or \((x, y)\) with \(x\) to the right and \(y\) down; NumPy arrays are indexed [row, col] = [y, x]; homogeneous points are written \(\tilde{\mathbf{x}} = (x, y, 1)^\top\), and "\(\sim\)" means equal up to scale.

2D transforms

Transform Matrix (3 × 3, homogeneous) DOF Preserves Min. points OpenCV
Translation \(\begin{bmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{bmatrix}\) 2 everything but position 1 warpAffine
Euclidean (rigid) \(\begin{bmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^\top & 1 \end{bmatrix}\) 3 lengths, angles 2 getRotationMatrix2D(c, deg, 1)
Similarity \(\begin{bmatrix} s\mathbf{R} & \mathbf{t} \\ \mathbf{0}^\top & 1 \end{bmatrix}\) 4 angles, ratios of lengths 2 estimateAffinePartial2D
Affine \(\begin{bmatrix} a & b & t_x \\ c & d & t_y \\ 0 & 0 & 1 \end{bmatrix}\) 6 parallel lines, area ratios 3 getAffineTransform, estimateAffine2D
Homography (projective) \(\begin{bmatrix} h_1 & h_2 & h_3 \\ h_4 & h_5 & h_6 \\ h_7 & h_8 & h_9 \end{bmatrix}\) 8 straight lines, cross-ratio 4 getPerspectiveTransform, findHomography
  • Apply a homography: \(\tilde{\mathbf{x}}' = \mathbf{H}\tilde{\mathbf{x}}\), then divide by the third coordinate: \(x' = \tilde{x}'_1/\tilde{x}'_3\).
  • Compose right to left: \(\mathbf{H}_{3 \leftarrow 1} = \mathbf{H}_{3 \leftarrow 2}\,\mathbf{H}_{2 \leftarrow 1}\). Invert to go the other way.
  • A homography relates two images exactly when the scene is planar (\(\mathbf{H} = \mathbf{K}[\mathbf{r}_1\;\mathbf{r}_2\;\mathbf{t}]\) for the plane \(Z = 0\)) or the camera only rotates (\(\mathbf{H} = \mathbf{K}_2\mathbf{R}\mathbf{K}_1^{-1}\)).
  • warpPerspective(src, H, (w, h)) maps src to dst with \(\mathbf{H}\) and samples by inverse mapping. Size is (width, height). To avoid cropping, prepend a translation that moves the warped corners to positive coordinates.

DLT for \(\mathbf{H}\). Each match \((x, y) \to (x', y')\) gives two rows of \(\mathbf{A}\mathbf{h} = \mathbf{0}\):

\[ \begin{bmatrix} x & y & 1 & 0 & 0 & 0 & -x'x & -x'y & -x' \\ 0 & 0 & 0 & x & y & 1 & -y'x & -y'y & -y' \end{bmatrix}\mathbf{h} = \mathbf{0}. \]

Solve with the SVD (last row of Vt). Always normalize first (centroid at 0, mean distance \(\sqrt{2}\)), then \(\mathbf{H} = \mathbf{T}'^{-1}\tilde{\mathbf{H}}\mathbf{T}\).

RANSAC

  1. Sample \(s\) points (minimal set). 2. Fit. 3. Count inliers with residual \(< t\). 4. Keep the best. 5. Refit on its inliers.
\[ N = \left\lceil \frac{\log(1 - p)}{\log(1 - w^s)} \right\rceil \qquad (p = \text{confidence},\; w = \text{inlier ratio},\; s = \text{sample size}) \]
\(w\) \(s = 2\) (line) \(s = 4\) (homography) \(s = 7\) (F, 7-pt) \(s = 8\) (F, 8-pt)
0.9 3 5 8 9
0.7 7 17 54 78
0.5 17 72 588 1,177

(\(p = 0.99\).) \(N\) doesn't depend on the number of points. Threshold: \(t \approx 1.96\sigma\) for 1D residuals, \(2.45\sigma\) for 2D reprojection errors (95% of inliers). Typical: 1–3 px for homographies, ~1 px (Sampson) for \(\mathbf{F}\). Adaptive RANSAC updates \(w\) from the best consensus set. Variants: MSAC, PROSAC, LO-RANSAC, MAGSAC++ (cv2.USAC_MAGSAC).

The camera model

\[ \lambda\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \underbrace{\begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}}_{\mathbf{K}}\,[\mathbf{R} \mid \mathbf{t}]\begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}, \qquad \mathbf{P} = \mathbf{K}[\mathbf{R} \mid \mathbf{t}] \;(3 \times 4,\ 11\ \text{DOF}). \]
Quantity Formula / meaning
World → camera \(\mathbf{X}_c = \mathbf{R}\mathbf{X}_w + \mathbf{t}\) (\(\mathbf{t}\) = world origin in camera coordinates)
Camera center (world) \(\mathbf{C} = -\mathbf{R}^\top\mathbf{t}\)
Normalized coordinates \((x_n, y_n) = (X_c/Z_c, Y_c/Z_c)\); \(\tilde{\mathbf{x}}_n = \mathbf{K}^{-1}\tilde{\mathbf{u}}\)
Pixels \(u = f_x x_d + c_x\), \(v = f_y y_d + c_y\) (after distortion)
Field of view \(\text{FOV}_x = 2\arctan\big(W / (2f_x)\big)\)
Size → distance \(Z = f_x \cdot \text{width}_{\text{real}} / \text{width}_{\text{px}}\)
Rodrigues \(\mathbf{R} = \mathbf{I} + \sin\theta[\mathbf{k}]_\times + (1 - \cos\theta)[\mathbf{k}]_\times^2\), rvec \(= \theta\mathbf{k}\)
Resize image by \(s\) \(f_x, f_y, c_x, c_y\) all scale by \(s\); distortion coefficients don't change

Distortion (OpenCV order (k1, k2, p1, p2, k3)), on normalized coordinates with \(r^2 = x^2 + y^2\):

\[ \begin{aligned} x_d &= x(1 + k_1r^2 + k_2r^4 + k_3r^6) + 2p_1xy + p_2(r^2 + 2x^2), \\ y_d &= y(1 + k_1r^2 + k_2r^4 + k_3r^6) + p_1(r^2 + 2y^2) + 2p_2xy. \end{aligned} \]

\(k_1 < 0\): barrel (wide-angle). \(k_1 > 0\): pincushion (telephoto). Beyond about 120° FOV, use cv2.fisheye.

Calibration workflow

  1. Print a checkerboard on a flat, rigid backing; measure the square size.
  2. Capture 15–30 views: tilted up to ~45° in different directions, at several distances, covering the whole frame including the corners.
  3. findChessboardCorners(gray, (cols, rows)) with inner corners, then cornerSubPix.
  4. calibrateCamera(objpoints, imgpoints, (w, h), None, None, flags=...); consider CALIB_FIX_K3 for ordinary lenses.
  5. Check RMS (typically < 0.5 px), per-view errors, and parameter uncertainties (calibrateCameraExtended). Remove bad views and repeat.
  6. Calibrate at the resolution, zoom, and focus you'll use.
import glob
import cv2
import numpy as np

COLS, ROWS, SQ = 9, 6, 25.0                                     # inner corners, square size in mm
objp = np.zeros((ROWS * COLS, 3), np.float32)
objp[:, :2] = np.mgrid[0:COLS, 0:ROWS].T.reshape(-1, 2) * SQ    # (0,0,0), (SQ,0,0), ... row by row
crit = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.01)
obj_pts, img_pts = [], []
for path in sorted(glob.glob("calib/*.png")):                   # your own calibration photos
    gray = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    ok, corners = cv2.findChessboardCorners(gray, (COLS, ROWS))
    if ok:
        obj_pts.append(objp)
        img_pts.append(cv2.cornerSubPix(gray, corners, (5, 5), (-1, -1), crit))
rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(obj_pts, img_pts, gray.shape[::-1], None, None,
                                                 flags=cv2.CALIB_FIX_K3)
newK, roi = cv2.getOptimalNewCameraMatrix(K, dist, gray.shape[::-1], alpha=0)
map1, map2 = cv2.initUndistortRectifyMap(K, dist, None, newK, gray.shape[::-1], cv2.CV_16SC2)
undistorted = cv2.remap(gray, map1, map2, cv2.INTER_LINEAR)     # reuse the maps for every video frame

Zhang's constraints (per view, \(\mathbf{H} = [\mathbf{h}_1\,\mathbf{h}_2\,\mathbf{h}_3]\), \(\boldsymbol{\omega} = \mathbf{K}^{-\top}\mathbf{K}^{-1}\)): \(\mathbf{h}_1^\top\boldsymbol{\omega}\mathbf{h}_2 = 0\) and \(\mathbf{h}_1^\top\boldsymbol{\omega}\mathbf{h}_1 = \mathbf{h}_2^\top\boldsymbol{\omega}\mathbf{h}_2\). Two equations per view, five unknowns: at least three views (two with zero skew). Then Levenberg-Marquardt on the reprojection error.

Pose (PnP) and AR

  • solvePnP(obj_pts_3d, img_pts_2d, K, dist) → rvec, tvec (object → camera). Needs ≥ 4 points (planar) or ≥ 6 (general DLT). Methods: SOLVEPNP_ITERATIVE (default), SOLVEPNP_EPNP, SOLVEPNP_SQPNP, SOLVEPNP_IPPE (planar).
  • With wrong matches: solvePnPRansac.
  • Draw: projectPoints(pts3d, rvec, tvec, K, dist) and drawFrameAxes(img, K, dist, rvec, tvec, length).

Epipolar geometry

Essential \(\mathbf{E}\) Fundamental \(\mathbf{F}\)
Constraint \(\hat{\mathbf{x}}_2^\top\mathbf{E}\hat{\mathbf{x}}_1 = 0\) (normalized coords) \(\tilde{\mathbf{u}}_2^\top\mathbf{F}\tilde{\mathbf{u}}_1 = 0\) (pixels)
Definition \(\mathbf{E} = [\mathbf{t}]_\times\mathbf{R}\) (for \(\mathbf{X}_2 = \mathbf{R}\mathbf{X}_1 + \mathbf{t}\)) \(\mathbf{F} = \mathbf{K}_2^{-\top}\mathbf{E}\mathbf{K}_1^{-1}\)
Rank, singular values 2, \((\sigma, \sigma, 0)\) 2, \((\sigma_1, \sigma_2, 0)\)
DOF 5 (scale of \(\mathbf{t}\) is lost) 7
Minimal solver 5-point (Nistér) 7-point; 8-point (linear)
OpenCV findEssentialMat, recoverPose findFundamentalMat
  • Epipolar lines: \(\mathbf{l}_2 = \mathbf{F}\tilde{\mathbf{u}}_1\), \(\mathbf{l}_1 = \mathbf{F}^\top\tilde{\mathbf{u}}_2\); line \((a, b, c)\) means \(au + bv + c = 0\).
  • Epipoles: \(\mathbf{F}\mathbf{e}_1 = \mathbf{0}\), \(\mathbf{F}^\top\mathbf{e}_2 = \mathbf{0}\) (\(\mathbf{e}_2\) is camera 1's center seen by camera 2).
  • Eight-point: normalize both point sets; each match gives a row \([x'x, x'y, x', y'x, y'y, y', x, y, 1]\); SVD; zero the smallest singular value of \(\mathbf{F}\); \(\mathbf{F} = \mathbf{T}_2^\top\tilde{\mathbf{F}}\mathbf{T}_1\).
  • Pose from \(\mathbf{E}\): \(\mathbf{E} = \mathbf{U}\operatorname{diag}(1,1,0)\mathbf{V}^\top\), \(\mathbf{R} \in \{\mathbf{U}\mathbf{W}\mathbf{V}^\top, \mathbf{U}\mathbf{W}^\top\mathbf{V}^\top\}\), \(\mathbf{t} = \pm\mathbf{u}_3\); pick the one with points in front of both cameras.
  • Degenerate: all points on one plane, or pure rotation (fit a homography instead).
  • Triangulation (DLT): rows \(u\,\mathbf{p}^{3\top} - \mathbf{p}^{1\top}\) and \(v\,\mathbf{p}^{3\top} - \mathbf{p}^{2\top}\) per view, SVD. triangulatePoints(P1, P2, pts1_2xN, pts2_2xN) returns 4 × N homogeneous points.

Stereo depth

\[ Z = \frac{fB}{d + d_{\text{offs}}}, \qquad \Delta Z \approx \frac{Z^2}{fB}\,\Delta d, \qquad d_{\max} = \frac{fB}{Z_{\min}} \]

(\(d = x_L - x_R\) in pixels on rectified images, \(f\) in pixels, \(B\) baseline, \(d_{\text{offs}} = c_{x,R} - c_{x,L}\), usually 0.) Lateral position: \(X = (x_L - c_x)Z/f\), \(Y = (y - c_y)Z/f\).

Want Do
Better far-range accuracy Longer baseline or focal length (error \(\propto Z^2/(fB)\))
Fewer holes on blank surfaces SGM instead of block matching; projected texture
Fewer wrong pixels Left-right check, uniqueness ratio, speckle filter
Sharper boundaries Smaller blocks, SGM, edge-aware post-filtering
Report quality honestly Coverage plus bad-1/bad-2 rate plus MAE on covered pixels

Rectify calibrated pairs: stereoRectify(K1, d1, K2, d2, size, R, T) → R1, R2, P1, P2, Q; initUndistortRectifyMap + remap; 3D from disparity: reprojectImageTo3D(disp, Q).

Key OpenCV functions and argument gotchas

Function Gotcha
findHomography(src, dst, method, thresh) Returns \(\mathbf{H}\) with dst ~ H src. method=0 is plain least squares (no outlier rejection). mask marks inliers. Points as float32 (N, 2) or (N, 1, 2).
BFMatcher(norm).knnMatch(d1, d2, k=2) queryIdx → d1, trainIdx → d2. NORM_HAMMING for ORB/BRIEF/AKAZE, NORM_L2 for SIFT. Inner lists may have < 2 entries.
FlannBasedMatcher KD-tree index needs float32; ORB needs the LSH index (algorithm=6).
warpPerspective(img, H, (w, h)) Size is (width, height). Use INTER_NEAREST for masks and label images.
findChessboardCorners(img, (cols, rows)) Inner corners, not squares. Refine with cornerSubPix.
calibrateCamera imageSize is (w, h). objectPoints in the same order as the detected corners, float32. Returns RMS first.
undistortPoints(pts, K, dist) Returns normalized coordinates unless you pass P=K (or P=newK).
solvePnP(obj, img, K, dist) Image points are the distorted detections. If you already undistorted, pass dist=None.
Rodrigues(rvec) Returns (R, jacobian); take [0]. Works both ways.
drawFrameAxes, drawKeypoints, line Colors follow the array's channel order. drawFrameAxes draws the X axis in BGR red, which shows as blue on an RGB array.
findFundamentalMat(p1, p2, FM_RANSAC, thr, conf) \(\mathbf{F}\) maps image 1 to lines in image 2 (p2ᵀ F p1 = 0). Can return None or stacked 3 × 3 solutions (7-point).
recoverPose(E, p1, p2, K) \(\mathbf{t}\) has unit length; the scale is up to you.
StereoBM, StereoSGBM .compute(L, R) int16 output × 16: divide by 16.0. numDisparities must be a multiple of 16. Invalid = (minDisparity − 1) × 16. BM needs 8-bit grayscale. SGBM: P1 = 8·cn·bs², P2 = 32·cn·bs².
calcOpticalFlowPyrLK(I0, I1, p0, None) p0 float32 (N, 1, 2); filter by status == 1.
calcOpticalFlowFarneback(I0, I1, None, 0.5, 3, 15, 3, 5, 1.2, 0) Returns (H, W, 2) flow as \((u, v)\) = (x, y) displacement.
Stitcher_create(mode).stitch(imgs) Check status == 0 and component(): views with low match confidence are dropped silently. PANORAMA assumes a rotating camera; SCANS assumes affine.

Quick sanity checks

  • A homography's corners should map where you expect: perspectiveTransform the four image corners and plot them.
  • x2áµ€ F x1 should be near zero for inliers; plot a few epipolar lines and check they pass through the matches.
  • After rectification, matched points should share a row (to a fraction of a pixel).
  • Reprojection error of a calibration or PnP pose should be about the corner-detection noise (0.1–0.5 px). Far larger means a bug; far smaller with few views means overfitting.
  • Depth from stereo should agree with a tape measure at a known distance before you trust the map.

See also