The Full Roadmap¶
This page is the handbook's complete syllabus: every tier, level, and chapter, and the concepts each chapter teaches. Use it to see where you are, what comes next, and how the pieces connect.
flowchart LR
subgraph B["🌱 Beginner: Images and Classical Vision"]
L0["Level 0<br/>Images and tools"] --> L1["Level 1<br/>Image processing"] --> L2["Level 2<br/>Classical vision"]
end
subgraph I["🔧 Intermediate: Deep Learning for Vision"]
L3["Level 3<br/>CNNs for vision"] --> L4["Level 4<br/>Object detection"] --> L5["Level 5<br/>Segmentation and<br/>dense prediction"]
end
subgraph E["🚀 Expert: Modern and 3D Vision"]
L6["Level 6<br/>Transformers and<br/>multimodal"] --> L7["Level 7<br/>Generative vision"] --> L8["Level 8<br/>Video, 3D, and<br/>deployment"]
end
L2 --> L3
L5 --> L6| Tier | Levels | You go from… | …to | Time (est.) |
|---|---|---|---|---|
| 🌱 Beginner | 0–2 | Knowing some Python | Understanding what an image is, processing images, and building classical vision systems: panoramas, calibrated cameras, depth from stereo, and tracking | 10–14 weeks |
| 🔧 Intermediate | 3–5 | Classical vision | Training and debugging CNNs, and building object detectors and segmentation models | 12–16 weeks |
| 🚀 Expert | 6–8 | Standard deep vision | Vision transformers, CLIP and vision-language models, diffusion, video, 3D and neural rendering, and shipping vision models to production | 12–18 weeks |
The estimates assume about an hour a day, most days.
The one rule
Every level ends with a capstone project. Don't move on until you can finish it without notes.
Companion handbook
This handbook focuses on vision. For the general foundations it builds on (Python, NumPy, linear algebra, probability, classical machine learning, backpropagation, and PyTorch), it links to the ML Handbook instead of repeating them. You don't need to finish that handbook first, but keep it open as a reference.
🌱 Beginner tier: Images and Classical Vision¶
Goal: Understand images from first principles and master classical computer vision, the toolkit that still powers calibration, 3D reconstruction, tracking, and preprocessing in every modern system.
At the end of this tier you can: explain how a camera turns light into numbers; manipulate images fluently with NumPy and OpenCV; filter, enhance, and segment images; detect and match features; stitch a panorama; calibrate a camera; recover depth from stereo; and track motion in video.
Level 0: Images and tools¶
| Chapter | Key concepts |
|---|---|
| What is computer vision? | Seeing vs understanding · the main tasks (classification, detection, segmentation, tracking, reconstruction, generation) · a short history from edge detectors to foundation models · where CV is used · why vision is hard · how to study it |
| Setting up your vision toolkit | The Python vision stack (NumPy, OpenCV, Pillow, scikit-image, matplotlib, PyTorch, torchvision) · what each library is for · installing them · built-in sample images · notebooks for vision work · GPUs |
| Digital images | Light, sensors, and the Bayer filter · sampling and quantization · resolution and bit depth · grayscale and color · color spaces (RGB, HSV, Lab, YCbCr) · gamma and sRGB · image formats and compression (PNG, JPEG, WebP, RAW) |
| Images as arrays | Height × width × channels · dtypes (uint8 vs float) and overflow · indexing, cropping, and channel manipulation · broadcasting for color math · HWC vs CHW layouts · normalization · stacking images into batches |
| Working with images and video | Reading and writing with OpenCV and Pillow · the BGR vs RGB trap · displaying correctly with matplotlib · resizing and aspect ratio · drawing shapes and text · reading video frame by frame · webcams · writing video |
Capstone: Build a photo-processing tool: batch-convert, resize, annotate, and analyze a folder of images
Level 1: Image processing¶
| Chapter | Key concepts |
|---|---|
| Point operations and histograms | Brightness and contrast · gamma correction · lookup tables · histograms · histogram equalization and CLAHE · thresholding (global, Otsu, adaptive) · color-based segmentation in HSV |
| Convolution and filtering | Convolution vs correlation · kernels · box and Gaussian blur · separable filters · sharpening and unsharp masking · median and bilateral filters · noise models · border handling · implementing convolution from scratch |
| Edges and gradients | Image derivatives · finite differences · Sobel and Scharr · gradient magnitude and direction · Laplacian, LoG, and DoG · the Canny edge detector step by step |
| The frequency domain | Fourier series intuition · the 2D DFT and FFT · magnitude and phase · the convolution theorem · low-pass and high-pass filtering · aliasing and the sampling theorem · image pyramids (Gaussian and Laplacian) |
| Morphology and classical segmentation | Binary images · structuring elements · erosion, dilation, opening, closing · connected components · contours and shape descriptors · distance transform · watershed · region growing · k-means color segmentation |
| Geometric transformations | Homogeneous coordinates · translation, rotation, scaling, shear · affine and projective transforms · homographies · forward vs inverse mapping · interpolation (nearest, bilinear, bicubic) · document rectification |
Capstone: Build a document scanner: find the page, rectify it, and clean it up
Level 2: Classical computer vision¶
| Chapter | Key concepts |
|---|---|
| Features and keypoints | What makes a good feature · the Harris corner detector derived · scale space · blob detection · SIFT step by step · ORB and binary descriptors · invariance to rotation, scale, and lighting |
| Matching and RANSAC | Descriptor distances · brute-force and approximate matching · the ratio test · outliers · RANSAC derived (how many iterations?) · estimating homographies · panorama stitching and blending |
| Cameras and calibration | The pinhole camera · intrinsic and extrinsic parameters · the projection matrix · lens distortion · calibrating with a checkerboard · undistortion · pose estimation (PnP) · augmented reality basics |
| Stereo and multi-view geometry | Epipolar geometry · the essential and fundamental matrices · the eight-point algorithm · triangulation · stereo rectification · disparity and depth · block matching and SGM · structure from motion overview |
| Motion and tracking | The brightness constancy assumption · optical flow (Lucas-Kanade, Farnebäck) · the aperture problem · background subtraction · KLT feature tracking · mean-shift and CamShift · Kalman filters for tracking |
| Classical recognition | Hand-crafted features (HOG, LBP, color histograms) · bag of visual words · sliding windows · Viola-Jones face detection · HOG + SVM pedestrian detection · why deep learning replaced these, and where they still win |
Capstone: Stitch a panorama and measure depth: features, RANSAC, homographies, and stereo
🔧 Intermediate tier: Deep Learning for Vision¶
Goal: Learn how deep networks see, and master the core tasks of modern vision: classification, detection, and segmentation.
At the end of this tier you can: design, train, and debug CNNs; fine-tune pretrained models on your own data; build and evaluate object detectors; train segmentation models; estimate human pose and depth; and use promptable segmentation models.
Level 3: CNNs for vision¶
| Chapter | Key concepts |
|---|---|
| From pixels to learned features | Why hand-crafted features hit a ceiling · neural network recap · why fully connected layers fail on images · convolution as a learned filter · weight sharing and translation equivariance · the ImageNet moment |
| Convolutional networks in depth | Conv, pooling, stride, padding, and dilation · receptive fields · 1×1 convolutions · depthwise-separable convolutions · batch norm · architectures from LeNet, AlexNet, VGG, Inception, and ResNet to EfficientNet and ConvNeXt |
| Training image classifiers | Datasets and dataloaders for images · normalization · data augmentation (geometric, color, Cutout, Mixup, CutMix, RandAugment) · label smoothing · learning-rate schedules · mixed precision · test-time augmentation · reading training curves |
| Transfer learning and fine-tuning | Pretrained backbones · feature extraction vs fine-tuning · which layers to freeze · discriminative learning rates · small-data strategies · domain shift · timm and torchvision model zoos |
| Understanding and debugging CNNs | Visualizing filters and feature maps · saliency maps · Grad-CAM · occlusion sensitivity · shortcut learning · adversarial examples (FGSM) · robustness and calibration · error analysis |
Capstone: Train an image classifier on a new domain, and explain what it learned
Level 4: Object detection¶
| Chapter | Key concepts |
|---|---|
| Detection fundamentals | Bounding box formats · IoU · non-maximum suppression · anchors and priors · box regression parameterizations · precision-recall and mAP (COCO and VOC) · dataset formats (COCO, YOLO, Pascal VOC) |
| Two-stage detectors | R-CNN, Fast R-CNN, and Faster R-CNN · region proposal networks · RoI pooling and RoI Align · feature pyramid networks · training losses and sampling |
| One-stage detectors | YOLO's grid idea · SSD · class imbalance and focal loss (RetinaNet) · anchor-free detectors (FCOS, CenterNet) · the modern YOLO family · speed vs accuracy trade-offs |
| Transformers for detection | Detection as set prediction · DETR · bipartite matching with the Hungarian algorithm · object queries · Deformable DETR · DINO-DETR and RT-DETR overview |
| Detection in practice | Labeling data · converting between formats · training a detector with torchvision · evaluating with COCO metrics · small objects · error analysis (TIDE-style) · deploying a detector |
Capstone: Build and evaluate an object detector end to end
Level 5: Segmentation and dense prediction¶
| Chapter | Key concepts |
|---|---|
| Semantic segmentation | Per-pixel classification · fully convolutional networks · encoder-decoder and U-Net · skip connections · transposed convolution and upsampling · dilated convolution and DeepLab · losses (cross-entropy, Dice, focal) · metrics (pixel accuracy, IoU, Dice) |
| Instance and panoptic segmentation | Instance vs semantic vs panoptic · Mask R-CNN · mask representations (bitmaps, polygons, RLE) · panoptic quality · Mask2Former overview |
| Keypoints and pose estimation | Keypoints as heatmaps · top-down vs bottom-up pose · part affinity fields · OKS and keypoint AP · facial landmarks · hand pose |
| Depth and other dense tasks | Monocular depth estimation · scale ambiguity · supervised vs self-supervised depth · surface normals · optical flow with deep networks (RAFT) · super-resolution · denoising |
| Promptable segmentation | Segment Anything (SAM) · prompts (points, boxes, masks) · image encoders and mask decoders · interactive and automatic segmentation · SAM 2 for video · using foundation models as labeling tools |
Capstone: Train a segmentation model and measure it properly
🚀 Expert tier: Modern and 3D Vision¶
Goal: Master the architectures and ideas at the frontier of vision, and learn to ship vision systems that work in the real world.
At the end of this tier you can: build and train a vision transformer; explain and use self-supervised and vision-language models; build zero-shot classifiers and open-vocabulary detectors; train and steer diffusion models; analyze video and track multiple objects; work with point clouds and neural rendering; and optimize and deploy vision models to servers and edge devices.
Level 6: Transformers and multimodal vision¶
| Chapter | Key concepts |
|---|---|
| Vision transformers | Images as sequences of patches · patch embedding · position embeddings · the class token · ViT from scratch · inductive bias and data hunger · DeiT and distillation · hierarchical transformers (Swin) · CNNs vs ViTs |
| Self-supervised learning for vision | Why learn without labels · pretext tasks · contrastive learning (SimCLR, MoCo) · non-contrastive methods (BYOL, DINO) · masked image modeling (MAE) · linear probing and k-NN evaluation · DINOv2 features |
| Vision-language models | Joint embedding spaces · CLIP and contrastive pretraining · zero-shot classification and prompt engineering · image-text retrieval · captioning · multimodal LLMs (LLaVA-style architectures) · visual question answering |
| Open-vocabulary and foundation models | Open-vocabulary detection (OWL-ViT, Grounding DINO) · grounding and referring expressions · combining detectors with SAM · foundation models as building blocks · evaluation and failure modes |
Capstone: Build a zero-shot image search and classification system with CLIP-style embeddings
Level 7: Generative vision¶
| Chapter | Key concepts |
|---|---|
| Image generation foundations | Generative modeling for images · autoencoders and VAEs · GANs, DCGAN, and training dynamics · StyleGAN ideas · evaluation (FID, IS, precision and recall) |
| Diffusion models for images | The forward and reverse processes · the noise-prediction objective · U-Net denoisers with timestep embeddings · DDPM and DDIM sampling · classifier-free guidance · latent diffusion and Stable Diffusion's architecture |
| Controllable generation and editing | Conditioning on text, class, and images · inpainting and outpainting · image-to-image · ControlNet · personalization (DreamBooth, LoRA) · super-resolution · watermarking and deepfake detection |
Capstone: Train a small diffusion model that generates images, and steer it with guidance
Level 8: Video, 3D, and deployment¶
| Chapter | Key concepts |
|---|---|
| Video understanding | Video as a tensor · temporal modeling · two-stream networks · 3D convolutions (C3D, I3D) · video transformers · action recognition · temporal localization |
| Multi-object tracking | Tracking by detection · the Kalman filter and Hungarian assignment (SORT) · appearance embeddings (DeepSORT) · ByteTrack · MOT metrics (MOTA, IDF1, HOTA) |
| 3D vision and point clouds | 3D representations (depth maps, point clouds, voxels, meshes) · depth sensors and LiDAR · point cloud processing (ICP registration) · PointNet · 3D object detection overview |
| Neural rendering | Novel view synthesis · volume rendering · neural radiance fields (NeRF) · positional encoding · 3D Gaussian splatting · when to use which |
| Deploying vision models | Latency, throughput, and memory budgets · exporting to ONNX · quantization and pruning for vision · TensorRT and OpenVINO overview · edge and mobile deployment · video pipelines · monitoring, drift, and data flywheels · privacy and responsible use |
Capstone: Ship a real-time video analytics pipeline: detect, track, count, and optimize
Alongside every level¶
- Cheat sheets: one-page references to keep open while you practice.
- Glossary: every term defined in one place.
- Progress checklist: tick off chapters and capstones as you go.
- A "mistakes I made" log: write down every BGR mix-up, off-by-one box, and leaky split, and what it taught you.