About this course
Apply deep learning to images and video for recognition, detection, and segmentation tasks.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
A complete computer-vision application for a chosen detection or segmentation domain: from dataset annotation in Roboflow through CNN and ViT fine-tuning with torchvision and timm, augmentation ablation with Albumentations, detection with YOLO and Detectron2, instance segmentation with Mask R-CNN, saliency analysis with Captum, ONNX export, and a live FastAPI inference service evaluated with mAP and mIoU.
Expected outcomes
- Apply classical image processing and feature extraction (spatial filtering, morphological ops, Harris, SIFT, ORB, HOG, Canny, Hough, optical flow) using OpenCV, and explain how learned CNN representations supplant hand-crafted descriptors
- Implement deep CNN architectures (VGG, ResNet, EfficientNet) and Vision Transformers using torchvision and Hugging Face timm, grounding each design choice in convolutional inductive bias, patch tokenization, and self-attention theory
- Apply transfer learning and fine-tuning from ImageNet pre-trained models with systematic augmentation ablation using Albumentations and structured dataset annotation workflows using Roboflow
- Build anchor-based and anchor-free object detection pipelines (Faster R-CNN, YOLO, FCOS) and segmentation systems (FCN, DeepLab, Mask R-CNN, panoptic) using Detectron2 and Ultralytics YOLO
- Interpret model decisions with Captum saliency maps, export models with ONNX, and deploy vision inference APIs with FastAPI
- Evaluate detection and segmentation models with mAP and mIoU, conduct qualitative error analysis, and document augmentation and architecture trade-offs in a complete CV project
Key topics
- CNNs & vision transformers
- Object detection
- Image segmentation
- Transfer learning
Theoretical foundations
The concepts and results this course rests on.
- spatial filtering (Gaussian, Sobel, Laplacian) and histogram equalization as linear signal-processing operations on image intensity; morphological erosion and dilation on binary regions
- classical feature descriptors: Harris corner response, SIFT and ORB keypoint matching, and HOG histograms as hand-crafted invariant representations
- algorithm-driven CV primitives: Canny edge detection, Otsu global thresholding, Hough line and circle transforms, and Lucas-Kanade optical flow for motion estimation
- convolutional inductive bias: local receptive fields, spatial weight sharing across positions, and translation equivariance as sources of parameter efficiency for image data
- deep CNN design: VGG uniform depth, ResNet skip connections and batch normalization for gradient flow, EfficientNet compound scaling, and hierarchical feature maps from edges to semantics
- Vision Transformer architecture: non-overlapping patch tokenization, class token aggregation, learned positional embeddings, and multi-head self-attention over image patches
- transfer learning regimes: ImageNet pre-training as feature initialization, linear probe vs. full fine-tuning trade-offs, and domain adaptation strategies for new visual domains
- object detection paradigms: anchor-based regression in Faster R-CNN and SSD vs. anchor-free heatmap prediction in YOLO, FCOS, and CenterNet; IoU-based non-maximum suppression
- segmentation families: FCN and DeepLab (dilated convolutions, ASPP) for semantic pixel labeling; Mask R-CNN (RoIAlign, mask head) for instance masks; panoptic combination of both
- data augmentation (random crop, color jitter, MixUp, CutMix, AutoAugment) and detection/segmentation evaluation with mean average precision (mAP) and mean intersection over union (mIoU)
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Deep Learning
- Linear algebra and calculus
Weekly schedule 13 weeks · lecture + practice
Classical computer vision
Wk 1
Image formation and spatial filtering
LectureWe cover image formation, color spaces, spatial filtering with Gaussian, Sobel, and Laplacian kernels, histogram equalization, and morphological erosion and dilation on binary images.
PracticeUse OpenCV to load images, apply Gaussian smoothing and Sobel edge detection, equalize histograms, and perform dilation and erosion on thresholded regions.
ProjectSelect a detection or segmentation domain, source raw images, and explore the data with OpenCV to understand lighting and background variation.
Wk 2
Classical feature extraction
LectureWe study Harris corner detection, SIFT and SURF keypoints, ORB binary descriptors, and HOG feature vectors, explaining how each achieves invariance to scale, rotation, and illumination.
PracticeUse Roboflow to upload project images, define label classes, annotate bounding boxes, and export the dataset in YOLO and COCO formats; establish train/val/test splits.
ProjectAnnotate 200+ project images in Roboflow and finalize the class taxonomy and split strategy.
Wk 3
Traditional CV algorithms and augmentation pipelines
LectureWe cover Canny edge detection, Otsu global thresholding, Hough line and circle transforms, and Lucas-Kanade optical flow for dense motion estimation.
PracticeConfigure and compose augmentation pipelines using Albumentations: random crop, horizontal flip, color jitter, and geometric distortions; verify output pixel statistics and visualize augmented samples.
ProjectFinalize the augmentation configuration for the project training set and validate that augmented images remain correctly labeled.
Wk 4
Convolutional inductive bias
LectureWe derive local receptive fields, spatial weight sharing across positions, and translation equivariance, and explain why these biases make convolutional layers parameter-efficient for image data.
PracticeBuild a small CNN using torchvision datasets and transforms; visualize learned filters and feature maps after the first convolutional layer; train it on the annotated project dataset.
ProjectTrain a torchvision CNN baseline on the project dataset, record accuracy and loss curves, and identify failure modes by examining misclassified images.
Deep architectures
Wk 5
Deep CNN architecturesPresentation
LectureWe cover VGG depth, ResNet residual connections and batch normalization, EfficientNet compound scaling, and how hierarchical feature maps progress from edges to textures to semantic parts.
PracticeTeam presentation 1: problem specification, dataset and annotation pipeline, evaluation protocol, and proposed model architecture.
ProjectLock the project specification and begin fine-tuning a ResNet backbone from Hugging Face timm on the annotated dataset.
Wk 6
Vision Transformers
LectureWe derive ViT patch tokenization, class token aggregation, learned positional embeddings, and multi-head self-attention at image scale, contrasting ViT inductive bias with CNNs.
PracticeLoad and fine-tune a ViT model from Hugging Face timm; compare validation accuracy and inference speed with the ResNet baseline; visualize attention maps.
ProjectSwap the backbone to a ViT from timm and benchmark against the CNN baseline; decide which backbone to carry forward.
Transfer learning and augmentation
Wk 7
Transfer learning and fine-tuning strategies
LectureWe study ImageNet pre-training as feature initialization, linear probe vs. full fine-tuning trade-offs, layer-wise learning-rate decay, and domain adaptation for new visual domains.
PracticeImplement MixUp and CutMix augmentation with Albumentations and ablate their effect on validation accuracy alongside baseline and random crop+jitter conditions.
ProjectRun a systematic augmentation ablation study (no augment, basic, MixUp, CutMix) and log all results for the interim presentation.
Wk 8
Augmentation policy and evaluation metricsPresentation
LectureWe cover AutoAugment and RandAugment policy search, then derive mean average precision (mAP) and mean intersection over union (mIoU) as the standard benchmarks for detection and segmentation.
PracticeTeam presentation 2: interim demo with augmentation ablation table, backbone comparison, and mAP or accuracy on the validation set.
ProjectFinalize augmentation configuration based on ablation results; prepare the dataset in Roboflow export format for the detection training phase.
Detection
Wk 9
Anchor-free object detection
LectureWe study anchor-free detection: YOLO grid prediction and objectness scoring, FCOS center-ness maps, and CenterNet heatmap keypoints; we compare them on speed, accuracy, and training complexity.
PracticeTrain a YOLOv8 model from Ultralytics YOLO on the project dataset; tune confidence and NMS thresholds; visualize predicted boxes and per-class confidence scores.
ProjectRun YOLO detection on the project dataset and record per-class mAP at IoU 0.50 and 0.50:0.95.
Wk 10
Anchor-based detection: Faster R-CNN and SSD
LectureWe derive region proposal networks, anchor boxes and regression targets, Faster R-CNN two-stage pipeline, SSD multi-scale anchors, and non-maximum suppression; we analyze the speed-accuracy trade-off.
PracticeConfigure and fine-tune Faster R-CNN in Detectron2 on the project dataset; compare two-stage mAP and inference latency against the YOLO baseline from week 9.
ProjectBenchmark Faster R-CNN vs. YOLO; document the speed-accuracy trade-off and select the primary detector for the project.
Segmentation
Wk 11
Semantic, instance, and panoptic segmentation
LectureWe cover FCN and DeepLabV3 (dilated convolutions, ASPP) for semantic pixel labeling; Mask R-CNN (RoIAlign, mask head) for per-instance masks; and panoptic segmentation combining both paradigms.
PracticeFine-tune Mask R-CNN in Detectron2 for instance segmentation on the project dataset; compute per-class mIoU and visualize predicted instance masks alongside ground truth.
ProjectExtend the project to instance segmentation; evaluate mAP and mIoU on the validation set and compare with the detection-only baseline.
Interpretability and deployment
Wk 12
Saliency, ONNX export, and inference API
LectureWe cover gradient-based attribution (saliency, integrated gradients) for explaining model decisions; ONNX computation graphs for cross-framework portability; and serving vision models as REST endpoints.
PracticeGenerate Captum saliency maps and integrated-gradient attributions for failure cases; export the best project model to ONNX; wrap it in a FastAPI endpoint that accepts an image and returns annotated predictions with confidence scores.
ProjectDeploy the project model as a FastAPI inference service; add Captum error-analysis visualizations to the project report.
Capstone
Wk 13
Final defensePresentation
LectureWe synthesize the classical-to-deep arc, compare CNN and ViT representational trade-offs, and survey open challenges in robustness, efficiency, and out-of-distribution generalization.
PracticeTeam presentation 3: live demo of the deployed FastAPI inference API, mAP and mIoU results, augmentation ablation, Captum error analysis, and an oral defense of all design choices.
ProjectDeliver the complete CV application: annotated dataset (Roboflow), fine-tuned model, ONNX export, FastAPI service, and written project report.
AI tools in this course.
Students use AI assistants to generate and refactor torchvision data pipelines, custom CNN and ViT modules, and Detectron2 and YOLO training configs, vibe-coding from a classifier up to a Mask R-CNN system. They prompt AI to build Albumentations augmentation stacks, synthesize annotation-format converters, and generate unit tests for IoU and mAP computation. AI also helps analyze confusion matrices, saliency maps, and failed detections, explaining why a model misclassifies or where boxes drift.
Student project
Teams build a complete CV application for a chosen detection or segmentation domain, progressing from dataset annotation through a fine-tuned model to a deployed inference API. The project includes an augmentation ablation study, mAP and mIoU evaluation, and qualitative error analysis using Captum saliency maps. Teams present three times (weeks 5, 8, 13) and deliver a working FastAPI service backed by an ONNX-exported model.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Retail product detection and countingMedical image segmentationAutonomous-driving scene parsingWildlife camera-trap recognitionDefect detection in manufacturingSatellite land-cover mappingSports player trackingDocument layout analysis
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- PyTorch: model implementation and training
- torchvision: datasets, transforms, and pretrained models
- timm: state-of-the-art image backbones
- Detectron2: detection and segmentation framework
- Ultralytics YOLO: real-time one-stage detection
- Albumentations: image augmentation
- OpenCV: image processing utilities
- Hugging Face Transformers: vision transformer models
- ONNX Runtime: optimized cross-platform inference
- Weights and Biases: experiment tracking
- Grad-CAM: visual explanations
- Roboflow: dataset management and annotation
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeStanford CS231N: Deep Learning for Computer Vision (Spring 2025)
- YouTubeDeep Learning for Computer Vision, EECS 498-007 (Justin Johnson)
In Hebrew · בעברית
- Dr. Amos Azaria, Ariel University (YouTube)Deep Learning and NLP - קורס למידה עמוקה ועיבוד שפות טבעיות
- Prof. Yossi Keshet (YouTube)למידת מכונה (Machine Learning)
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- PaperDeep Residual Learning for Image Recognition
- PaperAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PaperU-Net: Convolutional Networks for Biomedical Image Segmentation
- PaperMask R-CNN
- TextbookComputer Vision: Algorithms and Applications, 2nd edition
- TextbookDeep Learning
- TextbookDive into Deep Learning
- DocumentationPyTorch Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 1 |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Core · Semester 1 |
| Defense Technologies & Autonomous Systems | Core · Semester 1 |