About this course
Learn algorithms and systems for machine learning at scale, where datasets and models exceed a single machine.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Teams train a large vision or language model on a dataset that requires distributed setup, comparing single-GPU, data-parallel DDP, and DeepSpeed ZeRO-2 configurations; additionally implement one streaming ML pipeline with River and one approximate nearest-neighbor index, benchmarking each component against a naive single-machine baseline and reporting throughput, memory, and accuracy trade-offs.
Expected outcomes
- Design and implement data-parallel and model-parallel training pipelines using PyTorch DDP, Horovod, and DeepSpeed, profiling GPU utilization and communication overhead, applying gradient compression, and scaling across multi-node clusters against MLPerf baselines.
- Build ML pipelines over datasets too large for a single machine using Dask and Apache Spark MLlib, apply GPU-accelerated preprocessing with NVIDIA RAPIDS, and tune LightGBM and XGBoost at dataset scale with distributed cross-validation.
- Design and evaluate approximate nearest-neighbor indexes (LSH, HNSW, IVF-PQ) and probabilistic data structures (Bloom filters, Count-Min Sketch, HyperLogLog, reservoir sampling) with formal error-probability bounds for billion-scale data.
- Derive and apply Chinchilla compute-optimal scaling laws, power-law fits, and IsoFLOP frontiers to choose model size, dataset volume, and compute budget; analyze stochastic optimization algorithms (SGD, SVRG, SARAH) for convergence and regret bounds.
- Implement streaming ML models with River for online learning and concept drift, and deploy trained models at scale using NVIDIA Triton Inference Server, benchmarking throughput and latency under realistic serving load.
- Describe federated learning aggregation protocols, differential privacy mechanisms, and secure aggregation, and explain their communication-accuracy trade-offs relative to centralized distributed training.
Key topics
- Distributed training
- Big-data frameworks
- Approximate & streaming algorithms
- Scaling laws
Theoretical foundations
The concepts and results this course rests on.
- Data parallelism and gradient aggregation: all-reduce, ring-allreduce, gradient compression, and synchronous vs. asynchronous SGD
- Model parallelism and pipeline parallelism: tensor parallelism, pipeline stages, micro-batching, and activation checkpointing
- Communication-computation overlap: NVLink vs. InfiniBand bandwidth models, prefetching, and gradient checkpointing strategies
- Mixed-precision training and memory efficiency: FP16/BF16 arithmetic, dynamic loss scaling, and quantization-aware training
- Distributed ML with large-scale data frameworks: pipeline-based distributed feature engineering, distributed model training, and large-scale cross-validation across partitioned datasets
- Approximate nearest-neighbor search: LSH, HNSW, IVF-PQ indexing, and the recall-latency frontier at billion scale
- Probabilistic data structures: Bloom filters, Count-Min Sketch, HyperLogLog, and reservoir sampling with error-probability guarantees
- Scaling laws for model and dataset size: Chinchilla compute-optimal scaling, power-law fits, and IsoFLOP frontiers
- Stochastic optimization and convergence guarantees: SGD noise, variance reduction (SVRG, SARAH), and online regret bounds
- Federated learning and privacy-preserving distributed ML: aggregation protocols, differential privacy mechanisms, and secure aggregation
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Machine Learning
- Algorithms and data structures
- Distributed systems or databases basics
Weekly schedule 13 weeks · lecture + practice
Distributed Training
Wk 1
Data parallelism and gradient aggregation
LectureWe derive data parallelism: all-reduce and ring-allreduce collective algorithms, gradient compression, and the convergence effects of synchronous vs. asynchronous SGD (P1).
PracticeSet up a Ray Train distributed training job; instrument it to profile GPU utilization and communication overhead.
ProjectEstablish a single-GPU training baseline on the chosen vision or language dataset.
Wk 2
Model parallelism and pipeline parallelism
LectureWe cover tensor parallelism, pipeline stages, micro-batching, and activation checkpointing; analyze pipeline bubble overhead and the trade-off between pipeline depth and throughput (P2).
PracticeConfigure DeepSpeed ZeRO-2 and ZeRO-3 stages; compare memory footprint and throughput against a data-parallel baseline run.
ProjectImplement data-parallel DDP training and record single-GPU-to-DDP scaling efficiency.
Wk 3
Communication-computation overlap
LectureWe analyze NVLink vs. InfiniBand bandwidth models, prefetching strategies, and gradient checkpointing to overlap the backward pass with all-reduce collectives (P3).
PracticeRun a multi-node Horovod training job; profile the all-reduce timeline and quantify the fraction of time spent in communication vs. computation.
ProjectScale project training to a multi-GPU setup and measure communication overhead at different batch sizes.
Wk 4
Mixed-precision training and memory efficiency
LectureWe cover FP16 and BF16 arithmetic, dynamic loss scaling, quantization-aware training, and memory-efficient gradient accumulation for training larger models on fixed hardware (P4).
PracticeApply NVIDIA RAPIDS for GPU-accelerated data preprocessing; combine with mixed-precision training and benchmark memory reduction and training throughput.
ProjectEnable mixed-precision in the project training run and record memory savings and step-time improvement.
Wk 5
Distributed training synthesisPresentation
LectureWe synthesize gradient compression, asynchronous SGD variants, and ZeRO sharding; compare DDP, FSDP, and pipeline-parallel configurations across model size and hardware budget.
PracticeTeam presentation: each team defends its distributed training architecture, single-GPU baseline, and DDP scaling results.
ProjectLock the project specification and present the DeepSpeed ZeRO-2 vs. DDP comparison plan.
Approximate and Probabilistic Algorithms
Wk 6
Approximate nearest-neighbor search
LectureWe derive LSH, HNSW, and IVF-PQ indexing; analyze the recall-latency trade-off at billion scale and the effect of product quantization on index quality and memory (P6).
PracticeBuild a large-scale ANN index over an embedding dataset using Dask for distributed nearest-neighbor retrieval; benchmark recall@K vs. query latency against brute-force search.
ProjectImplement the project ANN index and record recall@K against a brute-force baseline.
Wk 7
Probabilistic data structures
LectureWe derive Bloom filters, Count-Min Sketch, and HyperLogLog with their error-probability bounds; cover reservoir sampling for fixed-memory stream summarization (P7).
PracticeImplement Count-Min Sketch and HyperLogLog inside an Apache Spark MLlib pipeline; validate accuracy-vs.-memory trade-offs on a large event log dataset.
ProjectAdd probabilistic cardinality estimation to the project data pipeline and report error vs. memory.
Wk 8
Scaling laws for model and dataset sizePresentation
LectureWe derive Chinchilla compute-optimal scaling, fit power-law curves to loss vs. parameter count and token count, and analyze IsoFLOP frontiers to choose model size and data volume for a fixed compute budget (P8).
PracticeTeam presentation: interim demo of the distributed training pipeline and ANN index with throughput and recall benchmarks.
ProjectApply scaling-law analysis to justify the project model size and dataset volume given the available compute.
Stochastic Algorithms and Big-Data ML
Wk 9
Stochastic algorithms and convergence guarantees
LectureWe analyze SGD noise and variance, derive variance-reduction methods (SVRG, SARAH), and establish convergence rates and online regret bounds for convex and non-convex objectives (P9).
PracticeTrain a LightGBM gradient-boosting model at dataset scale; compare convergence curves under different sampling strategies and learning-rate schedules.
ProjectApply DeepSpeed ZeRO-2 to the project model and compare throughput and memory against the DDP baseline.
Wk 10
Distributed ML with large-scale data frameworks
LectureWe cover pipeline-based distributed feature engineering, distributed model training, and large-scale cross-validation across partitioned datasets using modern big-data ML frameworks (P5).
PracticeBuild an XGBoost training pipeline across partitioned data with distributed cross-validation; tune hyperparameters at dataset scale and profile training throughput per partition.
ProjectAdd a distributed feature-engineering stage to the project pipeline using a big-data framework.
Wk 11
Streaming ML and online learning
LectureWe define the streaming model; cover online learning, concept drift detection, and incremental update strategies for models that must adapt continuously as data arrives.
PracticeBuild a streaming classification pipeline with River; simulate concept drift and measure model adaptation speed and accuracy relative to batch retraining.
ProjectImplement the project streaming ML pipeline with River and benchmark it against a batch-retrain baseline on throughput and accuracy.
Wk 12
Federated learning and privacy-preserving distributed ML
LectureWe cover federated averaging, differential privacy mechanisms (noise calibration, privacy budget), and secure aggregation protocols; analyze the communication cost and accuracy degradation of federated rounds (P10).
PracticeDeploy the trained project model with NVIDIA Triton Inference Server; load-test throughput and latency under simulated serving traffic and compare single-model vs. ensemble serving.
ProjectFinalize all three project components: distributed training comparison, streaming ML pipeline, and ANN index; assemble final benchmarks.
Capstone
Wk 13
Final defensePresentation
LectureWe synthesize distributed training, approximate algorithms, streaming ML, scaling laws, and federated learning, and survey open frontiers in scalable AI systems research.
PracticeTeam presentation: final demo comparing single-GPU, DDP, and DeepSpeed ZeRO-2 configurations; streaming ML pipeline and ANN index benchmarks with oral defense of design choices.
ProjectDeliver the complete project: distributed training benchmarks, streaming ML pipeline, and ANN index, with all results fully reproducible.
AI tools in this course.
Students use AI assistants to generate and refactor PyTorch DDP and DeepSpeed launch scripts, Ray Train and Horovod distributed training configs, and Dask and Spark MLlib pipeline code, vibe-coding the all-reduce, ZeRO-sharded, and multi-node training stages. They prompt AI to implement HNSW and IVF-PQ indexes, write Count-Min Sketch and HyperLogLog tests, synthesize streaming event data for River pipelines, and draft NVIDIA Triton model configs. AI also helps interpret throughput profiles, scaling-efficiency curves, and recall-latency benchmarks to locate the bottleneck a job hit.
Student project
Teams train a large model (vision or language) on a dataset that requires distributed setup, comparing single-GPU, data-parallel DDP, and DeepSpeed ZeRO-2 configurations. Teams additionally implement one streaming ML pipeline with River and one approximate nearest-neighbor index, benchmarking each component against a naive single-machine baseline and reporting throughput, memory, and accuracy trade-offs across all three systems.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Distributed image classifier trainingLarge-scale recommendation systemStreaming clickstream analyticsApproximate near-duplicate detection at web scaleDistributed embedding indexing serviceReal-time fraud-detection pipelinePetabyte log aggregation and alertingCompute-optimal language model pretraining study
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- Ray Train: distributed training orchestration and GPU utilization profiling
- DeepSpeed: ZeRO optimization stages and large-model training
- Dask: parallel dataframes and large-scale ANN retrieval pipelines
- River: streaming ML models and online learning with concept drift handling
- NVIDIA RAPIDS: GPU-accelerated data preprocessing and feature engineering
- Horovod: multi-node distributed training and communication profiling
- LightGBM: gradient boosting at dataset scale with distributed training support
- XGBoost: large-scale boosting across partitioned datasets with distributed cross-validation
- Apache Spark MLlib: distributed ML pipelines and probabilistic data structure computation
- NVIDIA Triton Inference Server: scalable model serving and inference throughput benchmarking
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeMining Massive Datasets, Stanford CS246 [Full Course]
- MIT OCWMathematics of Big Data and Machine Learning (MIT RES.LL-005)
In Hebrew · בעברית
- HIT - Holon Institute of Technology (Campus IL)מבוא למדעי הנתונים: כלים ושיטות
- Prof. Yossi Keshet (YouTube)למידת מכונה (Machine Learning)
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- TextbookMining of Massive Datasets, 3rd edition
- TextbookDesigning Data-Intensive Applications
- PaperScaling Laws for Neural Language Models
- TextbookDeep Learning
- TextbookDive into Deep Learning
- DocumentationRay Documentation
- DocumentationPyTorch Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 1 |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Elective |