About this course
Architect large systems whose core is data and machine learning, balancing throughput, latency, consistency, and the demands of AI workloads.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Teams build one end-to-end data-intensive system that ingests a real multi-source dataset through a Kafka topic, processes it via PySpark ETL and dbt ELT transformations into a partitioned Parquet lakehouse stored in Cassandra, integrates a trained ML model served through MLflow for inference on arriving records, and adds Great Expectations quality gates and Evidently drift monitoring across the full pipeline.
Expected outcomes
- Design reliable, scalable, and maintainable data-intensive systems by applying storage-engine internals (B-trees, LSM-trees, WAL) to justify read/write trade-offs and select appropriate storage formats including Parquet and Delta Lake.
- Build batch ETL pipelines using PySpark on Databricks with the MapReduce model, processing records through map, shuffle-sort, and reduce phases into partitioned Parquet and Delta Lake lakehouse tables.
- Implement event-time streaming pipelines with Apache Kafka and Apache Flink, correctly handling watermarks, tumbling/sliding/session windows, late arrivals, and exactly-once delivery semantics.
- Analyze distributed replication and partitioning strategies, apply CAP theorem and PACELC to system configuration decisions, and implement Raft/2PC consensus for fault-tolerant exactly-once guarantees.
- Model analytical data using Kimball star schema, SCD types, and data vault, automate ELT transformations with dbt and Airflow, enforce data quality contracts with Great Expectations, and track lineage with DataHub.
- Integrate trained ML models into production via MLflow, instrument Evidently drift monitoring across the full pipeline, and identify and remediate hidden technical debt including feedback loops and training-serving skew.
Key topics
- Data pipelines & streaming
- System architecture & trade-offs
- Distributed storage
- ML system integration
Theoretical foundations
The concepts and results this course rests on.
- Reliability, scalability, and maintainability as orthogonal design properties with distinct failure modes and levers (P1)
- Storage-engine internals: B-trees vs LSM-trees, write-ahead log (WAL), compaction strategies, and read/write amplification (P2)
- MapReduce computational model: map phase, shuffle and sort, reduce phase, combiner optimization, and partitioner design (P3)
- Event-time dataflow: watermarks, tumbling, sliding, and session windows, late arrivals, and trigger semantics (P4)
- Replication strategies: single-leader, multi-leader, and leaderless; replication lag; and conflict resolution policies (P5)
- Partitioning schemes: range vs hash partitioning; hotspot risk; secondary index design; and cross-partition query cost (P6)
- CAP theorem and PACELC: consistency, availability, and partition tolerance; latency vs consistency under normal operation (P7)
- Distributed consensus and exactly-once semantics: Paxos, Raft, two-phase commit, and idempotent message delivery (P8)
- Hidden technical debt in ML systems: feedback loops, training-serving skew, undeclared consumers, and data dependency rot (P9)
- Dimensional data modeling and ELT architecture: Kimball star schema (fact tables, dimension tables, conformed dimensions), snowflake schema, SCD types 1/2/3, data vault for audit-complete raw storage, and ELT push-down transforms with reverse ETL (P10)
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Databases
- Algorithms and data structures
- Machine Learning
Weekly schedule 13 weeks · lecture + practice
Foundations
Wk 1
System Foundations: Reliability, Scalability, Maintainability
LectureIntroduce reliability, scalability, and maintainability as orthogonal system properties; map their distinct failure modes and design levers; survey the landscape of data-intensive system challenges.
PracticeSet up the project repository, agree on data domain and pipeline goals, and sketch the end-to-end system diagram with explicit reliability and scalability requirements.
ProjectProject repository and system-design baseline established with stated reliability and scalability goals.
Wk 2
Storage-Engine Internals and Columnar Formats
LectureExamine B-trees vs LSM-trees, write-ahead log (WAL) mechanics, compaction strategies, and read/write amplification; introduce Apache Parquet as a columnar storage format optimized for analytical workloads via column pruning and predicate pushdown.
PracticeProfile B-tree and LSM-tree storage behavior and write project raw data as Apache Parquet files, measuring the compression ratio and query scan cost versus row-oriented storage.
ProjectProject raw data is stored as Parquet and the storage format choice is benchmarked and justified.
Wk 3
Event-Time Dataflow: Kafka and Flink
LectureIntroduce event-time vs processing-time semantics, watermarks, tumbling/sliding/session windows, late arrivals, and trigger semantics; contrast Apache Kafka's durable distributed log with Apache Flink's stateful stream processor and its checkpointing model.
PracticeStand up an Apache Kafka topic for project event ingestion; build an Apache Flink job that reads from Kafka, applies a tumbling-window aggregate with a watermark, and counts and routes late arrivals.
ProjectProject ingests events through a Kafka topic and a Flink streaming job processes them with correct watermark and windowing semantics.
Wk 4
Batch Processing: MapReduce, Spark, and the Lakehouse
LectureCover the MapReduce computational model (map, shuffle-sort, reduce, combiners, partitioners) and the Apache Spark execution engine (DAG planning, stages, shuffles, catalyst optimizer); introduce Delta Lake for ACID lakehouse tables with time-travel and schema enforcement.
PracticeBuild a PySpark ETL pipeline on Databricks that reads raw Kafka-staged data, applies multi-source joins and aggregations, writes curated Delta Lake tables in Parquet format, and measures shuffle and stage behavior via the Spark UI.
ProjectProject produces curated Delta Lake datasets through a tested PySpark batch pipeline on Databricks.
Wk 5
ETL Architecture and Specification MilestonePresentation
LectureReview ETL vs ELT architectural boundaries, orchestration responsibilities and data contracts; demonstrate Apache Airflow DAG anatomy — operators, sensors, XComs, and SLA monitoring — as the orchestration backbone for the pipeline.
PracticeStudent teams present their project specification: data sources, Kafka ingestion topic design, Spark/PySpark ETL flow, Parquet/Delta lakehouse schema, Airflow orchestration plan, and scaling goals.
ProjectApproved specification with ETL/ELT architecture, data contracts, Airflow orchestration plan, and pipeline design delivered.
Distributed Systems
Wk 6
Replication Strategies
LectureAnalyze single-leader, multi-leader, and leaderless replication; model replication lag; examine read-your-writes and monotonic-read consistency guarantees; and compare conflict-resolution policies (last-write-wins, version vectors, CRDTs) under concurrent writes.
PracticeConfigure replication topology for the project Cassandra store, simulate replication lag by throttling a replica, and observe staleness anomalies against different consistency levels.
ProjectProject data store is replicated and replication lag behavior is documented with measured consistency anomalies.
Wk 7
Partitioning, CAP, and PACELC
LectureContrast range and hash partitioning; assess hotspot risk, secondary index design under partitioning, and cross-partition query cost; apply CAP theorem and PACELC to configuration choices, distinguishing latency vs consistency under normal operation from behavior under partition.
PracticeAdd consistent hash partitioning to the project Cassandra store, run cross-partition queries, and validate latency vs consistency behavior under a simulated network partition.
ProjectProject store is partitioned and its CAP/PACELC operating point is measured and justified in the design document.
Wk 8
Interim Demo MilestonePresentation
LectureCover distributed consensus: Paxos and Raft leader election, log replication and safety, two-phase commit (2PC), and idempotent message delivery as the foundation of exactly-once semantics in distributed pipelines.
PracticeStudent teams present an interim demo showing Kafka ingestion, Flink streaming, and the Spark ETL pipeline with the replicated, partitioned Cassandra store running live against real project data.
ProjectWorking streaming and batch pipeline with replicated, partitioned store demonstrated end-to-end.
Analytical Engineering & ML Systems
Wk 9
Exactly-Once Semantics and Consensus in Practice
LectureApply Raft and 2PC to real system designs; examine Kafka transactional producers, Delta Lake ACID transaction log with optimistic concurrency, and Flink checkpointing as concrete exactly-once mechanisms; discuss idempotent consumer design and retry safety.
PracticeEnable Kafka exactly-once transactional producers and activate Delta Lake transaction log in the project; inject duplicate messages and failed writes, then verify idempotent delivery and rollback behavior.
ProjectProject pipeline delivers exactly-once guarantees verified by injected failure tests against the Delta Lake store.
Wk 10
Dimensional Modeling and ELT Architecture
LectureIntroduce Kimball star schema (fact tables, dimension tables, conformed dimensions), snowflake schema, slowly changing dimensions (SCD types 1/2/3), and data vault for audit-complete raw storage; explain ELT philosophy — push-down transforms, in-database ML, and reverse ETL for operational data sync.
PracticeBuild dbt models that transform project Delta Lake tables into a Kimball star schema with SCD-2 dimensions; configure Apache Airflow DAGs to orchestrate the full Kafka-to-dbt ELT flow end-to-end.
ProjectProject exposes a dimensional star schema built by dbt, orchestrated end-to-end by Airflow with SLA-monitored DAGs.
Wk 11
Data Quality, Lineage, and Governance
LectureExamine the cost of poor data quality in production pipelines, the role of data contracts for producer-consumer decoupling, and approaches to data cataloging and lineage tracking; introduce Great Expectations validation suites and DataHub lineage graphs and metadata ingestion.
PracticeAdd Great Expectations expectation suites to the project ETL pipeline at the raw, curated, and dimensional layers; register datasets, jobs, and column-level lineage in DataHub using its metadata ingestion API.
ProjectProject enforces Great Expectations quality gates at every layer and all datasets and transformations are catalogued with lineage in DataHub.
Wk 12
ML Technical Debt and Production Monitoring
LectureAnalyze hidden technical debt in ML systems: feedback loops, training-serving skew, undeclared consumers, and data dependency rot; introduce MLflow for experiment tracking, model registry, and low-latency model serving behind a pipeline; and Evidently for data drift and model performance monitoring reports.
PracticeRegister the project ML model in MLflow and configure it for inference on arriving Kafka records; add Evidently drift monitoring reports and data quality profiles across the full Kafka-to-prediction pipeline.
ProjectProject serves predictions from an MLflow-registered model and monitors data and model drift with Evidently across the full pipeline.
Capstone
Wk 13
Final Demo and DefensePresentation
LectureSynthesize data-intensive and AI system architecture across the full course, revisiting reliability, scalability, consistency, dimensional modeling, and ML debt dimensions; discuss how each architectural decision propagates through the system.
PracticeStudent teams present the final end-to-end demo with an oral defense covering Kafka ingestion, Flink and Spark processing, Delta Lake storage, dbt dimensional modeling, Great Expectations and DataHub governance, MLflow serving, and Evidently drift monitoring.
ProjectFinal end-to-end data-intensive AI system delivered with documentation and oral defense of all architectural and operational decisions.
AI tools in this course.
Students use AI assistants to generate PySpark DataFrame transformations, Spark SQL queries, Databricks notebooks and jobs, Flink transformations, and Kafka producers, consumers, and schema definitions. They prompt tools to build ETL data-quality checks, synthesize event streams, and write tests for exactly-once and fault-tolerance paths, while connecting agents to database, Databricks, MLflow, and pipeline MCP servers to inspect state and metrics. AI helps wire model serving, feature processing, and Evidently drift monitoring, and to analyze lineage and operational telemetry into remediation steps. Because generated distributed code can hide consistency, shuffle, partitioning, or replay bugs, students validate every pipeline change against partitioning, replication, and delivery-guarantee behavior.
Student project
Teams build one end-to-end data-intensive system that ingests a real multi-source dataset through a Kafka topic, processes it via PySpark ETL and dbt ELT transformations into a partitioned Parquet lakehouse stored in Cassandra, integrates a trained ML model served through MLflow for inference on arriving records, and adds Great Expectations quality gates and Evidently drift monitoring across the full pipeline.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Real-time analytics platformRecommendation system pipelineFraud detection streamLog analytics and searchClickstream insights enginePredictive maintenance systemSocial media trend trackerSensor data lakehouse
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- Apache Kafka: stream and buffer event data
- Apache Spark: run batch ETL and distributed processing
- PySpark: implement Spark transformations and DataFrame pipelines in Python
- Databricks: develop notebooks, jobs, and lakehouse workflows on managed Spark
- Apache Parquet: store columnar analytical datasets
- Apache Flink: process streams with event-time semantics
- PostgreSQL: store relational and derived data
- Apache Cassandra: provide distributed partitioned storage
- Apache Airflow: orchestrate data pipelines
- dbt: transform and model warehouse data
- Great Expectations: validate ETL outputs and data contracts
- MLflow: track and serve machine learning models
- Feast: manage a feature store
- Docker: package pipeline and serving components
- Evidently: monitor data and model drift
- DataHub: catalog datasets and track data lineage
- Delta Lake: provide ACID transactions and time-travel on lakehouse tables
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeMIT 6.824 Distributed Systems (Spring 2020)
- MIT OCWDistributed Computer Systems Engineering (6.824)
In Hebrew · בעברית
- Campus ILמבוא למדעי הנתונים: כלים ושיטות
Primary literature
Seminal works for advanced study.
- PaperMapReduce: Simplified Data Processing on Large Clusters
- PaperDynamo: Amazon's Highly Available Key-value Store
- PaperSpanner: Google's Globally-Distributed Database
- PaperIn Search of an Understandable Consensus Algorithm (Raft)
- PaperThe Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing
- PaperHidden Technical Debt in Machine Learning Systems
References
Books and resources link to an online or publisher page.
- TextbookDesigning Data-Intensive Applications
- TextbookStreaming Systems
- PaperMapReduce: Simplified Data Processing on Large Clusters
- PaperDynamo: Amazon's Highly Available Key-value Store
- PaperHidden Technical Debt in Machine Learning Systems
- TextbookDesigning Machine Learning Systems
- DocumentationApache Kafka Documentation
- DocumentationApache Spark Documentation
- DocumentationDatabricks Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Core · Semester 2 |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 2 |
| AI and Quantum Computing for Finance | Core · Semester 2 |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Elective |