About this course
Apply DevOps discipline to machine-learning and LLM systems, automating training, deployment, monitoring, and continuous delivery of models in production.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Build, deploy, and operate a production AI service end to end: a containerized, CI/CD-gated REST API on Kubernetes, fed by a medallion data pipeline, with an MLflow model registry, a gateway-fronted RAG feature, live drift monitoring, and a security review against the OWASP LLM Top 10.
Expected outcomes
- Design and operate end-to-end CI/CD pipelines for ML: automate data validation, training triggers, and model evaluation gates using version-controlled promotion workflows and Git-based experiment branching.
- Build medallion data pipelines with schema enforcement and data contract testing, and design feature stores that prevent training-serving skew through point-in-time joins and online/offline parity.
- Apply blue-green, canary, shadow mode, and champion-challenger deployment strategies; provision GPU/TPU infrastructure; and serve models at scale with BentoML and Ray Serve.
- Implement statistical drift detection for covariate shift, label drift, and concept drift using EvidentlyAI and Arize AI, and wire automated retraining triggers into the CI/CD pipeline.
- Deploy and operate LLM inference services with RAG pipelines, prompt versioning, token-economy cost dashboards, and trace-level observability via Langfuse, and build evaluation suites and guardrails for production.
- Govern AI systems against the OWASP LLM Top 10, maintain audit trails, and carry one running AI service from specification through a fully governed production deployment across all five operational layers.
Key topics
- CI/CD pipelines
- Model serving & versioning
- Monitoring & drift detection
- LLMOps & agent operations
Theoretical foundations
The concepts and results this course rests on.
- Git-based data versioning and code-data co-evolution: branching strategies for AI experiments and reproducible lineage (P1)
- The ML CI/CD contract: automated data validation, training triggers, model evaluation gates, and version-controlled staged rollout (P2)
- Infrastructure economics for AI workloads: GPU/TPU provisioning, spot instance preemption strategies, and distributed training job scheduling (P3)
- Safe model release theory: blue-green, canary, shadow mode, and online champion-challenger evaluation with statistical validity (P4)
- SLIs, SLOs, error budgets, and the RED method applied to AI services: latency percentiles, throughput, and actionable alert thresholds (P5)
- The reproducibility triple: parameter logging, artifact versioning, environment pinning, and experiment lineage for auditable ML (P6)
- Feature store design: point-in-time joins, online/offline parity guarantees, and feature versioning to prevent training-serving skew (P7)
- Statistical drift theory: covariate shift, label drift, and concept drift; PSI, KS tests, and embedding distance as detection metrics (P8)
- Medallion data architecture: Bronze-Silver-Gold layers, schema enforcement, and data contract testing between producers and consumers (P9)
- LLMOps fundamentals: prompt versioning, RAG pipeline observability, token cost accounting, and LLM evaluation metrics (faithfulness, answer relevance, retrieval recall) in production (P10)
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Machine Learning
- Software engineering and Python
- Operating systems and networking
Weekly schedule 13 weeks · lecture + practice
Part I: Versioning, CI/CD & Experiment Tracking
Wk 1
Production AI & the Ops Landscape
LectureThe prototype-to-production gap and the 90/10 inversion; SLIs, SLOs, and error budgets; the five operational layers (DevOps, DataOps, MLOps, LLMOps, AgentOps); Git-based data versioning, experiment branches, and code-data co-evolution (P1).
PracticeSet up a team repository with branch protection and an experiment-branching convention; containerize a hello-service with pinned versions and a DVC-tracked dataset pointer.
ProjectTeam repo created with experiment-branching strategy documented; hello-service containerized; project domain shortlisted.
Wk 2
Infrastructure for AI Workloads
LectureGPU/TPU provisioning, spot instance preemption strategies, and distributed training job schedulers; cloud storage for datasets and model artifacts; the cost model and blast radius; annotation pipelines and their role in the training loop (P3).
PracticeProvision a GPU-enabled cloud instance with a budget alert; curate a small labeled dataset using Label Studio and push it to cloud storage.
ProjectCloud environment provisioned; a labeled training sample created with Label Studio and versioned in the team bucket.
Wk 3
CI/CD Pipelines for ML
LectureDORA metrics and the testing pyramid; the ML CI/CD contract: automated data validation, training triggers, model evaluation gates, and staged rollout; Prefect for pipeline orchestration and idempotent retries (P2).
PracticeBuild a GitHub Actions pipeline with Prefect-managed training; implement a data validation gate and an evaluation threshold that blocks promotion on regression.
ProjectCommit to the project use case; CI/CD pipeline live with a Prefect-orchestrated training workflow; every merge triggers validation and evaluation gating.
Wk 4
Experiment Tracking & Model Registry
LectureThe reproducibility triple (git SHA, data version, environment); parameter logging, artifact versioning, environment pinning, and experiment lineage; the model registry, model cards, and version-controlled promotion workflows (P6).
PracticeInstrument training with MLflow (runs, parameters, metrics, artifacts); register models with a version-controlled promotion workflow; reproduce a run from a pinned snapshot using ClearML.
ProjectTraining fully tracked with MLflow; model v1 registered with a promotion workflow; results reproducible from a ClearML-pinned configuration.
Part II: DataOps
Wk 5
Data Quality, Medallion Architecture & Dataset VersioningPresentation
LectureBronze-Silver-Gold medallion architecture; schema enforcement and data contract testing between producers and consumers; DVC for dataset versioning and lineage; idempotent pipeline design (P9).
PracticeStudent Presentation 1 (Specification): each team presents the problem statement and success metrics (SLOs), system and data architecture, CI/CD and versioning status, and a risk and governance register; submits a written report and a tagged release.
ProjectAn orchestrated Bronze-Silver-Gold pipeline with retries; dataset versioned with DVC and reproducible from a pinned snapshot; Presentation 1 delivered.
Part III: Feature Stores & Deployment Strategies
Wk 6
Feature Store Design & Training-Serving Skew Prevention
LectureFeature store architecture; point-in-time joins and temporal correctness; online versus offline store parity; feature versioning and shared feature definitions; training-serving skew as a root cause of silent model degradation (P7).
PracticeSet up Feast as a feature store; define shared feature views serving both the training pipeline and the inference API; validate online/offline parity with quality checks.
ProjectFeast integrated; shared feature definitions drive both training and inference from one source; online/offline parity test passes in CI.
Wk 7
Deployment Strategies for ML Models
LectureSafe model release strategies: blue-green for zero-downtime swaps, canary for gradual traffic shifting, shadow mode for risk-free comparison, and champion-challenger for online evaluation; serving patterns (online, batch, streaming) (P4).
PracticePackage and deploy a model with BentoML; execute a canary rollout and rollback; run a shadow-mode comparison against the champion; scale the serving layer with Ray Serve.
ProjectModel v1 deployed via BentoML with a canary rollout demonstrated; Ray Serve cluster handling production load; shadow-mode comparison logged.
Part IV: Monitoring & Observability
Wk 8
Monitoring, Observability & SLOs for AI ServicesPresentation
LectureLatency SLOs, throughput metrics, error budgets, and alert thresholds for AI services; the RED method applied to model endpoints; building actionable runbooks; blameless postmortems and day-two operations (P5).
PracticeStudent Presentation 2 (Interim): teams demonstrate live a working data-to-model pipeline, tracked and registered model, canary rollout in progress, a RED dashboard with active alerts, and a monitoring plan; submit a report and a tagged release.
ProjectRED dashboard live with p95 latency and error-budget burn-rate alerts; runbook written; Presentation 2 delivered.
Part V: Drift, LLMOps & Governance
Wk 9
Model Drift Detection & Automated Retraining Triggers
LectureCovariate shift, label drift, and concept drift: what each is, why it matters, and how to detect it; PSI, KS tests, and embedding distance as detection metrics; retraining trigger design paired with documented runbook actions (P8).
PracticeInstrument the serving pipeline with EvidentlyAI for data and model drift reports; configure automated retraining triggers wired into the CI/CD pipeline.
ProjectEvidentlyAI drift detectors live on model inputs and outputs; automated retraining trigger documented and tested with synthetic drift injection.
Wk 10
LLMOps: Token Economy, RAG Pipelines & Vector Databases
LecturePrompt versioning and the prompt-as-code discipline; RAG pipeline architecture (embeddings, vector databases, chunking strategies, grounded prompts); token cost accounting and dashboards; LLM evaluation in production: faithfulness, answer relevance, and retrieval recall (P10).
PracticeBuild a RAG service over a real corpus with a vector database; implement prompt versioning; wire Arize AI for trace-level cost, latency, and retrieval-quality logging.
ProjectRAG pipeline serving queries from the project corpus; Arize AI instrumented with token cost and latency dashboards; prompt versions tracked in the registry.
Wk 11
LLM Evaluation Suites, Guardrails & Langfuse Observability
LectureBuilding evaluation suites that mean something: LLM-as-judge with calibration, regression suites wired into CI, and human spot-check protocols; guardrails against prompt injection, jailbreaking, and data leakage; Langfuse for trace-level LLM observability.
PracticeBuild a representative eval suite and run it as a CI regression gate; add Langfuse tracing to every LLM call (prompt version, tokens, cost, latency); demonstrate a guardrail blocking an injection attempt.
ProjectEval suite with at least fifty items wired into CI; Langfuse tracing live on all LLM calls; one guardrail with a measured rejection rate demonstrated.
Wk 12
AI Governance & OWASP LLM Top 10
LectureOWASP LLM Top 10 risks and mitigations (prompt injection, insecure output handling, supply-chain vulnerabilities, sensitive information disclosure, and others); AI supply-chain security: secrets, dependency tracking, provenance, and SBOM; audit trails, model cards, and governance frameworks (NIST AI RMF).
PracticeConduct a structured security review of the project system against OWASP LLM Top 10; implement audit logging and update model governance artifacts (model card, data card).
ProjectSecurity review report complete with mitigations for every applicable risk; audit trail in place; model cards updated; all five operational layers (DevOps, DataOps, MLOps, LLMOps, governance) operational.
Part VI: Final Presentation
Wk 13
Security, Governance & SynthesisPresentation
LectureSynthesis: how DevOps, DataOps, MLOps, LLMOps, and AgentOps compose into a governed AI system; end-to-end walkthrough of the five-layer stack; lessons from production failures and what good looks like.
PracticeStudent Presentation 3 (Final, with oral defense): teams deliver an end-to-end production demo (data in, decision out, live), show observability with an actionable alert and runbook, demonstrate evaluation, guardrails, and a cost and latency report, and defend a security and governance review against OWASP LLM Top 10.
ProjectGoverned production deployment with audit trail, Langfuse and Arize AI observability live, and the repository tagged v1.0.
AI tools in this course.
Students treat AI assistants as a core part of the operations toolchain: generating Terraform, CI/CD, and Kubernetes config, refactoring service code, and drafting data-quality and drift checks. They wire LLM features through the course gateway, build RAG and extraction pipelines, and expose project tools as MCP servers that their own agents call under step, cost, and permission bounds. AI is used to generate evaluation sets, synthesize test data, and analyze RED dashboards, traces, and drift metrics into concrete actions. Because the course grades exactly what an assistant cannot do for a student, operating under load, reading telemetry, and defending decisions, every generated artifact is traced, evaluated, and reviewed before it ships.
Student project
Teams of three or four carry a chosen AI service from specification through governed production deployment, implementing all five operational layers: DevOps, DataOps, MLOps, LLMOps, and AgentOps. The project includes an Airflow-managed medallion data pipeline with Great Expectations quality gates and schema contracts, a GitHub Actions CI/CD pipeline with MLflow evaluation gating and version-controlled model promotion, a RAG service with Langfuse trace-level observability and token-cost dashboards, and a security and governance review against OWASP LLM Top 10. Teams present three times: a specification at week 5, a live working system at week 8, and a final governed deployment with oral defense at week 13.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Predictive-maintenance IoT monitorSupport-docs Q&A chatbotInvoice or form processorMenu or receipt nutrition estimatorSmart-home energy advisorCode-review assistantResearch-paper summariserChurn-prediction service
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- Docker: containerise services with pinned dependencies
- Kubernetes: orchestrate, scale, and self-heal services
- GitHub Actions: run CI/CD pipelines on every merge
- Terraform: provision infrastructure as code
- Prometheus & Grafana: collect metrics and build RED dashboards
- Apache Airflow: orchestrate data pipelines with retries and backfills
- DVC: version datasets and reproduce results
- Great Expectations: validate data quality as code
- Apache Kafka: stream events into the data lake
- MLflow: track experiments and run the model registry
- EvidentlyAI: monitor data and model drift
- LiteLLM: route LLM traffic through one gateway with fallbacks
- FAISS / Qdrant: vector search for retrieval-augmented generation
- Langfuse: trace and observe LLM and agent applications
- Model Context Protocol: expose tools to agents
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
In Hebrew · בעברית
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- TextbookDesigning Data-Intensive Applications
- TextbookSite Reliability Engineering
- TextbookThe Site Reliability Workbook
- DocumentationAWS Well-Architected Framework
- PaperHidden Technical Debt in Machine Learning Systems
- DocumentationMLflow Documentation
- PaperRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- DocumentationOWASP Top 10 for LLM Applications
- DocumentationBuilding Effective Agents
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Core · Semester 2 |
| Networking & Cyber Security | Core · Semester 2 |
| AI & Robotics | Core · Semester 2 |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Elective |