About this course
Study the models that generate images, text, and audio, from diffusion models to GANs and autoregressive generators.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Teams implement one generative model family from scratch (VAE or GAN) in PyTorch, train and evaluate it on a chosen image dataset using FID measurement, then fine-tune a pre-trained Stable Diffusion model for a custom domain using LoRA and evaluate the domain adaptation quality with Gradio demos.
Expected outcomes
- Derive the ELBO via Jensen's inequality and implement a VAE encoder-decoder with reparameterization trick, KL divergence, and reconstruction loss from scratch in PyTorch.
- Analyze GAN minimax training and Nash equilibrium, apply Wasserstein distance, gradient penalties, spectral normalization, and progressive growing to stabilize adversarial learning.
- Derive score matching and denoising score estimation, implement the DDPM noise schedule and DDIM accelerated sampling, and connect diffusion to score-based SDEs.
- Apply classifier-free guidance, negative prompts, ControlNet, and IP-Adapter for conditional generation, and fine-tune Stable Diffusion with LoRA for domain-specific image synthesis.
- Implement autoregressive generation with causal masking and teacher forcing, and evaluate generative models rigorously using FID, IS, precision-recall, and LPIPS.
- Address ethical responsibilities for generative systems, including deepfake detection, C2PA content provenance watermarking, synthetic data bias auditing, and disclosure obligations.
Key topics
- Diffusion models
- GANs & VAEs
- Autoregressive generation
- Evaluating generative output
Theoretical foundations
The concepts and results this course rests on.
- Jensen's inequality, the evidence lower bound, and amortized variational posterior inference
- VAE encoder-decoder architecture, the reparameterization trick, KL divergence, and reconstruction loss
- GAN minimax objective, the optimal discriminator, Nash equilibrium, and Wasserstein distance with gradient penalties
- Mode collapse, spectral normalization, progressive growing, and minibatch discrimination for GAN stabilization
- Stein score function, denoising score estimation, noise-conditional networks, and Langevin dynamics
- DDPM forward noising process, reverse denoising objective, DDIM accelerated sampling, and score-based SDE
- Classifier-free guidance scale, negative prompts, ControlNet spatial conditioning, and IP-Adapter image prompting
- Autoregressive left-to-right likelihood factorization, causal masking, teacher forcing, and exposure bias
- FID, Inception Score, precision-recall, and LPIPS as evaluation metrics for generative output quality and diversity
- Deepfake detection, C2PA content provenance watermarking, synthetic data bias, and disclosure obligations
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Deep Learning
- Probability and linear algebra
Weekly schedule 13 weeks · lecture + practice
Generative foundations
Wk 1
What is a generative model
LectureGenerative modeling as distribution learning: maximum likelihood, latent variable models, and the taxonomy of VAE, GAN, autoregressive, and diffusion model families (P1 overview).
PracticeSet up the PyTorch training harness and Hugging Face Hub workflow; load an image dataset, inspect its statistics, and push a first checkpoint to the Hub.
ProjectChoose the image dataset and define the generative task for the running project.
Wk 2
Latent variables and the ELBO
LectureJensen's inequality, the evidence lower bound, amortized posterior inference, and the KL-reconstruction decomposition (P1).
PracticeImplement variational inference on a toy latent-variable model in PyTorch; visualize the ELBO components during training.
ProjectEstablish a probabilistic baseline generator for the project dataset.
Variational models
Wk 3
Variational autoencoders
LectureVAE encoder-decoder architecture, the reparameterization trick, and the reconstruction-versus-KL trade-off (P2).
PracticeBuild a VAE from scratch in PyTorch; train on the project dataset and compute LPIPS perceptual similarity between reconstructions and originals.
ProjectDeliver a working VAE generator with latent-space interpolation visualizations.
Adversarial models
Wk 4
GAN minimax training and equilibrium
LectureGAN minimax objective, the optimal discriminator, Nash equilibrium, and the Jensen-Shannon and Wasserstein distance connections (P3).
PracticeTrain a conditional GAN from scratch in JAX / Flax on the project dataset; compare training dynamics with the PyTorch VAE baseline.
ProjectAdd a GAN generator to the project and compare sample diversity with the VAE.
Wk 5
GAN stabilization and specification reviewPresentation
LectureMode collapse, training instability, spectral normalization, progressive growing, and minibatch discrimination for stabilizing GAN training (P4).
PracticeTeam presentation: each team defends its generative model specification, dataset choice, and evaluation plan.
ProjectLock the project specification; apply at least one GAN stabilization technique and record the effect on sample quality.
Diffusion models
Wk 6
Score matching and denoising score estimation
LectureThe Stein score function, denoising score estimation, noise-conditional score networks, and Langevin dynamics sampling (P5).
PracticeUse Hugging Face Diffusers to run a pre-built diffusion pipeline; inspect scheduler configurations and compare Euler, PNDM, and DPM-Solver sampling.
ProjectIntegrate a Hugging Face Diffusers pipeline into the project and compare its samples with the VAE and GAN outputs.
Wk 7
DDPM, DDIM, and score-based SDE
LectureDDPM forward noising process and reverse denoising objective, DDIM accelerated sampling, and the unified score-based SDE framework (P6).
PracticeLoad Stable Diffusion as the base model via Hugging Face Diffusers; run text-to-image inference, modify the noise schedule, and compare step counts against quality.
ProjectAdopt Stable Diffusion as the diffusion backbone for the project domain.
Wk 8
Diffusion synthesisPresentation
LectureLatent diffusion models, the role of the VAE latent space in Stable Diffusion, and the unifying view across DDPM, score SDE, and flow matching.
PracticeTeam presentation: interim demo comparing VAE, GAN, and diffusion samples on the project dataset with qualitative and quantitative analysis.
ProjectDemonstrate working VAE-or-GAN and diffusion generators side by side with sample grids.
Conditional generation
Wk 9
Classifier-free guidance and conditioning
LectureGuidance scale, negative prompts, ControlNet spatial conditioning, and IP-Adapter image prompting for controllable diffusion generation (P7).
PracticeBuild and run a conditional generation workflow in ComfyUI; chain ControlNet with a depth or edge map to control composition.
ProjectAdd conditioning to the diffusion generator and demonstrate control over generated outputs.
Wk 10
LoRA fine-tuning and domain adaptation
LectureLoRA parameter-efficient fine-tuning for Stable Diffusion, domain-specific dataset preparation, and prompt engineering with negative prompts for quality control.
PracticeCall the DALL-E / Imagen API as a held-out baseline; compare API-generated outputs against the LoRA-fine-tuned model on the project domain.
ProjectFine-tune Stable Diffusion with LoRA on the project domain and compare with the from-scratch generators.
Autoregressive models
Wk 11
Autoregressive generation
LectureLeft-to-right likelihood factorization, causal masking, teacher forcing, and exposure bias in autoregressive image and sequence generators (P8).
PracticeTrain a small autoregressive pixel model on the project dataset and share the checkpoint to Hugging Face Hub with a model card.
ProjectAdd an autoregressive generator as a third family in the comparison and verify that the Hub-hosted checkpoint reproduces the results.
Evaluation
Wk 12
Evaluation metrics and deployment
LectureFID, Inception Score, precision-recall for generative diversity, and LPIPS perceptual similarity as complements to likelihood-based metrics (P9).
PracticeCompute FID and IS with torch-fidelity, clean FID with cleanfid, and LPIPS on all project generators; wrap the best model in a Gradio demo for interactive evaluation.
ProjectProduce a shared evaluation table across all model families; finalize the Gradio demo for the final presentation.
Ethics and capstone
Wk 13
Ethics of generative AI and final defensePresentation
LectureDeepfake detection methods, C2PA content provenance watermarking, synthetic data bias auditing, and disclosure obligations for deployed generative systems (P10).
PracticeTeam presentation: live Gradio demo with sample grids, FID and LPIPS evaluation table, LoRA domain-adaptation results, and oral defense of design choices and ethical safeguards.
ProjectDeliver the complete generative pipeline: from-scratch VAE or GAN, LoRA-fine-tuned Stable Diffusion, evaluation results, and deployed Gradio demo.
AI tools in this course.
Students use AI assistants to generate and refactor PyTorch VAE and GAN code, vibe-coding the DDPM noise schedule, reparameterization trick, and classifier-free guidance. They prompt AI to write LoRA fine-tuning scripts, Gradio demo wrappers, and evaluation harnesses using torch-fidelity and cleanfid. AI also helps interpret FID curves and sample grids, diagnosing mode collapse, posterior collapse, or guidance artifacts from quantitative and visual evidence.
Student project
Teams implement one generative model family from scratch (VAE or GAN) in PyTorch, train and evaluate it on a chosen image dataset with FID measurement, then fine-tune a pre-trained Stable Diffusion model for a custom domain using LoRA and evaluate the domain adaptation quality with a Gradio demo. The project is presented three times across the semester and culminates in a complete, deployable generative pipeline.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Conditional image synthesisText-to-image generationMolecular structure generationAnomaly detection via generative densityData augmentation generatorStyle transfer and image editingTabular synthetic-data generationDomain-adapted portrait or product generator
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Target roles
Graduates of this course are prepared for the following professional roles.
Generative Model Engineer
Implements and trains VAEs, GANs, and diffusion models from scratch in PyTorch, evaluates generation quality with FID and IS metrics, applies conditioning and guidance techniques, fine-tunes pre-trained diffusion models for domain-specific generation, and deploys generative inference pipelines with Gradio demos.
AI Content Generation Specialist
Adapts pre-trained generative models for creative and enterprise use cases using ControlNet and LoRA fine-tuning, engineers conditioning prompts and negative prompts for diffusion models, evaluates output quality and diversity, and integrates generative pipelines into product workflows with appropriate guardrails.
Tools & platforms
- Hugging Face Diffusers: diffusion pipeline implementation and inference
- ComfyUI: visual node-based workflow for conditional generation
- Stable Diffusion: pre-trained latent diffusion base model
- torch-fidelity: FID and Inception Score evaluation
- LPIPS: perceptual similarity metric
- Gradio: interactive generative demo deployment
- JAX / Flax: alternative functional framework for GAN experiments
- cleanfid: standardized clean FID computation
- Hugging Face Hub: model sharing and versioning
- DALL-E / Imagen API: API-based generation for baseline comparison
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeStanford CS236: Deep Generative Models (2023)
- YouTubeBerkeley CS294-158: Deep Unsupervised Learning (Spring 2024)
In Hebrew · בעברית
- Google Cloud (Coursera)Introduction to Generative AI - בעברית
- Dr. Amos Azaria, Ariel University (YouTube)Deep Learning and NLP - קורס למידה עמוקה ועיבוד שפות טבעיות
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- PaperDenoising Diffusion Probabilistic Models
- PaperGenerative Adversarial Networks
- PaperAuto-Encoding Variational Bayes
- PaperScore-Based Generative Modeling through Stochastic Differential Equations
- PaperHigh-Resolution Image Synthesis with Latent Diffusion Models
- TextbookProbabilistic Machine Learning: An Introduction
- TextbookDeep Learning
- DocumentationHugging Face Diffusers Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 2 |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Core · Semester 2 |
| Defense Technologies & Autonomous Systems | Elective |