About this course
Model data that unfolds over time and learn to make sequential decisions under uncertainty.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Built an end-to-end forecasting and control agent in Python with PyTorch, statsmodels, and Gymnasium, coupling a neural sequence forecaster to a deep reinforcement-learning controller trained with DQN and PPO inside a simulated environment.
Expected outcomes
- Test temporal data for stationarity using ADF and KPSS tests, select lag orders from ACF/PACF plots and information criteria, and fit ARIMA models with seasonal decomposition
- Engineer lag and rolling-window features, apply walk-forward cross-validation with purged look-ahead bias, and evaluate probabilistic forecasts with proper scoring rules
- Build and train deep sequence models including RNNs, LSTMs, and temporal attention networks for multi-step forecasting tasks
- Formulate sequential decision problems as Markov decision processes and implement model-free RL agents including DQN, PPO, and SAC in simulation environments
- Balance exploration and exploitation using epsilon-greedy, UCB, and Thompson sampling strategies, including curiosity-driven intrinsic rewards
- Detect and adapt to concept drift using ADWIN and Page-Hinkley tests and deploy online learning pipelines with incremental model updating
Key topics
- Time-series forecasting
- Sequence models
- Reinforcement learning
- Decision under uncertainty
Theoretical foundations
The concepts and results this course rests on.
- stationarity, unit roots, and differencing: ADF and KPSS tests and seasonal differencing
- autocorrelation and partial autocorrelation: ACF/PACF plots and lag selection with AIC and BIC
- ARIMA and seasonal decomposition: AR, I, MA components; STL decomposition; additive and multiplicative seasonality
- Markov decision processes: states, actions, rewards, transition probabilities, discount factor, and return
- temporal difference learning and Q-learning: Bellman update, epsilon-greedy, experience replay, and DQN
- policy gradient theorem and actor-critic methods: REINFORCE, PPO, SAC, advantage estimation, and entropy regularization
- sequence modeling with RNNs and LSTMs: vanishing gradients, gating mechanisms, and seq2seq with temporal attention
- walk-forward cross-validation and data leakage: expanding window, blocked k-fold, and purging look-ahead bias
- exploration-exploitation trade-off: epsilon-greedy, UCB, Thompson sampling, and curiosity-driven intrinsic rewards
- online learning and concept drift adaptation: ADWIN, KSWIN, Page-Hinkley test, and incremental model updating
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Machine Learning
- Probability and statistics
- Linear algebra
Weekly schedule 13 weeks · lecture + practice
Time-series foundations
Wk 1
Stationarity and autocorrelation
LectureStationarity, unit roots, and the ADF and KPSS tests; seasonal differencing; ACF and PACF plots; lag selection with AIC and BIC.
Practicestatsmodels: apply ADF and KPSS tests, plot ACF/PACF, and select lag order on real financial or energy data.
ProjectLoad the team's temporal dataset, characterize its stationarity, and document the autocorrelation structure.
Wk 2
ARIMA and seasonal decomposition
LectureAR, I, MA components; ARIMA identification, fitting, and diagnostics; STL decomposition; additive vs. multiplicative seasonality.
Practicestatsmodels: fit ARIMA and run STL decomposition; tsfresh: extract lag and rolling-window features automatically from the time series.
ProjectEstablish statistical baselines and automated feature engineering on the project series.
Wk 3
Walk-forward validation and leakage prevention
LectureWalk-forward cross-validation; expanding and sliding windows; blocked k-fold; purging look-ahead bias; proper scoring rules for probabilistic forecasts.
PracticeProphet (Meta): fit a decomposable trend-and-seasonality model; implement a walk-forward harness and compare Prophet against ARIMA baselines.
ProjectBuild the project's walk-forward evaluation harness and lock the scoring protocol.
Wk 4
Experiment tracking and probabilistic forecasting
LectureProbabilistic forecasting: predictive distributions, prediction intervals, calibration, and CRPS; managing many forecasting experiments systematically.
PracticeWeights and Biases: instrument a forecasting training loop, compare ARIMA and Prophet runs, and visualize calibration and walk-forward plots.
ProjectLog all baseline experiments to W&B and prepare charts for the specification presentation.
Wk 5
Specification presentationPresentation
LectureMotivating neural sequence models: limitations of ARIMA and the case for learned temporal representations.
PracticeTeam presentation: problem framing, dataset characterization, statistical baselines, walk-forward protocol, and W&B dashboard.
ProjectLock the specification, evaluation protocol, and dataset; begin preparing the neural forecaster.
Sequential decision theory
Wk 6
Markov decision processes
LectureStates, actions, rewards, transition probabilities, discount factor, and return; value functions, Bellman equations, and policy evaluation.
PracticeGymnasium: build a custom environment around the project domain; implement and solve a tabular MDP with value iteration.
ProjectFormulate the project decision problem as an MDP and stand up a Gymnasium environment.
Wk 7
Temporal difference learning and DQN
LectureTemporal difference learning; Q-learning; Bellman update; epsilon-greedy exploration; experience replay; target networks; DQN and the deadly triad.
PracticeStable-Baselines3: train a DQN agent; MuJoCo / Brax: run a physics-based simulation environment and benchmark agent performance.
ProjectTrain an initial DQN agent on the project Gymnasium environment.
Wk 8
Policy gradient methodsPresentation
LecturePolicy gradient theorem; REINFORCE; advantage estimation; actor-critic; PPO clipped objective; SAC entropy regularization.
PracticeTeam presentation: MDP formulation, Gymnasium environment, DQN training curves, and an interactive demo of the learned policy.
ProjectImplement a PPO or SAC agent and compare with DQN across environment seeds.
Deep sequence models and adaptation
Wk 9
RNNs, LSTMs, and temporal attention
LectureRecurrent networks and backpropagation through time; vanishing gradients; LSTM and GRU gating; seq2seq with encoder-decoder attention; causal masking for long-horizon forecasting.
PracticePyTorch: implement an LSTM sequence-to-sequence forecaster and an attention-based decoder; compare walk-forward accuracy against ARIMA and Prophet baselines.
ProjectReplace the statistical baseline with an LSTM or attention forecaster; log results to W&B.
Wk 10
Exploration-exploitation trade-off
LectureEpsilon-greedy decay schedules; upper confidence bound (UCB); Thompson sampling; curiosity-driven intrinsic rewards; regret bounds.
PracticeStable-Baselines3 with MuJoCo / Brax: compare exploration strategies across environments; Ray RLlib: scale parallel rollouts across multiple seeds.
ProjectAdd principled exploration to the project agent and evaluate policy generalization across seeds.
Wk 11
Hyperparameter optimization
LectureBayesian optimization for hyperparameters; multi-fidelity search (Hyperband); population-based training; interaction between forecaster and agent hyperparameters.
PracticeOptuna: run a sweep over LSTM depth, learning rate, and RL reward-scale hyperparameters; Ray RLlib: distribute the RL sweep across workers.
ProjectRun Optuna sweeps over the forecaster and agent; push the best configs to the W&B dashboard.
Wk 12
Online learning and concept drift
LectureConcept drift taxonomy; ADWIN, KSWIN, and Page-Hinkley drift detectors; incremental model updating; coupling drift detection to agent retraining.
PracticePyTorch: implement incremental fine-tuning on a drifting stream; Ray RLlib: retrain the agent after a detected distribution shift in the environment.
ProjectIntegrate the forecast-driven decision pipeline with drift detection and adaptive retraining.
Capstone
Wk 13
Final defensePresentation
LectureCourse synthesis: from stationarity and ARIMA through deep sequence models to sequential decision agents; open research problems and industry directions.
PracticeTeam presentation: end-to-end demo comparing the statistical baseline, deep forecaster, and RL agent with rigorous walk-forward evaluation, plus oral defense of design choices.
ProjectDeliver the integrated pipeline with walk-forward evaluation across all three approaches and a full W&B report.
AI tools in this course.
Students use AI assistants to generate and refactor the statsmodels ARIMA baselines, the LSTM and temporal-transformer forecasters, and the Gymnasium environment and reward code, vibe-coding the DQN and PPO training loops. They prompt AI to synthesize simulated episodes and edge-case series, to wire reward shaping and replay buffers, and to set up Optuna sweeps. AI also helps read backtest plots and learning curves, explaining why a policy diverged or a forecast lost calibration.
Student project
Teams choose a real temporal dataset from finance, energy, sensor monitoring, or logistics, and build a full pipeline from a statistical baseline through a deep sequence forecaster to a sequential decision agent that acts on the forecast. All three approaches are evaluated side-by-side with rigorous walk-forward cross-validation.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Energy demand forecasting and storage controlInventory and supply-chain replenishmentAlgorithmic trading agentTraffic signal controlCloud autoscaling controllerRide-hailing fleet dispatchDynamic pricing agentSmart-grid load balancing
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- PyTorch: sequence model implementation
- statsmodels: ARIMA and classical time-series models
- sktime: unified time-series forecasting API
- Prophet: decomposable trend and seasonality forecasting
- Gymnasium: reinforcement-learning environments
- Stable-Baselines3: reference RL algorithm implementations
- Ray RLlib: scalable distributed RL
- NumPy: numerical computation
- pandas: time-series data handling
- Matplotlib: forecast and policy visualization
- Optuna: hyperparameter optimization
- Weights and Biases: experiment tracking
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeDeepMind x UCL: Introduction to Reinforcement Learning (David Silver)
- YouTubeBerkeley CS285: Deep Reinforcement Learning (Fall 2023)
- MIT OCWTime Series Analysis I (MIT 18.S096)
In Hebrew · בעברית
- Dr. Amos Azaria, Ariel University (YouTube)Reinforcement Learning 1 - למידה מונחית חיזוקים
- Ben-Gurion University (Campus IL)מבוא לבינה מלאכותית: מתאוריה לפרקטיקה
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- TextbookReinforcement Learning: An Introduction, 2nd edition
- TextbookForecasting: Principles and Practice, 3rd edition
- TextbookProbabilistic Machine Learning: An Introduction
- PaperPlaying Atari with Deep Reinforcement Learning
- PaperProximal Policy Optimization Algorithms
- TextbookDive into Deep Learning
- DocumentationRay Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 1 |
| AI and Quantum Computing for Finance | Core · Semester 1 |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Core · Semester 2 |