About this course
Process and fuse data from radar, electro-optical, infrared, and RF sensors using signal processing and estimation to build a coherent picture of the environment.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Built a multi-sensor estimation and tracking pipeline in Python with FilterPy and Stone Soup, implementing Kalman, unscented, and particle filters with data association to fuse radar, EO-IR, and inertial measurements into validated multi-target tracks with covariance-consistency checks.
Expected outcomes
- Analyze discrete-time signals and systems with sampling, the DFT, and digital filtering
- Formulate state estimation as recursive Bayesian inference over a state-space model
- Derive and implement the Kalman filter and prove its optimality for the linear Gaussian case
- Extend estimation to nonlinear systems with the extended and unscented Kalman filters
- Implement particle filtering for non-Gaussian and multimodal state estimation
- Fuse measurements from heterogeneous sensors into a single coherent state estimate
- Solve the data-association problem for multi-target tracking
- Characterize radar, EO-IR, and RF sensor models, noise, and measurement geometry
- Evaluate estimator performance with covariance consistency and error metrics
- Design and defend a complete sensor fusion and tracking system as a team project
Key topics
- Digital signal processing
- Kalman filtering & estimation
- Multi-sensor fusion
- Radar, EO/IR and RF
Theoretical foundations
The concepts and results this course rests on.
- the recursive Bayes filter as the unifying framework for state estimation
- the linear Gaussian state-space model and Kalman optimality in mean-square error
- linearization and the sigma-point unscented transform for nonlinear estimation
- sequential Monte Carlo, importance sampling, and resampling
- the data-association problem and multiple hypothesis tracking
- sampling theory, the discrete Fourier transform, and digital filter design
- estimator consistency, the covariance, and the Cramer-Rao lower bound
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Signals and systems or linear algebra
- Probability and statistics
- Programming in Python or C++
Weekly schedule 13 weeks · lecture + practice
Signal processing
Wk 1
Discrete-time signals and sampling
LectureSampling and the Nyquist criterion, discrete-time signals and systems, convolution, and the discrete and fast Fourier transforms.
PracticeSample, transform, and inspect real sensor signals in Python and identify aliasing and spectral content.
ProjectRepository, a sensor dataset, and a signal-inspection notebook.
Wk 2
Digital filtering and spectral estimation
LectureFIR and IIR filter design, the z-transform, frequency response, and power spectral density estimation for noisy measurements.
PracticeDesign and apply digital filters to denoise sensor streams and estimate their spectra.
ProjectFiltering and preprocessing front end for the sensor data.
Estimation
Wk 3
Probability and Bayesian estimation
LectureRandom variables, the Gaussian, conditional expectation, and the recursive Bayes filter as the foundation of all state estimation.
PracticeImplement a recursive Bayes filter on a discretized state space and visualize belief updates.
ProjectBayesian estimation baseline on a toy tracking problem.
Wk 4
The Kalman filter
LectureState-space models, the linear Gaussian assumption, and the derivation of the Kalman filter as the optimal recursive minimum-mean-square-error estimator from Kalman 1960.
PracticeImplement a Kalman filter for a constant-velocity target and check covariance consistency.
ProjectKalman tracker estimating state from noisy measurements.
Milestone
Wk 5
Specification presentationPresentation
LectureScoping a fusion project: sensors, state model, motion model, evaluation metrics, and technical risks. Rubric for the specification defense.
PracticeSTUDENT PRESENTATION milestone, specification. Teams present their estimation problem, the sensors and state-space model, the filtering approach, an evaluation plan with metrics, and a milestone plan.
ProjectApproved project specification and an estimation design.
Nonlinear estimation
Wk 6
Extended and unscented Kalman filters
LectureLinearizing nonlinear dynamics with the EKF and its Jacobians, and the sigma-point unscented transform of Julier and Uhlmann for better consistency.
PracticeImplement an EKF and a UKF for a nonlinear tracking model and compare their accuracy and consistency.
ProjectNonlinear filter handling the project motion model.
Wk 7
Particle filters
LectureSequential Monte Carlo, importance sampling, resampling, and the bootstrap particle filter of Gordon, Salmond, and Smith for non-Gaussian state estimation.
PracticeImplement a particle filter and apply it to a multimodal or non-Gaussian tracking problem.
ProjectParticle filter alternative evaluated against the Kalman variants.
Milestone
Wk 8
Interim demo presentationPresentation
LectureDemonstrating an estimation slice: showing a working single-sensor tracker with quantified accuracy and consistency.
PracticeSTUDENT PRESENTATION milestone, interim demo. Teams demo a working single-sensor filter on real or recorded data, report error and covariance-consistency metrics, and show estimated tracks against ground truth.
ProjectWorking single-sensor tracker with quantified performance.
Fusion
Wk 9
Multi-sensor data fusion
LectureCentralized versus decentralized fusion, measurement and track fusion, the information filter, and handling asynchronous and out-of-sequence measurements.
PracticeFuse two heterogeneous sensors into one state estimate and measure the accuracy gain.
ProjectMulti-sensor fusion improving the track estimate.
Wk 10
Data association and multi-target tracking
LectureThe data-association problem, nearest neighbor, probabilistic data association, and multiple hypothesis tracking from Reid for multiple targets in clutter.
PracticeAdd gating and data association to track multiple targets in cluttered measurements.
ProjectMulti-target tracker with association handling clutter.
Sensors
Wk 11
Radar, EO-IR, and RF sensor models
LectureRadar range-Doppler processing and detection, EO-IR imaging geometry, RF angle-of-arrival, and the noise and measurement models each sensor contributes to fusion.
PracticeModel a radar or EO-IR measurement source and integrate its detections into the fusion pipeline.
ProjectRealistic heterogeneous sensor models feeding the tracker.
Integration
Wk 12
Evaluation and robustness
LectureEstimator consistency with the normalized estimation error squared, the Cramer-Rao bound, track metrics, and robustness to dropout and bias.
PracticeRun a full evaluation with consistency tests and stress the system with sensor dropout and bias.
ProjectValidated, robust fusion system ready for the final defense.
Milestone
Wk 13
Final demo and oral defensePresentation
LectureCourse synthesis: from the Bayes filter to a deployed multi-sensor tracking system and the estimation tradeoffs that defined it.
PracticeSTUDENT PRESENTATION milestone, final demo with oral defense. Teams present the finished fusion system, walk through their filtering and association architecture, justify estimator choices with consistency evidence, and answer technical questions.
ProjectFinal sensor fusion and tracking system with documentation.
AI tools in this course.
Students use AI assistants and vibe-coding to implement and refactor estimators in Python, turning filter equations into NumPy and FilterPy code for Kalman, unscented, and particle filters and for data association. They drive notebooks, Stone Soup, and the dataset tooling through assistants and MCP servers, asking the model to set up a tracking scenario, generate synthetic sensor measurements and ground truth, and run batch experiments. AI helps generate unit tests for filter updates and clutter handling and to analyze covariance-consistency results such as the normalized estimation error squared. Students treat the model as a partner for deriving and checking estimator math, while verifying every result against consistency metrics and ground-truth tracks.
Student project
Each team builds one multi-sensor estimation and tracking pipeline across the term on real or realistically simulated sensor data. The project grows weekly from signal preprocessing and a single Kalman tracker to a fused multi-target system with data association, heterogeneous sensor models, and validated covariance consistency. The same artifact is presented at the specification, interim, and final milestones.
Requirements
- Build a working system, not a set of disconnected exercises.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Radar and camera fusion for vehicle trackingIMU and GPS fusion for navigationPedestrian multi-target tracker from lidarDrone tracking from fused RF and EO-IRIndoor localization from IMU and UWBMaritime target tracking from radar in clutterAcoustic and visual fusion for source localization
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, and architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with oral defense | 50% |
Tools & platforms
- Python with NumPy and SciPy: numerical estimation and DSP
- scipy.signal: digital filter design and spectral analysis
- FilterPy: Kalman and Bayesian filter implementations in Python
- MATLAB Sensor Fusion and Tracking Toolbox: estimation and tracking
- Stone Soup: open-source multi-target tracking framework
- GTSAM: factor-graph smoothing and estimation library
- ROS 2: sensor data transport and the robot ecosystem
- OpenCV: vision-based detection and measurement extraction
- Matplotlib: track and covariance visualization
- pandas: sensor log handling and alignment
- Jupyter: interactive estimation experiments
- rosbag and HDF5: recorded sensor dataset storage
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
In Hebrew · בעברית
- Campus IL (Ariel University)אלגוריתמי ניווט ושיערוך מקום
- Dr. Roi Yozevitch (YouTube)אריאל - אלגוריתמי ניווט
Primary literature
Seminal works to read for graduate-level depth.
References
Books and resources link to an online or publisher page.
- TextbookEstimation with Applications to Tracking and Navigation
- TextbookProbabilistic Robotics
- TextbookDiscrete-Time Signal Processing, 3rd Edition
- TextbookFundamentals of Radar Signal Processing, 3rd Edition
- CourseKalman and Bayesian Filters in Python
- DocumentationSensor Fusion and Tracking Toolbox Documentation
- Documentationscipy.signal Reference
- DocumentationFilterPy Documentation
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Elective |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Core · Semester 1 |