About this course
Bring AI into the physical world through robot middleware, sensor fusion, SLAM, motion planning, behavior trees, model-based control, and sim-to-real transfer for mobile manipulation systems.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
An autonomous mobile robot system in Gazebo and ROS2: the robot localizes with ORB-SLAM3, navigates to goal positions while avoiding dynamic obstacles with Nav2, executes a manipulation task with MoveIt2, and demonstrates sim-to-real transfer to a higher-fidelity environment in NVIDIA Isaac Sim.
Expected outcomes
- Architect and configure a ROS2 middleware stack with pub/sub topics, service calls, action servers, and lifecycle node management for a complete robot system.
- Fuse IMU, LiDAR, and camera data using EKF and UKF for robust state estimation, and build SLAM pipelines with ORB-SLAM3 and LIO-SAM across feature-based, direct, and graph-based approaches.
- Represent environments as probabilistic occupancy grids and configure Nav2 costmap layers; plan global and local paths with RRT, PRM, A*, and TEB/MPC trajectory optimization.
- Sequence autonomous missions with behavior trees in py_trees and design model-based controllers (PID, LQR, MPC) to track planned trajectories.
- Evaluate and close the sim-to-real gap using domain randomization and system identification, and implement safety-critical watchdog and fail-safe behaviors for physical deployments.
- Process 3D point clouds with PCL (voxelization, ICP, segmentation) and integrate perception with MoveIt2 manipulation planning in a full perceive-plan-act pipeline.
Key topics
- ROS2 middleware & node architecture
- Sensor fusion & SLAM
- Motion planning & navigation
- Behavior trees & model-based control
- Sim-to-real transfer
- Point cloud processing & manipulation
Theoretical foundations
The concepts and results this course rests on.
- Robot middleware architecture: publish/subscribe decoupling, synchronous service calls, long-running action servers, parameter management, and ROS2 lifecycle node state machine
- Sensor fusion for state estimation: IMU, LiDAR, and camera integration; extended Kalman filter (EKF) and unscented Kalman filter (UKF) derivation and implementation
- Simultaneous localization and mapping: feature-based (ORB-SLAM3), direct, and graph-based pose optimization with loop closure and bundle adjustment
- Occupancy grids and probabilistic map representations: log-odds Bayesian update, ray casting, and Nav2 costmap inflation layers
- Motion planning: sampling-based algorithms (RRT/PRM), graph-search algorithms (A*/Dijkstra), and trajectory optimization (TEB, receding-horizon MPC)
- Behavior trees for hierarchical task sequencing: tick execution model, selector and sequence composites, condition nodes, and action nodes
- Model-based control: PID tuning, LQR optimal control with cost matrices (Q, R), and model-predictive control with receding horizon optimization
- Sim-to-real transfer: domain randomization, system identification, sim-to-real gap quantification, and hardware deployment validation
- Safety and fault tolerance in physical autonomous systems: watchdog timers, fallback behaviors, and fail-safe modes for lifecycle management
- Point cloud processing and 3D scene understanding: voxelization, ICP scan registration, surface normal estimation, and Euclidean cluster segmentation
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Machine Learning and basic reinforcement learning
- Linear algebra
- Calculus and basic physics
Weekly schedule 13 weeks · lecture + practice
ROS2 Foundations
Wk 1
Robot middleware architecture
LectureRobot middleware architecture: decoupled pub/sub topics, synchronous service calls, long-running action servers, parameter management, and lifecycle node state machine in ROS2.
PracticeLaunch a multi-node system in ROS2: implement publishers, subscribers, a service server, an action server, and lifecycle node transitions; inspect topic graphs with rqt.
ProjectInitialize the team ROS2 workspace; define the robot node graph, topic schema, and message interfaces for the project.
Estimation & SLAM
Wk 2
Sensor fusion: EKF and UKF
LectureSensor fusion for state estimation: IMU, LiDAR, and camera integration; derivation of the extended Kalman filter (EKF) and unscented Kalman filter (UKF) for nonlinear systems.
PracticeSpawn a differential-drive robot in Gazebo, attach IMU and LiDAR plugins, and implement an EKF node that fuses noisy odometry with IMU measurements.
ProjectIntegrate sensor drivers for the project robot in Gazebo and collect calibration data to tune the EKF state estimator.
Wk 3
Simultaneous localization and mapping
LectureSLAM approaches: feature-based, direct, and graph-based pose optimization with loop closure; data association, pose graph construction, and bundle adjustment.
PracticeConfigure ORB-SLAM3 in ROS2 monocular and RGB-D modes; run it on a recorded bag file and visualize the 3D map and trajectory in RViz2.
ProjectIntegrate ORB-SLAM3 into the project robot stack and produce a map of the Gazebo simulation environment.
Wk 4
Occupancy grids and navigation costmaps
LectureProbabilistic map representations: log-odds Bayesian update, LiDAR ray casting, and Nav2 costmap layers (static, obstacle, inflation); map server and AMCL localization.
PracticeBuild a LiDAR-inertial map with LIO-SAM on a ROS2 bag, then load it into Nav2 and inspect costmap inflation layers and AMCL particle cloud in RViz2.
ProjectGenerate the occupancy map for the project environment and tune Nav2 costmap parameters for the robot footprint and obstacle clearance.
Wk 5
Sensor stack and SLAM reviewPresentation
LectureReview of sensor fusion, SLAM approaches, and probabilistic map representations; discussion of trade-offs between ORB-SLAM3 and LIO-SAM in different environments.
PracticeTeam presentation 1: each team defends its robot specification, sensor stack, SLAM pipeline, and costmap configuration with live RViz2 demos.
ProjectLock the perception and mapping specification; receive feedback and refine the SLAM and EKF integration before the motion planning phase.
Planning & Behaviors
Wk 6
Motion planning algorithms
LectureMotion planning: sampling-based planners (RRT, RRT*, PRM), graph-search planners (A*, Dijkstra), and trajectory optimization with TEB and receding-horizon MPC; configuration space and obstacle avoidance.
PracticeImplement and benchmark RRT and A* path planners on rigid-body robots in PyBullet; compare path length, computation time, and obstacle clearance across environments.
ProjectImplement and tune the global planner for the project robot and validate collision-free paths through the Gazebo environment.
Wk 7
Behavior trees for task sequencing
LectureBehavior trees for hierarchical task sequencing: tick execution model, selector and sequence composites, condition and action nodes; comparison with finite state machines; integration with ROS2 action servers.
PracticeAuthor a multi-goal navigation mission in py_trees: design a tree with patrol, docking, and recovery subtrees, and wire it to the Nav2 NavigateToPose action server.
ProjectDefine the robot mission logic as a behavior tree and integrate with the Nav2 navigation stack; implement at least one recovery behavior subtree.
Control
Wk 8
Model-based controlPresentation
LectureModel-based control: PID tuning, LQR optimal control with cost matrices Q and R, and model-predictive control with receding horizon optimization; trajectory tracking for differential-drive and arm robots.
PracticeTeam presentation 2: live demo of the robot localizing with ORB-SLAM3, navigating with Nav2, and executing a behavior tree mission; set up a MoveIt2 motion planning group for the arm after the presentation.
ProjectTune the Nav2 local planner with a model-based controller; configure a MoveIt2 planning group for the manipulation task and verify basic arm motion.
Sim-to-Real
Wk 9
Sim-to-real transfer
LectureSim-to-real transfer: domain randomization of sensor noise, friction, mass, and lighting; system identification methods; quantifying the sim-to-real gap; deployment testing on physical or higher-fidelity hardware.
PracticeRe-run the navigation and SLAM pipeline in NVIDIA Isaac Sim with randomized physics parameters; measure localization and navigation error compared to Gazebo baselines.
ProjectApply domain randomization to sensor noise and dynamics in Isaac Sim; document the sim-to-real gap and identify which parameters most affect transfer quality.
Wk 10
Safety and fault tolerance
LectureSafety and fault tolerance in physical autonomous systems: watchdog timers, ROS2 lifecycle node deactivation, fallback behavior subtrees, fail-safe modes, and hazard detection for mobile robots.
PracticeImplement ROS2 watchdog nodes that monitor heartbeat topics and trigger lifecycle node deactivation; inject faults into the sensor and navigation stack and verify fallback recovery.
ProjectAdd a safety monitor node and recovery behavior tree to the project robot; stress-test with simulated sensor dropouts and navigation failures.
Perception & Manipulation
Wk 11
Point cloud processing and 3D scene understanding
LecturePoint cloud processing and 3D scene understanding: voxelization, ICP scan registration, surface normal estimation, Euclidean cluster segmentation, and integration with robot perception pipelines.
PracticeProcess a LiDAR point cloud with PCL in ROS2: voxel-grid downsample, register two scans with ICP, estimate surface normals, and segment candidate grasp objects by cluster size and height.
ProjectIntegrate a PCL-based object detection module into the robot stack; publish detected object poses as ROS2 geometry_msgs for the manipulation planner.
Wk 12
Manipulation and full-stack integration
LectureIntegration of perception, SLAM, navigation, manipulation, and safety into a complete autonomous system; pipeline orchestration, message latency, and diagnostic tooling for end-to-end debugging.
PracticePlan and execute a pick-and-place sequence with MoveIt2: receive an object pose from the PCL detector, plan a Cartesian approach path, and execute the grasp action while Nav2 holds the base position.
ProjectComplete end-to-end integration: the robot navigates to a goal with Nav2, detects an object with PCL, and executes the manipulation task with MoveIt2.
Capstone
Wk 13
Final defensePresentation
LectureSynthesis of robot middleware, sensor fusion, SLAM, motion planning, behavior trees, model-based control, and sim-to-real transfer; open problems in embodied AI and safe autonomous systems.
PracticeTeam presentation 3: full end-to-end live demo (localize, navigate, avoid obstacles, manipulate) with task metrics; oral defense of architecture, parameter choices, and sim-to-real results.
ProjectDeliver the complete autonomous mobile robot system with documented sim-to-real transfer results and quantitative task success, safety, and robustness metrics.
AI tools in this course.
Students use AI assistants to generate and refactor ROS2 node boilerplate, Nav2 parameter YAML files, MoveIt2 planning scene configurations, and ORB-SLAM3 integration launch files; they prompt AI to synthesize EKF noise covariance matrices, behavior tree XML structures, and PCL pipeline code for voxelization and ICP. AI helps wire message interfaces between Nav2 action clients and py_trees action nodes, debug TF2 transform trees, and explain why costmap inflation or a SLAM loop closure failed. Students also use AI to interpret RViz2 trajectory logs, analyze sim-to-real gap metrics from Isaac Sim runs, and draft safety watchdog recovery logic.
Student project
Teams design and implement an autonomous mobile robot system in Gazebo and ROS2. The robot must localize with ORB-SLAM3, navigate autonomously with Nav2 to goal positions while avoiding dynamic obstacles, and complete a manipulation task with MoveIt2. The project concludes with a sim-to-real transfer demonstration to a higher-fidelity environment in NVIDIA Isaac Sim, validating that the system generalizes beyond the Gazebo training environment.
Requirements
- Build a working integrated system, not a set of disconnected exercises: SLAM, navigation, behavior trees, and manipulation must form one running pipeline.
- Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
- Show real depth: quantitative task success metrics, safety validation under injected faults, and a documented sim-to-real transfer study.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).
Example projects
Autonomous mobile robot: SLAM, Nav2, and MoveIt2 pick-and-placeWarehouse logistics robot with dynamic obstacle avoidanceIndoor delivery robot with autonomous navigation and manipulationMobile manipulation: navigate, detect with PCL, and pick with MoveIt2Search-and-rescue robot with 3D LiDAR mapping and path planningAutonomous inspection robot with LIO-SAM and behavior tree mission
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running project three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): problem, objectives, sensor stack, and SLAM architecture | 20% |
| Project · Interim | Presentation 2 (week 8): the working localization, navigation, and behavior tree system demonstrated live | 30% |
| Project · Final | Presentation 3 (week 13): end-to-end demo with manipulation, sim-to-real transfer results, and oral defense | 50% |
Tools & platforms
- ROS2: robot middleware for pub/sub topics, service calls, action servers, and lifecycle node management
- Gazebo: physics-based robot simulation for sensor emulation and system integration testing
- NVIDIA Isaac Sim: GPU-accelerated high-fidelity simulation for sim-to-real transfer validation
- Nav2: ROS2 navigation stack for autonomous mobile robot navigation with costmaps and local planners
- MoveIt2: motion planning framework for robot manipulation and arm trajectory execution
- ORB-SLAM3: feature-based visual SLAM for localization and mapping with cameras
- PCL (Point Cloud Library): 3D point cloud processing: voxelization, ICP registration, normal estimation, and segmentation
- py_trees: Python behavior tree library for hierarchical robot task sequencing
- PyBullet: fast rigid-body physics simulation for motion planning algorithm development and benchmarking
- LIO-SAM: tightly-coupled LiDAR-inertial SLAM system for accurate outdoor and indoor mapping
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- UniversityRobotic Manipulation (MIT 6.4210/6.4212), Russ Tedrake
- UniversityUnderactuated Robotics (MIT 6.832), Russ Tedrake
In Hebrew · בעברית
- Ariel University (Campus IL)רובוטים אוטונומיים (Autonomous Robots)
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- TextbookProbabilistic Robotics
- TextbookPlanning Algorithms
- TextbookReinforcement Learning: An Introduction, 2nd edition
- PaperProximal Policy Optimization Algorithms
- TextbookProbabilistic Machine Learning: An Introduction
- DocumentationROS 2 Documentation
- TextbookDive into Deep Learning
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Core · Semester 2 |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Core · Semester 2 |
| Defense Technologies & Autonomous Systems | Core · Semester 2 |