HIT · CS Concentrations

COURSE · GM2

Spatial Computing

מחשוב מרחבי

spatial computing, pose tracking, and immersive interaction across virtual and augmented reality

Build and defend an immersive VR or AR application

Year 313 weeks2h lecture + 2h practiceProject-based

About this course

Build spatial and immersive applications: 3D tracking and scene understanding, virtual and augmented reality, natural interaction, and spatial UX across XR devices.

Course format. Thirteen weeks, four contact hours each: a two-hour lecture (concepts and theory) and a two-hour practice session. The course is project-based; teams carry one running project end to end and present it three times, in weeks 5, 8, and 13.
What you will build

Built a cross-device spatial computing application on OpenXR with the Unity XR Interaction Toolkit, implementing hand-tracked interaction, SLAM-based inside-out tracking, plane detection and spatial anchors, and a comfort-tuned locomotion system that meets the device motion-to-photon latency budget.

Expected outcomes

  • Explain the optical and perceptual basis of head-mounted displays and stereoscopic rendering
  • Represent rigid-body pose with SE(3) transforms and apply them to head and hand tracking
  • Build cross-device XR applications against the OpenXR runtime and input model
  • Implement spatial mapping and tracking using SLAM and visual-inertial odometry concepts
  • Design immersive interaction techniques for selection, manipulation, and locomotion
  • Apply human-factors guidelines to reduce cybersickness and respect comfort and latency budgets
  • Anchor virtual content to the real world using plane detection and spatial anchors in AR
  • Evaluate immersive user experience with presence and usability measures
  • Compare VR and AR device capabilities, tracking systems, and rendering constraints
  • Design and defend a complete spatial computing application as a team project

Key topics

  • VR/AR & OpenXR
  • Tracking & spatial mapping (SLAM)
  • Natural interaction & UX
  • On-device performance

Theoretical foundations

The concepts and results this course rests on.

  • the SE(3) rigid-body transform group and quaternion rotation representation
  • stereoscopic rendering and the optics and perception of head-mounted displays
  • simultaneous localization and mapping and visual-inertial odometry
  • epipolar geometry and multiple-view geometry for tracking and reconstruction
  • the registration problem and Azuma's defining criteria for augmented reality
  • sensory conflict theory and the motion-to-photon latency budget for comfort
  • presence and the perceptual factors that sustain immersion

Prerequisites

This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.

Course-specific prerequisites:

  • Computer Graphics or 3D graphics basics
  • Linear algebra
  • Computer vision basics

Weekly schedule 13 weeks · lecture + practice

Foundations
Wk 1
Spatial computing and the XR landscape
LectureFrom Sutherland's head-mounted display to modern VR and AR. Display optics, field of view, stereoscopy, and the latency and frame-rate requirements for comfort.
PracticeSet up the XR toolchain and deploy a hello-world scene to a headset or AR-capable device.
ProjectRepository, device target, and a deployed XR scene rendering in stereo.
WatchVR at 50: Ivan Sutherland's 1968 head-mounted display (ACM SIGGRAPH)
Wk 2
Pose, frames, and SE(3)
LectureRigid-body transforms, rotation representations with quaternions and matrices, and the SE(3) group. Coordinate frames for world, head, and controllers.
PracticeTrack and visualize head and controller poses, and convert between coordinate frames.
ProjectPose tracking and frame visualization in the running app.
WatchRigid body transformations and rotation matrices (UMD)
Platform
Wk 3
OpenXR and the input model
LectureThe OpenXR architecture: instances, sessions, swapchains, reference spaces, and the action-based input abstraction across vendors.
PracticeWire up OpenXR action sets for buttons, grips, and poses, and run on more than one runtime.
ProjectCross-runtime input layer driving the application.
WatchOpenXR: state of the union, Khronos GDC 2019
Interaction
Wk 4
Hand tracking and interaction primitives
LectureArticulated hand tracking, ray and direct interaction, grab and throw physics, and the design space of 3D selection and manipulation.
PracticeImplement ray-based and direct-grab interaction with hand or controller tracking.
ProjectCore interaction toolkit integrated into the scene.
WatchUnity and Meta SDK: a first hand-tracked interaction
Milestone
Wk 5
Specification presentationPresentation
LectureScoping an XR project: device target, interaction model, comfort budget, and technical risks. Rubric for the specification defense.
PracticeSTUDENT PRESENTATION milestone, specification. Teams present their application concept, target device and runtime, interaction and locomotion design, a comfort and latency budget, and a milestone plan.
ProjectApproved project specification and interaction storyboard.
Tracking
Wk 6
SLAM and visual-inertial tracking
LectureSimultaneous localization and mapping from PTAM and ORB-SLAM, feature tracking, and visual-inertial odometry with IMU preintegration for robust pose.
PracticeRun a SLAM or VIO pipeline and inspect tracked features, the map, and drift behavior.
ProjectInside-out tracking understood and instrumented in the project.
WatchIntroduction to SLAM (Cyrill Stachniss) · SLAM course playlist (Cyrill Stachniss)
Wk 7
Spatial mapping and anchors
LectureDense reconstruction from KinectFusion, plane detection, mesh-based spatial mapping, and persistent spatial anchors for world-locked content.
PracticeDetect planes and place anchored virtual content that stays fixed as the user moves.
ProjectWorld-anchored content for AR or a mapped play space for VR.
WatchKinectFusion: real-time 3D reconstruction (Microsoft Research)
Milestone
Wk 8
Interim demo presentationPresentation
LectureDemonstrating an XR vertical slice: what a credible interim build shows for tracking, interaction, and comfort.
PracticeSTUDENT PRESENTATION milestone, interim demo. Teams demo a working slice with tracked interaction and anchored content on the target device, and report measured latency and comfort observations.
ProjectPlayable vertical slice running on the target headset or device.
Experience
Wk 9
Immersive UX and locomotion
LectureLocomotion techniques, teleport versus continuous motion, diegetic UI, and spatial audio. Presence and the factors that build or break it.
PracticeAdd a locomotion system and a spatial UI, and evaluate them with a comfort checklist.
ProjectLocomotion and UI layer refining the experience.
WatchMel Slater: presence and the illusions of virtual reality · Continuous versus teleport locomotion in VR
Wk 10
Comfort, latency, and cybersickness
LectureSensory conflict theory, motion-to-photon latency, foveated and reprojection techniques, and design rules that reduce cybersickness.
PracticeMeasure motion-to-photon latency and apply comfort mitigations, then run a small user test.
ProjectComfort-tuned build with measured latency.
WatchContinuous versus teleport locomotion and comfort in VR
Augmented reality
Wk 11
AR rendering and occlusion
LectureOptical versus video see-through, registration and the Azuma survey definition of AR, depth occlusion, lighting estimation, and passthrough rendering.
PracticeAdd real-world occlusion and lighting estimation so virtual objects blend with the scene.
ProjectBelievable AR compositing or refined VR rendering.
WatchAR realism with the ARCore Depth API (Google)
Polish
Wk 12
Performance and multi-user
LectureXR performance budgets, single-pass stereo rendering, and the basics of shared and colocated multi-user spatial sessions.
PracticeOptimize the render path to the device frame budget and add an optional shared or multi-user feature.
ProjectFeature-complete build ready for the final defense.
WatchAdvanced VR rendering (Alex Vlachos, Valve)
Milestone
Wk 13
Final demo and oral defensePresentation
LectureCourse synthesis: from SE(3) and SLAM to a deployed spatial computing application and the human-factors tradeoffs that shaped it.
PracticeSTUDENT PRESENTATION milestone, final demo with oral defense. Teams present the finished application live on device, walk through their tracking and interaction architecture, justify comfort and performance choices, and answer technical questions.
ProjectFinal spatial computing application with documentation.
AI tools in this course.

Students use AI assistants and vibe-coding to write and refactor OpenXR session and input code, generate interaction and locomotion scripts, and wire up Unity XR Interaction Toolkit components from natural-language intent. They interact with engine tooling and device SDKs through MCP servers that surface the headset build, the OpenXR runtime, and on-device profilers, asking the model to wire action sets or place spatial anchors and then explain the tracking behavior. AI generates test scenes, scripted pose sequences, and synthetic IMU and tracking data for repeatable comfort and latency experiments. Students use the model to analyze motion-to-photon measurements and user-test results and to reason about cybersickness mitigations, always validating suggestions against on-device behavior.

Student project

Each team builds one immersive VR or AR application across the term against OpenXR on a real headset or AR-capable device. The project grows weekly from a tracked stereo scene to a complete experience with hand interaction, spatial mapping, anchored content, locomotion, and a met comfort and latency budget. The same artifact is presented at the specification, interim, and final milestones.

Requirements

  • Build a working system, not a set of disconnected exercises.
  • Be original: a new system that solves a real problem, not a re-implementation of a tutorial or course demo.
  • Show real depth: real data, real users or realistic load, and engineering trade-offs that are measured rather than assumed.
  • Carry one running project from specification to a deployed, defensible result across the whole term.
  • Work in a team of three or four and defend the design at each of the three presentations (weeks 5, 8, and 13).

Example projects

VR training simulator for a hands-on procedureAR maintenance assistant overlaying step instructionsRoom-scale escape-room puzzle gameAR indoor navigation with anchored waypointsCollaborative multi-user design review spaceVR data visualization walkthroughMixed-reality tabletop strategy game

Assessment & grading

Grading is project-based, with no written exam. Teams of three or four present one running project three times.

ComponentWhat it coversWeight
Project · SpecificationPresentation 1 (week 5): problem, objectives, and architecture20%
Project · InterimPresentation 2 (week 8): the working system demonstrated live30%
Project · FinalPresentation 3 (week 13): end-to-end demo with oral defense50%

Tools & platforms

Free online courses

Existing free, video-based courses this course can build on, for self-study or as a teaching basis.

Primary literature

Seminal works to read for graduate-level depth.

References

Books and resources link to an online or publisher page.

Role in each concentration

ConcentrationRole
Intelligent Software SystemsElective
Networking & Cyber SecurityElective
AI & RoboticsElective
AI and Quantum Computing for FinanceElective
Immersive Systems & Game DevelopmentCore · Semester 2
Defense Technologies & Autonomous SystemsElective
← GM1 · Computer GraphicsDF1 · Sensor Fusion & Signal Processing →