About this course
Move from raw audio waveforms to intelligent speech and audio systems. The course builds a rigorous signal-processing foundation covering sampling theory, the Fourier transform, and spectral representations, then applies that foundation to speech feature extraction (MFCCs, pitch, formants), speaker recognition, automatic speech recognition (ASR) from HMMs through CTC to Whisper, and audio classification for environmental sound, music, and acoustic events. The final third of the course addresses real-time pipeline design, streaming inference, and low-latency deployment so that systems can operate on live audio in defense, robotics, and field-intelligence scenarios.
Course format. Thirteen weeks, four contact hours each: a two-hour lecture covering signal-processing theory and deep-learning architectures, and a two-hour practice session for hands-on Python coding with audio datasets. The course is project-based; teams carry one audio or speech intelligence project end to end and present it three times, in weeks 5, 8, and 13.
What you will build
Designed and delivered a complete end-to-end audio and speech intelligent system: ingested raw audio, extracted and normalized acoustic features, trained or fine-tuned a neural model for the target task (transcription, speaker verification, or acoustic event detection), evaluated it on a held-out test set with standard metrics, and deployed a real-time streaming inference pipeline with documented latency and accuracy trade-offs.
Expected outcomes
- Explain sampling theory, quantization, and the Discrete Fourier Transform, and compute spectrograms and mel-spectrograms from raw audio waveforms
- Extract MFCC, pitch, and formant features from speech signals and explain their acoustic and phonetic significance
- Design and implement a feature extraction pipeline for audio and speech data, including windowing, normalization, and data augmentation
- Distinguish voiced from unvoiced speech segments using energy, zero-crossing rate, and autocorrelation-based analysis
- Build a speaker recognition and speaker diarization system using embedding models and clustering
- Describe the architecture of an HMM-based ASR system including acoustic model, language model, and decoder
- Explain CTC loss and attention-based ASR architectures, and fine-tune a Whisper model for a target domain
- Train and evaluate an audio classification model for environmental sound, music genre, or acoustic event detection
- Implement a real-time audio processing pipeline with streaming inference, ring-buffer management, and low-latency constraints
- Deliver a complete end-to-end audio and speech intelligent system with documented architecture, evaluation metrics, and latency analysis
Key topics
- Sampling, quantization, and Fourier analysis
- Spectrograms and mel-spectrograms
- MFCCs and speech feature extraction
- Speaker recognition and diarization
- ASR: HMM, CTC, and Whisper
- Audio classification for sound and music
- Real-time audio pipeline design
Theoretical foundations
The concepts and results this course rests on.
- Nyquist-Shannon sampling theorem and the conditions for aliasing-free reconstruction from discrete samples
- Short-Time Fourier Transform (STFT) and the time-frequency uncertainty principle: the trade-off between temporal and spectral resolution
- Mel frequency scale and perceptual auditory models: the mel filter bank and its cochlear motivation
- Cepstral analysis: homomorphic filtering, the cepstrum, liftering, and the derivation of Mel-Frequency Cepstral Coefficients
- Hidden Markov Models: the three fundamental problems (evaluation, decoding, and training), the Viterbi algorithm, and Baum-Welch EM estimation
- Connectionist Temporal Classification (CTC): the forward-backward algorithm, the conditional independence assumption, and beam-search decoding
- Attention mechanisms and the transformer self-attention formulation for sequence-to-sequence audio modeling
Prerequisites
This is a Year-3 course. It assumes the mandatory CS core: data structures and algorithms, operating systems, computer networks, databases, software engineering, and the core mathematics (linear algebra, probability and statistics, calculus, discrete mathematics). It additionally requires the specific prior courses listed below.
Course-specific prerequisites:
- Signal and systems or digital signal processing fundamentals
- Linear algebra and Fourier analysis
- Python programming and familiarity with NumPy and PyTorch
- Machine learning fundamentals (classification, neural networks, overfitting)
Weekly schedule 13 weeks · lecture + practice
Foundations
Wk 1
Audio signal fundamentals
LectureDigital audio representation: sampling rate, quantization depth, and bit rate. The Nyquist-Shannon theorem and aliasing. Discrete-time signals and systems: convolution, impulse response, and FIR filters. Energy and power of audio signals.
PracticeLoad audio files with librosa, visualize waveforms, apply a low-pass FIR filter, and verify aliasing artifacts at sub-Nyquist rates.
ProjectRepository established, audio dataset selected, and initial listening and waveform analysis of the target audio domain completed.
Wk 2
Time-frequency analysis and spectrograms
LectureThe Discrete Fourier Transform and the Fast Fourier Transform. Short-Time Fourier Transform: windowing functions (Hann, Hamming), hop length, and the time-frequency resolution trade-off. Magnitude and phase spectra. Power spectrograms and the decibel scale. The mel scale and mel-spectrograms.
PracticeCompute and visualize STFT spectrograms for speech, music, and noise with different window sizes; generate mel-spectrograms and observe the frequency warp.
ProjectBaseline spectrogram visualization of the project audio dataset completed and documented.
Wk 3
MFCC and speech feature extraction
LectureCepstral analysis and homomorphic filtering. The MFCC pipeline: pre-emphasis, windowing, mel filter bank, log compression, and the Discrete Cosine Transform. Delta and delta-delta coefficients. Pitch detection: autocorrelation, YIN algorithm, and pYIN. Formant analysis and Linear Predictive Coding.
PracticeExtract MFCCs with librosa, visualize feature trajectories over time, and implement pitch tracking with pYIN on a set of vowel recordings.
ProjectFeature extraction pipeline for the project implemented and benchmark features computed on the full dataset.
Wk 4
Voiced/unvoiced classification and data augmentation
LectureVoiced versus unvoiced speech: energy, zero-crossing rate, autocorrelation, and spectral flatness measures. Voice activity detection (VAD). Data augmentation for audio: time stretching, pitch shifting, additive noise, SpecAugment. Building robust feature extraction pipelines with normalization and frame-level labeling.
PracticeImplement a VAD using energy and ZCR thresholds, apply SpecAugment to a mel-spectrogram batch, and measure the effect of augmentation on a baseline classifier.
ProjectData augmentation strategy defined and applied to the training split; VAD pre-processing integrated into the feature pipeline.
Milestone
Wk 5
Specification presentationPresentation
LectureScoping an audio and speech intelligence project: task definition, dataset selection, evaluation metrics, system architecture, and risk analysis.
PracticeSTUDENT PRESENTATION milestone, specification. Teams present the target task, dataset, feature pipeline, model architecture plan, evaluation protocol, and expected latency constraints.
ProjectApproved project specification with dataset, feature pipeline, architecture plan, and evaluation protocol.
Speaker and speech models
Wk 6
Speaker recognition and diarization
LectureSpeaker verification versus speaker identification. i-vector and x-vector embedding models. Cosine similarity and PLDA scoring. Speaker diarization: segmentation, embedding extraction, agglomerative clustering, and overlap handling. Evaluation: Equal Error Rate, DER, and JER.
PracticeFine-tune a pretrained x-vector model from SpeechBrain on a speaker dataset, compute EER on a held-out verification set, and run a simple two-speaker diarization experiment.
ProjectCore model component implemented and first evaluation numbers recorded for the specification baseline.
Wk 7
ASR: HMM-based acoustic modeling
LectureThe automatic speech recognition pipeline: acoustic model, pronunciation lexicon, and language model. HMMs for phoneme modeling: the three fundamental problems, the Viterbi decoder, and the Baum-Welch algorithm. Gaussian Mixture Model and DNN acoustic models. The Kaldi toolkit architecture.
PracticeDecode a small corpus with a pretrained Kaldi model, inspect the WFST decoding graph, and measure Word Error Rate on a standard benchmark set.
ProjectBaseline model producing first evaluation results; project architecture validated against the specification.
Milestone
Wk 8
Interim demo presentationPresentation
LecturePresenting an audio intelligence system in progress: live demo on a short audio clip, evaluation metrics, and remaining architectural decisions.
PracticeSTUDENT PRESENTATION milestone, interim demo. Teams demonstrate the working baseline system processing real audio, report current evaluation metrics, and describe the plan for weeks 9 to 13.
ProjectWorking baseline system with quantitative evaluation results and a revised plan for the final system.
End-to-end and classification models
Wk 9
End-to-end ASR: CTC and attention-based models
LectureConnectionist Temporal Classification: the CTC loss, the forward-backward algorithm, and beam-search decoding. Attention-based encoder-decoder models for speech. Whisper: architecture, multitask training, multilingual capability, and fine-tuning on custom domains. Comparing WER across model families.
PracticeFine-tune a Whisper model on a domain-specific dataset using Hugging Face Transformers, evaluate WER before and after fine-tuning, and test on out-of-distribution accents.
ProjectEnd-to-end neural model integrated and delivering improved evaluation metrics over the baseline.
Wk 10
Audio classification: environmental sound, music, and events
LectureAudio classification as an image classification problem: CNN on mel-spectrograms. Pretrained audio models: VGGish, PANNs, and Audio Spectrogram Transformer (AST). Transfer learning and few-shot audio classification. Evaluation: accuracy, macro F1, mAP for multi-label tasks. Datasets: ESC-50, UrbanSound8K, AudioSet, GTZAN.
PracticeFine-tune a PANNs model on ESC-50, visualize class activation maps on spectrogram inputs, and evaluate on a custom holdout set.
ProjectClassification or detection component trained and evaluated; confusion matrix analysis completed to guide remaining refinements.
Wk 11
Real-time audio pipelines and streaming inference
LectureReal-time audio I/O: ring buffers, callback-based processing, and latency sources. Streaming STFT and online feature extraction. Streaming ASR architectures: incremental decoding, chunked attention, and CTC prefix beam search. Latency metrics: algorithmic latency, model latency, and total end-to-end latency. Low-latency deployment with PyAudio and TorchAudio streaming.
PracticeBuild a real-time keyword spotter: read microphone input with PyAudio, extract features online, run inference, and measure end-to-end latency at different chunk sizes.
ProjectReal-time inference pipeline implemented and latency measured; streaming and batch results compared in the evaluation report.
Integration
Wk 12
System integration and evaluation
LectureEnd-to-end system integration: combining pre-processing, model inference, post-processing, and output formatting. Error analysis for audio and speech systems: substitution, deletion, and insertion analysis; confusion matrices; hard-sample mining. Model compression: pruning, quantization, and knowledge distillation for edge deployment.
PracticeProfile the end-to-end pipeline, identify the bottleneck component, apply INT8 quantization to the neural model, and measure accuracy and latency before and after.
ProjectComplete integrated system; final evaluation report drafted with error analysis and latency profiling, ready for the defense.
Milestone
Wk 13
Final demo and oral defensePresentation
LectureCourse synthesis: from audio waveform to deployed intelligent system, and the engineering and modeling choices that shaped it.
PracticeSTUDENT PRESENTATION milestone, final demo with oral defense. Teams demonstrate the finished audio and speech intelligent system on live or pre-recorded audio, walk through their architecture and evaluation strategy, present latency profiling and error analysis, and answer technical questions on design decisions.
ProjectFinal system with source code, evaluation report, latency analysis, and documentation.
AI tools in this course.
Students use AI assistants to accelerate audio pipeline development: generating librosa and torchaudio boilerplate for feature extraction, writing data-loading and augmentation code, and drafting evaluation scripts with standard metrics. AI tools help interpret evaluation results by suggesting error-analysis strategies and pointing to relevant dataset subsets for debugging. Students use AI to explore architecture choices (CTC versus attention versus Whisper) by generating comparison code and summarizing trade-offs from the literature. All AI-generated pipeline code must be validated against a held-out test set; students verify that reported metrics are reproducible before accepting any suggestion.
Student project
Each team designs and builds one audio or speech intelligent system across the term, progressing from a raw audio dataset through feature extraction and model training to a real-time inference pipeline. The project grows weekly from basic acoustic analysis through a trained neural model to a complete system with streaming capability, latency profiling, and error analysis. The same artifact is presented at the specification, interim, and final milestones.
Requirements
- Build a working end-to-end system, not a notebook-only experiment.
- Be original: a real audio intelligence problem, not a direct reproduction of a tutorial or dataset baseline.
- Show real depth: end-to-end from raw audio to evaluated output, with a real-time streaming component and documented latency.
- Carry one running project from specification to a deployed, defensible result across the whole term.
- Work in a team of three or four and defend the system at each of the three presentations (weeks 5, 8, and 13).
Example projects
Speaker-verified transcription system with Whisper and x-vector verificationAcoustic anomaly detector for industrial machinery with real-time alertingMulti-speaker meeting diarization and transcript alignment pipelineMusic genre classifier using mel-spectrogram CNNs and Audio Spectrogram TransformerReal-time voice activity detector and keyword spotter with edge deploymentEnvironmental sound event detector for urban surveillance audioLow-latency streaming ASR for field-intelligence transcription on a Raspberry Pi
Assessment & grading
Grading is project-based, with no written exam. Teams of three or four present one running system three times.
| Component | What it covers | Weight |
|---|---|---|
| Project · Specification | Presentation 1 (week 5): task definition, dataset, feature pipeline, architecture plan, and evaluation protocol | 20% |
| Project · Interim | Presentation 2 (week 8): working baseline system with quantitative evaluation results and a revised plan | 30% |
| Project · Final | Presentation 3 (week 13): complete integrated system with real-time pipeline, evaluation report, latency analysis, and oral defense | 50% |
Tools & platforms
- librosa: Python library for audio analysis, feature extraction (MFCCs, spectrograms, pitch), and time-frequency transforms
- PyTorch Audio (torchaudio): neural audio processing, streaming data pipelines, and GPU-accelerated feature computation
- Whisper: OpenAI general-purpose ASR model for transcription, translation, and fine-tuning on custom domains
- SpeechBrain: open-source speech processing toolkit for speaker recognition, ASR, and diarization
- Hugging Face Transformers: pretrained audio models including Wav2Vec2, HuBERT, and Whisper with fine-tuning pipelines
- PyAudio: real-time audio I/O via the PortAudio library for microphone capture and streaming playback
- SoundFile: reading and writing audio files in WAV, FLAC, and OGG formats
- scikit-learn: classical ML classifiers, dimensionality reduction, and evaluation metrics for audio feature experiments
- Git: version control for audio pipeline code, model checkpoints, and evaluation results
Free online courses
Existing free, video-based courses this course can build on, for self-study or as a teaching basis.
- YouTubeAudio Signal Processing for Machine Learning (The Sound of AI)
- StanfordCS224S: Spoken Language Processing
- CourseraAudio Signal Processing for Music Applications (UPF Barcelona)
In Hebrew · בעברית
- Campus ILעיבוד אותות ולמידת מכונה לאודיו
Primary literature
Seminal works for advanced study.
References
Books and resources link to an online or publisher page.
- TextbookFundamentals of Speech Recognition
- TextbookSpeech and Language Processing, 3rd Edition (draft)
- TextbookAudio Source Separation and Speech Enhancement
- Documentationlibrosa Documentation
- DocumentationSpeechBrain Tutorials
Role in each concentration
| Concentration | Role |
|---|---|
| Intelligent Software Systems | Elective |
| Networking & Cyber Security | Elective |
| AI & Robotics | Elective |
| AI and Quantum Computing for Finance | Elective |
| Immersive Systems & Game Development | Elective |
| Defense Technologies & Autonomous Systems | Core · Semester 2 |