Ali-hey-0/audio-ai-field-guide

What makes this different from a course transcript: every section here is organized around why, not just what. Instead of "here's how Whisper works," you get "here's the decision Whisper's architects made, here's the alternative they rejected, and here's when you'd actually want that alternative instead." If you've internalized one Transformer architecture already (from NLP or CV), this guide is written to make the transfer explicit rather than re-teaching attention from scratch.

Before any unit-level detail, internalize this branching pipeline. Nearly every audio ML system on Earth is one of these two directions:

                              ┌───────────────────────────┐
                              │          AUDIO AI          │
                              └─────────────┬───────────────┘
                                            │
                  ┌─────────────────────────┴─────────────────────────┐
                  ▼                                                    ▼
       Audio → Model → Text / Label                         Text → Model → Audio
      ────────────────────────────────                     ────────────────────────
       ANALYTICAL / UNDERSTANDING                            GENERATIVE / SYNTHESIS
       • ASR (Speech → Text)                                 • TTS (Text → Speech)
       • Audio Classification                                 • Audio Generation
       • Speaker Recognition                                  • Voice Cloning
       • Emotion Recognition

And underneath both branches, the same five-stage skeleton repeats with only the edges changing:

Raw Signal → Preprocessing → Feature Extraction → Transformer Backbone → Task Head → Prediction

If you already know NLP Transformers (BERT/GPT), here's the direct mapping — this is the single most useful analogy in the entire field:

~80% of what you already know transfers directly. The only genuinely new machinery is how raw signal becomes a sequence of vectors in the first place — everything downstream of that (self-attention, encoder/decoder split, task heads, fine-tuning strategy) is the same story, just with audio nouns instead of text nouns.

pip install datasets[audio]

from datasets import load_dataset
dataset = load_dataset("PolyAI/minds14")   # bank call-center intents, the course's running example
example = dataset[0]

The critical, easy-to-miss insight: the audio column is not a file path — it's a lazy, smart feature type. Every sample decomposes into a consistent shape:

Sample
  ├── audio
  │     ├── path            → file location on disk
  │     ├── array            → THE WAVEFORM: [0.02, -0.03, 0.11, ...]
  │     └── sampling_rate    → e.g. 16000  (16,000 samples per second)
  ├── transcription          → human-labeled text  (ASR ground truth)
  └── intent / label / speaker / emotion   → task-specific target

Example row from MINDS-14:

The intent field itself is stored as an integer class ID (e.g. 13), not a readable string — dataset.features["intent_class"].int2str(13) converts it back to "PAY_BILL". This pattern (store as int, decode with a helper) is standard across Hugging Face classification datasets, not MINDS-14-specific.

The detail that trips up people coming from NLP/CV: there is no equivalent of "just load the image and look at it." The model — and you, when debugging — only ever sees audio["array"], a plain 1-D float array. The WAV file itself is never touched after decoding.

Sample → Audio → Waveform → Label

Only the label changes across tasks:

This is why Hugging Face's Audio feature type is task-agnostic — the exact same loading, decoding, and resampling code works whether you're about to train Whisper or a speaker-recognition model. The infrastructure layer doesn't care what you're predicting.

Real-world datasets arrive at mismatched sampling rates:

Audio A → 16 kHz,  2s
Audio B → 44.1 kHz, 8s
Audio C → 48 kHz,  15s

Feed these directly into a model and it will misbehave — most speech Transformers (Whisper, Wav2Vec2) are trained at 16 kHz, and frequency characteristics at other rates don't match what the model actually learned. Some models will simply refuse the input; others will silently degrade.

The fix — cast the column, and the library resamples lazily on access:

from datasets import Audio
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

Old Audio → Re-read → Resample → New Audio     (happens transparently on every access)

Verify it worked by checking sampling_rate before/after — 44100 → 16000 confirms success. The sample count also changes proportionally (44,100 samples/sec of audio → 16,000 samples/sec of the same audio), but total playback duration and pitch are unaffected.

Non-obvious point worth memorizing: resampling changes how many numbers represent one second of sound — it does not change speed or pitch. This is the single most common conceptual confusion for people new to digital audio: resampling ≠ time-stretching. A 2-second clip is still 2 seconds after resampling; it just now has fewer (or more) numbers describing it.

Why mismatched rates actively hurt a model, not just "look wrong":

• Frequency features the model learned to recognize occur at different absolute sample positions than expected.
• Performance degrades even when the audio content is identical.
• Some architectures reject the input outright rather than degrading gracefully.

The course unit focuses on resampling alone, but production pipelines typically add:

• ✅ Resampling to the model's expected rate
• ✅ Loudness/volume normalization
• ✅ Silence trimming
• ✅ Noise reduction
• ✅ Padding or truncation to a fixed length (for batching)
• ✅ Feature extraction — Mel Spectrogram or MFCC, if the architecture needs hand-crafted features rather than learning its own

Audio File → Read Audio → Check Sampling Rate → Resample (e.g. → 16 kHz) → Ready for Model

dataset = load_dataset("...", streaming=True)

The problem this solves: imagine a 2 TB dataset. The standard flow —

Download everything → Store on disk → Start processing

— fails when you don't have 2 TB free, when downloading alone takes days, or when you only actually need a few thousand samples for a quick experiment.

What streaming does instead:

Dataset → request sample → download just that sample → process → next sample

Why random access breaks under streaming — the mechanical reason: the library has no index into a stream it hasn't consumed yet. It genuinely does not know where sample #10,000 begins until it has read up to that point — the dataset behaves like an iterator, not an array. This is the same fundamental constraint you'd hit with any generator-based or network-streamed data source; it isn't audio-specific, but it bites especially often in audio because files are large enough that streaming is frequently the right default.

When to use streaming:

• ✅ The dataset is very large
• ✅ Disk space is constrained
• ✅ You only need a single pass (e.g., one training run)

When not to:

• ❌ The dataset is small
• ❌ You need repeated, indexed access to specific samples
• ❌ You'll run many experiments against the same data (re-streaming repeatedly wastes bandwidth)

• 🤗 datasets is the standard loading interface; pip install datasets[audio] enables audio decoding.
• Every sample is a dict; the audio field always decomposes into path, array, sampling_rate.
• The model sees array only — never the file. Everything else (transcription, intent, speaker, emotion) is a label.
• Resampling changes sample count, not playback speed — always match the model's expected rate (commonly 16 kHz).
• cast_column("audio", Audio(sampling_rate=16000)) is the standard, lazy way to resample.
• Streaming (streaming=True) trades random access for near-zero storage cost — mandatory above a few hundred GB, unnecessary below it.
• This unit is foundational: preprocessing, feature extraction, and every model you'll meet later all build directly on this sample structure.

Nearly every audio AI product decomposes into one of seven task families. The organizing question: the input is (almost) always audio — only the output category changes.

Audio
   │
   ├── Speech Recognition (ASR)        → Text
   ├── Audio Classification            → Class label
   ├── Speaker Recognition             → Speaker ID
   ├── Speech Emotion Recognition      → Emotion label
   ├── Text To Speech (TTS)            → Speech          [input is Text, not Audio]
   ├── Audio Generation                → Novel waveform  [input is a Prompt, not Audio]
   └── Speech Translation              → Text/Speech in another language

Goal: convert spoken language into text.

🎤 "Hello everyone"  →  📝 "Hello everyone"

Flagship models: Whisper, Wav2Vec2. Applications: automatic captioning, voice assistants, voice-to-text typing, call-center transcription.

Goal: identify what a sound is — output is a label, not text.

Audio → "Dog Bark"  |  "Gunshot"  |  "Music"  |  "Rain"

Goal: identify who is speaking.

Audio → "Ali"  or  "Unknown Speaker"

Applications: voice-based authentication, security, identifying who spoke in meeting transcripts.

Goal: identify the speaker's emotional state.

Audio → Happy | Angry | Sad

Heavily used in call-center analytics and conversation-quality monitoring.

Goal: the inverse of ASR — text goes in, speech comes out.

"Welcome" → 🔊 (synthesized speech audio)

Applications: audiobooks, dubbing, voice assistants, GPS navigation voices.

Goal: produce audio from a prompt — no text needs to be spoken, and the output need not be speech at all.

"Epic cinematic music" → 🎵 (generated music)
"Ocean waves"          → 🌊 (generated ambient sound)

Notable models: MusicGen, AudioGen, Bark (for some use cases), Stable Audio.

Goal: combine recognition and translation — speech in one language becomes text or speech in another.

Persian Speech → Text → Translate → English Speech
German Speech  → English Text

Whisper natively supports some of this; SeamlessM4T is purpose-built for it.

Why generation is harder than recognition — the asymmetry that shows up everywhere in ML: ASR only needs to guess what was already said from a fixed, existing signal. Audio generation must synthesize, from nothing, every acoustic property that makes sound convincing — frequency content, rhythm, amplitude envelope, timing, harmonics, and moment-to-moment continuity. There's no ground-truth signal to anchor against during inference, only a prompt. This "analysis is easier than synthesis" asymmetry is general across ML, but it's unusually visible in audio because humans have extremely fine-grained perceptual priors for what "natural" sound is — a synthetic voice with a 2% timing error is often immediately noticeable in a way a 2% error in, say, an image classifier's confidence score never would be.

Audio → Preprocessing → Feature Extraction → Neural Network → Prediction

Only Prediction changes:

The practical payoff: because this skeleton is shared, tooling, preprocessing code, and even a large fraction of intuition transfer freely between tasks. Learn one pipeline well (ASR with Whisper is the natural first choice) and the others become "same skeleton, different head" rather than new subjects.

This is genuinely the first point in the curriculum where you run a model rather than just inspecting data.

Without a pipeline, you'd have to do all of this yourself:

Load Audio → Resample → Feature Extraction → Load Model → Inference → Decode → Text

That's roughly 30–50 lines of code. transformers.pipeline() collapses it to:

from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="...")
result = pipe(audio)
# {"text": "My name is Ali."}

What happens behind the scenes on every call:

Audio File → Read Audio → Resample (if needed) → Processor → Model (Whisper/Wav2Vec2) → Decoder → Text

This exact pattern — Input → Pipeline → Output, with only the pipeline type and model changing — repeats across every audio task:

Where pipeline() stops being the right tool: it's excellent for prototyping, learning, and quick tests, but production systems typically drop down to the Processor + Model primitives directly, for finer control over batching, GPU placement, and custom decoding logic. Treat pipeline() as a scaffold you graduate away from, not a production dependency.

Audio File
     │
     ▼
  Pipeline
     ├── Read Audio
     ├── Resample
     ├── Processor
     ├── Model
     ├── Decoder
     ▼
   Text

• Seven task families cover nearly all of Audio AI: ASR, Classification, Speaker Recognition, Emotion Recognition, TTS, Audio Generation, Speech Translation.
• The differentiator between tasks is output type, not input type (with TTS/Generation as the input-side exception).
• TTS reads exact text aloud; Audio Generation synthesizes novel content from a prompt — don't conflate them.
• Generation is intrinsically harder than recognition/analysis — nothing to anchor against at inference time.
• pipeline() is the fastest path to running any of these models, but drop to Processor + Model for production control.

This is the conceptual keystone of the whole field. Internalize this unit and every subsequent architecture — ASR, TTS, classification — reads as "here's how we adapt this same backbone," not as new material.

Pre-Transformer audio models processed sequentially:

Sample 1 → Sample 2 → Sample 3 → ...

This was slow (no parallelism) and weak at long-range dependencies (information decays as it propagates through the recurrence). Transformers instead examine the whole sequence simultaneously via self-attention — exactly the same argument that displaced RNNs in NLP, ported over unchanged.

This is the real question this unit answers, and the pipeline has four concrete stages:

Waveform → Feature Extractor → Embeddings → Transformer Encoder → Task Head → Prediction

Stage 1 — Waveform. Raw floats: [0.01, 0.03, -0.2, ...]. Meaningless to a Transformer as-is — no structure a self-attention layer could exploit yet.

Stage 2 — Feature Extractor. The model does not work directly on the raw waveform.

Waveform → Feature Extractor → Feature Vector

• Classical approach: hand-engineered features — MFCC or Mel Spectrogram.
• Modern approach: models like Wav2Vec2 replace this entirely with a learned CNN — the model discovers its own useful features rather than relying on hand-crafted signal processing. This mirrors the broader ML trend of replacing feature engineering with learned representations, just applied to acoustics instead of pixels or n-grams.

Stage 3 — Embedding. Exactly like NLP: each frame becomes a high-dimensional vector.

Frame → 768 numbers

Stage 4 — Transformer Encoder.

Embedding → Self-Attention → Encoder → Contextual Representation

This is structurally identical to what happens inside BERT.

How Self-Attention helps, concretely: in the sentence "Please transfer money," correctly parsing "transfer" may require attending to both the start and end of the sentence. Attention learns that dependency directly. In audio, the exact same mechanism lets the model attend to frames hundreds of milliseconds — or several seconds — earlier, which matters enormously for anything where meaning spans a long acoustic context (e.g. disambiguating a word based on intonation established seconds earlier).

Stage 5 — Task Head. This is the only stage that varies by task:

                    ┌── ASR             → Characters → Sentence
Transformer Encoder ├── Classification  → Label
                    └── Speaker ID      → Speaker embedding/class

• Encoder's job: understand the input. Audio → Meaning
• Decoder's job: produce the output. Meaning → Text, generated step by step.

The master key for reading any new audio model: before reading anything else about an unfamiliar model, ask "does it have a decoder?" That single fact predicts most of its capability profile — CTC-style direct prediction vs. flexible autoregressive generation, translation capability, relative speed, and typical use case.

Audio
  │
  ▼
Waveform
  │
  ▼
Feature Extraction
  │
  ▼
Embeddings
  │
  ▼
Transformer
  │
  ▼
Representation
  │
  ▼
Task Head
  │
  ▼
Prediction

• ✅ Parallel processing — all frames processed simultaneously, not one-at-a-time.
• ✅ Long-range dependencies — attention doesn't decay with distance the way recurrence effectively does.
• ✅ Transfer learning — pretrain once on millions of hours of audio; fine-tune cheaply per downstream task. This is the economic argument underpinning the entire modern paradigm: the backbone is reusable capital, and only the head or a lightweight fine-tune is task-specific marginal cost.

If you already know BERT or GPT, roughly 80% of this unit's concepts are already familiar. The only real delta:

NLP:   Text     → Tokens              → Embeddings → Transformer
Audio: Waveform → Features/Embeddings              → Transformer

The base architecture is nearly identical; what changes is how raw data becomes vectors a Transformer can consume.

1. Transformer is the backbone of nearly every modern audio model.
2. The Transformer's input is never the raw WAV file — it's a vector representation (features/embeddings) of it.
3. Self-attention lets the model relate distant parts of the audio signal to each other.
4. Most models share one Transformer backbone and differ only in their task-specific head (ASR, Classification, Speaker ID, etc.).
5. The key architectural fork: Wav2Vec2 is Encoder-only; Whisper is Encoder–Decoder. This single fact predicts most of a model's behavior.

You have 2 seconds of audio → 32,000 samples. You have one output: "Hello".

Which samples correspond to which letters? You don't know — and virtually no real dataset provides that granularity. Training would seem to require:

0.15s → H
0.28s → e
0.41s → l

but datasets almost never contain this level of timing annotation. All you actually have is:

Audio → "Hello"

This is the alignment problem, and it's the reason ASR isn't "just" per-frame classification. Two architectural answers exist, representing a genuine and recurring trade-off in sequence modeling broadly.

Core idea: stop requiring frame-level time alignment entirely. The model predicts something for every frame — including a special blank token meaning "no output here" — then two simple deterministic rules collapse that into final text.

The model's raw, per-frame output:

Frame 1 → H
Frame 2 → H
Frame 3 → <blank>
Frame 4 → e
Frame 5 → <blank>
Frame 6 → l
Frame 7 → l
Frame 8 → <blank>
Frame 9 → o

Decode rule 1 — collapse consecutive repeats:

H H → H

Decode rule 2 — remove all blanks:

H <blank> e <blank> l l <blank> o → Hello

Why the blank token is the real insight, not an implementation detail: without it, the model has no way to distinguish "I predicted H twice because it's genuinely two separate H sounds" from "I predicted H across three consecutive frames because one H sound happens to span three frames." The blank token breaks that ambiguity by explicitly marking "not yet decided / no new character right now," which is what makes rule 1 (collapse repeats) safe to apply.

A simple intuition for the whole mechanism:

Imagine telling someone: "Whenever you're unsure, write nothing (blank). Whenever you're confident, write that letter — if you write the same letter several times in a row, we'll merge them later." That's the entire philosophy of CTC. It's why CTC can train from nothing more than (Audio, Transcript) pairs, with zero need for per-character timing labels.

Architecture:

Waveform → CNN Feature Extractor → Transformer Encoder → Linear Layer → Character Probabilities → CTC Decoder → Text

Why Wav2Vec2 uses CTC:

• ✅ No large decoder needed
• ✅ Simpler training
• ✅ Faster
• ✅ Well-suited specifically to transcription (as opposed to translation or paraphrase)

The hard limitation: CTC assumes the output order exactly mirrors the input order (monotonic alignment). It cannot reorder the way translation requires:

Audio → "Hello World"   (output order must match input order)

This makes CTC unsuitable for translation, summarization, or free-form text generation — any task that requires reordering or content transformation, not just transcription.

Core idea: instead of one output per frame, compress the entire input into a meaning representation, then generate the output autoregressively — one token at a time — with no requirement that output order mirrors input order.

Encoder: Audio → Meaning Representation
Decoder: Meaning → H → He → Hel → Hell → Hello    (generated step by step)

Why the decoder matters: at each generation step it decides "what word should come next?" — which is what makes reordering possible:

Input:  "Je m'appelle Ali"
Output: "My name is Ali"

The model isn't forced to preserve French word order to produce correct English.

Attention inside the decoder: at each generation step, the decoder asks "which part of the input is most relevant to the token I'm producing right now?" — e.g., generating the word "computer" might draw attention to a specific segment of the input audio.

Transformer Seq2Seq architecture (what Whisper actually is):

Audio → Audio Encoder → Hidden Representation → Text Decoder → Generated Text

Example: German speech in → English text out. Or Persian speech in → Persian text out — Whisper handles both transcription and translation through the same architecture, differing only in what the decoder is prompted/trained to produce.

A simple intuition for the two approaches, side by side:

A one-line summary worth memorizing:

CTC says: "map sound directly to letters." Seq2Seq says: "understand the meaning of the sound, then generate the right text."

Why the field moved toward Seq2Seq over time: real-world use cases go beyond plain transcription — simultaneous translation, conversation summarization, response generation, cross-language rendering, and general text improvement all require exactly the flexibility CTC structurally lacks.

Audio → Preprocessing → Feature Extraction → Speech Model → Decoder → Text

1. Audio input. Raw samples [0.02, 0.05, -0.01, ...], typically normalized to a consistent sampling rate (e.g. 16 kHz) and length before reaching the model.

2. Feature extraction — two competing approaches:

3. The model itself — the two families you now recognize:

A) CTC-based (Wav2Vec2):
Audio → Encoder → Character Probabilities → CTC Decoder → Text

B) Encoder-Decoder (Whisper):
Audio → Encoder → Representation → Decoder → Text
   (decoder generates step-by-step: <start> → I → I want → I want to → I want to book)

4. Decoding — turning probabilities into final text. The model doesn't output text directly; it outputs probabilities per position:

H: 0.8
A: 0.1
B: 0.05

The decoder must find the best overall path through these probabilities:

5. Evaluation — see Unit 5 for the full metric breakdown; the headline metric is WER (Word Error Rate).

1. ASR = converting Speech → Text.
2. Two dominant modern architectures: CTC (e.g. Wav2Vec2) and Encoder–Decoder (e.g. Whisper).
3. Modern models increasingly learn their own audio representation instead of relying on handcrafted features like MFCC.
4. The decoder is what actually produces or selects the final text output.
5. WER (Word Error Rate) is the primary evaluation metric.
6. The model is only one factor — data quality, background noise, accent, and language resource availability all materially affect real-world ASR performance.

By the end of this unit, you've effectively understood the backbone of systems like Whisper, Google Speech-to-Text, Azure Speech, and most production call-transcription systems.

The core idea: rather than training a speech-recognition model from scratch on thousands of hours of data, use an existing pre-trained model and fine-tune it on your own data only if needed.

Pre-trained Model → Fine-tuning (if needed) → Custom ASR System

Training ASR from zero requires: thousands or millions of hours of audio, matching transcripts, substantial GPU resources, and long training time. Models like Whisper have already absorbed enormous amounts of this cost — reusing that investment is almost always the correct starting point.

The Wav2Vec2 self-supervision insight, worth dwelling on: most of the field's progress over the last several years has come from decoupling "learn what speech sounds like" (cheap — unlabeled audio is abundant) from "learn to map speech to this specific label" (expensive — requires transcripts). Whisper took the opposite bet: enormous weakly labeled data instead of formal self-supervision. Both bets paid off — which tells you there isn't one correct scaling recipe, only different trade-offs between data cost, label quality, and compute budget.

Audio → Processor → Tensor → Model → Prediction

Pretrained Model + Your Dataset → Fine-tuned Model

Why this beats training from scratch: the model has already learned language structure, acoustic patterns, and pronunciation — fine-tuning only needs to specialize it.

The analogy the source material uses, and it's a good one: fine-tuning a large pre-trained model is like training a general practitioner into a specialist, rather than educating a new doctor from zero.

The bigger-isn't-always-better trade-off, made concrete:

For offline or resource-constrained systems, you're always balancing accuracy vs. latency vs. memory — there's no universally correct choice, only the right choice for your deployment constraints.

1. Pre-trained ASR models are already trained on huge data volumes and ready to use directly.
2. Key models: Wav2Vec2 (Encoder + CTC, lighter, ASR-focused) · Whisper (Encoder–Decoder, strong multilingual) · SpeechT5 / SeamlessM4T (more multi-task oriented).
3. In Hugging Face you typically work with three components: Pipeline (simplest) → Processor (input prep) → Model (the network).
4. Fine-tuning = adapting a ready-made model to your specific dataset, not building from zero.
5. In real projects, model choice isn't accuracy alone — speed, memory, hardware, and use case all matter.

The engineering framing worth internalizing: in real projects, the best strategy usually isn't "pick the biggest model." It's Pre-trained Model + real target-domain data + smart fine-tuning = a genuinely useful ASR system.

The core question: which dataset should you use to train or evaluate a speech-to-text model? Many people assume the model is the most important variable — in ASR, data quality and composition usually matter more.

An ASR model only learns what it has seen in training data:

The framing worth internalizing: the dataset is the experience the model gets of the real world. Every gap in the dataset becomes a gap in deployed performance — and unlike a model architecture choice, a data gap is invisible until it fails in production.

Every sample needs two things:

Audio + Transcript

e.g. audio_001.wav paired with "Where is the nearest bank?" — teaching the model this sound → this text.

1. Audio quality. Check sampling rate, noise level, microphone quality, file format.

2. Transcript quality. The paired text must be genuinely accurate — and "accurate" is itself a judgment call:

Audio: "I wanna go home"
Transcript option A: "I want to go home"   (normalized)
Transcript option B: "I wanna go home"     (verbatim)

Which is correct depends on your goal — for general-purpose ASR, the verbatim spoken form is usually the better training target.

3. Diversity. A good dataset spans:

• Speakers — male, female, varied ages
• Accents — multiple regional/national accents
• Recording conditions — quiet room, street noise, phone calls

4. Volume — but quality beats quantity. More data is generally better, but:

1,000 hours of poor-quality audio can be worse than 100 hours of very high-quality audio.

If you're customizing a general model (e.g. Whisper) for a specific domain (e.g. Persian customer-support calls), your fine-tuning data should resemble the actual deployment conditions, not a clean studio approximation of them:

❌ Bad:  Studio Speech        → Customer Call Model
✅ Good: Real Customer Calls  → Customer Call Model

Standard split:

Dataset
  ├── Train        (80%)  — model learns
  ├── Validation    (10%)  — tune hyperparameters/model selection
  └── Test          (10%)  — final, honest evaluation

Data leakage — a genuinely audio-specific trap: the same speaker must not appear in both train and test splits.

❌ Train: Ali speaking     Test: Ali speaking

If this happens, the model may simply memorize Ali's voice rather than learning general ASR — your test accuracy will look great and be meaningless. This is subtler than the equivalent NLP leakage risk, because speaker identity is a strong, easily-overfit signal that has nothing to do with the transcription task itself.

The challenge is structurally larger. Common mitigations:

• Leverage multilingual datasets
• Fine-tune large pre-trained models rather than training from scratch
• Data augmentation — synthetically expanding what you have:
  • Adding noise
  • Changing speed
  • Changing pitch

1. An ASR dataset is fundamentally Audio + Transcript pairs.
2. Dataset quality directly determines model quality — arguably more than architecture choice.
3. Speaker diversity, accent diversity, and recording-condition diversity are all critical.
4. The dataset should resemble the model's actual deployment environment, not an idealized studio version of it.
5. Data must be split into Train/Validation/Test — and speaker identity is a classic, easy-to-miss leakage vector in audio specifically.
6. For low-resource languages, Pre-trained Model + Fine-tuning is usually the most viable path.

Having covered two architectures for turning speech into text (CTC, Seq2Seq), this unit addresses a fundamentally different question:

"What is this sound?" — not "what was said?"

Goal: take an audio file and assign it a label.

🔊 audio.wav → "Dog Bark" | "Speech" | "Music" | "Engine Noise"

Audio Waveform → Feature Extractor → Audio Encoder → Classification Head → Class Label

Stage 1 — Feature conversion. Most architectures can't work on raw waveform directly:

Waveform → Mel Spectrogram   (or)   Waveform → Learned Features

Stage 2 — Encoder. The model learns task-relevant acoustic features. For dog-bark detection: bark-frequency signature, temporal pattern, intensity. For music detection: rhythm, harmony, frequency content.

Stage 3 — Classification head. The encoder's output is a vector, e.g. [0.23, 0.71, 0.05, ...], which is fed through:

Linear Layer → Softmax → Probability

Example output: Speech: 0.95, Music: 0.03, Noise: 0.02 → final prediction: Speech.

Audio → Feature Extractor → Transformer Encoder → Pooling → Classifier → Label

Why pooling is necessary: audio files have variable length (3 seconds vs. 20 seconds), but classification needs exactly one fixed-size answer per file. Pooling (e.g. Average Pooling) compresses the variable-length frame sequence into one fixed-length vector:

Frames → Average Pooling → One Vector

Wav2Vec2 for Classification — although best known for ASR, its encoder can be repurposed:

Audio → Wav2Vec2 Encoder → Classifier → Emotion / Speaker / Event

Audio Spectrogram Transformer (AST) — conceptually borrowed from the Vision Transformer: instead of image patches, it operates on spectrogram patches:

Audio → Spectrogram → Patches → Transformer → Class

In practice, many modern audio systems build one general-purpose audio encoder and swap only the head:

                Shared Audio Encoder
                        │
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
      ASR            Emotion         Speaker ID
      Head             Head             Head

The hard part — actually understanding raw audio — is learned once and reused across many downstream applications. This is the same "foundation model + lightweight heads" pattern you'd recognize from vision and language, applied here to acoustic representations.

1. Audio Classification identifies sound type/category, not spoken content.
2. Standard architecture: Audio → Feature Extractor → Encoder → Classifier → Label.
3. The encoder's output is a semantic representation of the audio.
4. Because file lengths vary, Pooling compresses variable-length sequences into a fixed vector.
5. Wav2Vec2 and Audio Spectrogram Transformer (AST) are both commonly used here.
6. Core distinction from ASR: ASR asks "what was said?"; Classification asks "what kind of sound is this?"

WER = (S + D + I) / N

• S (Substitution) — a word wrongly swapped: coffee → copy
• D (Deletion) — a word dropped entirely: "I really like coffee" → "I like coffee"
• I (Insertion) — an extra word added: "I like coffee" → "I really like coffee"
• N — total words in the reference (ground-truth) text

Worked example:

Reference:  I love machine learning
Hypothesis: I love machine learner

I         ✓
love      ✓
machine   ✓
learning  ✗   ← substitution: learning → learner

One substitution out of 4 reference words → WER = 1/4 = 25%. Lower WER = better.

Why WER is the standard: ASR's entire purpose is producing correct text — internal model behavior is irrelevant if the delivered word count is wrong. WER answers exactly the question that matters: how many words did it get right?

Same formula, applied at the character level instead of word level:

Reference:  hello
Hypothesis: helo      ← one character deleted

RTF = Processing Time / Audio Duration

Example: a 1-minute audio file processed in 10 seconds:

RTF = 10 / 60 = 0.16

Definition: how long after speaking does the system produce a result? For a voice assistant, the gap between the user saying "سلام" (hello) and receiving a response might be 50ms or 5 seconds — and that gap is often more perceptually important than raw accuracy.

In the real world, you need to know how the model performs across:

• Noise — speech layered with background noise
• Accent — different regional accents
• Speaker — different voices, ages, genders

Models are compared on standardized datasets (LibriSpeech, Common Voice, TED-LIUM) precisely so comparisons are apples-to-apples.

A concrete illustrative comparison:

For a mobile voice assistant, Model B is very plausibly the better choice — despite the "worse" WER — because interactive latency matters more than a few extra percentage points of accuracy in that context.

The practical formula: a genuinely good real-world ASR evaluation combines Accuracy + Speed + Memory + Latency — never accuracy alone. Benchmark leaderboards optimize for the first dimension and silently hide the other three.

ASR Model
   │
   ▼
Prediction → Generated Text
   │
   ▼
Compare with Reference
   │
   ├── WER
   ├── CER
   ├── Latency
   ├── RTF
   └── Robustness

1. The primary ASR metric is WER (Word Error Rate).
2. WER accounts for three error types: Substitution, Deletion, Insertion.
3. CER measures character-level error — useful for morphologically complex languages.
4. In production systems, speed and latency matter as much as raw accuracy.
5. A complete practical ASR evaluation covers: Accuracy (WER/CER) + Speed (RTF) + User Experience (Latency) + Robustness.

Connecting back to architecture: CTC-based models (Wav2Vec2) and Encoder–Decoder models (Whisper) are both typically evaluated with WER on decoded output, then compared across multiple benchmark datasets. A genuinely excellent ASR model isn't just one that outputs correct text — it must also run within the time and hardware constraints of its actual deployment target.

The real question this unit answers: pre-trained models like Whisper and Wav2Vec2 exist and can be used directly — but what if the general-purpose model just isn't good enough for your specific use case?

Re-train an already-pretrained model on your own specific data so it improves at a particular task.

Pre-trained ASR (e.g. general English Whisper)
        +
Bank Call Dataset (e.g. Persian bank support calls)
        ↓
Fine-tuned ASR

Fine-tuning only needs to align the model's existing knowledge with your new domain — not build that knowledge from zero.

Dataset → Preprocessing → Feature Extraction → Pre-trained Model → Training → Evaluation → Fine-tuned Model

1. Prepare the dataset — Audio + Transcript pairs, e.g. call_001.wav + "شماره حساب من را بررسی کنید".

2. Prepare the audio — resample (e.g. 48000 Hz → 16000 Hz), normalize volume, discard corrupted samples.

3. Prepare the text — tokenize into IDs the model can consume: "hello" → [15, 32, 44, ...].

4. Train — at each step: the model receives audio, produces a prediction (e.g. "helo"), that's compared against the ground truth ("hello"), a loss is computed, and weights are nudged slightly.

5. Evaluate — check WER before and after:

Before fine-tuning: WER = 20%
After fine-tuning:  WER = 8%

Fine-tuning Wav2Vec2:

Audio → Wav2Vec2 Encoder → CTC Head → Text

Typically, encoder weights are adjusted directly.

Fine-tuning Whisper:

Audio → Encoder → Decoder → Text

Because the model is large, it's common to freeze portions of it and train only specific parts.

1. Layer freezing

Encoder ❄️  frozen
Decoder 🔥  trainable

Benefit: lower memory footprint, faster training.

2. Parameter-Efficient Fine-Tuning (PEFT), e.g. LoRA Instead of updating billions of parameters, train only a small number of newly-added parameters. This is the general PEFT playbook — not architecturally unique to audio — applied here to a speech backbone.

Probably skip it if:

• The target language is common/general
• The audio is unremarkable/standard
• The base model is already strong

Probably do it if:

• You have domain-specific terminology
• You have a specific accent to handle
• The environment is noisy
• The target language is low-resource

If the fine-tuning dataset is small, the model may simply memorize those exact samples rather than generalizing:

Trained on: 100 people from one company
Then tested on: new people
→ performance drops sharply

Mitigations:

• More data
• Data augmentation
• Proper validation methodology

Starting from real audio (e.g. "Hello"), generate variants by:

• Adding noise
• Changing speed
• Changing pitch
• Simulating phone-call audio quality

This artificially broadens the effective diversity of a small dataset without needing new real recordings.

1. Fine-tuning adapts a ready-made ASR model to your specific data — it does not mean training from scratch.
2. Required data: Audio + Transcript.
3. Core pipeline stages: Preprocessing → Feature Extraction → Tokenization → Training → Evaluation.
4. Large models like Whisper are commonly fine-tuned more efficiently via layer freezing or LoRA.
5. Specialized-domain dataset quality typically matters more than how many parameters you unfreeze.
6. Watch for overfitting, especially when data is scarce — which is usually exactly the situation that motivated fine-tuning in the first place.

The engineering framing worth internalizing: the best real-world strategy usually isn't "pick the largest model" — it's Pre-trained Model + real target-domain data + smart fine-tuning = a practical ASR system.

Where ASR datasets are Audio → Text, TTS datasets are structured in the opposite direction:

Text + Recording of that text spoken by a human

The model learns: "if I see this text, how would a human pronounce it?"

{
  "text": "Hello, how are you?",
  "audio": "hello.wav",
  "speaker": "speaker_id"
}

1. Recording quality — matters far more here than in ASR, because the model must produce natural-sounding audio, not just interpret existing audio.

❌ Bad:  phone recording + background noise
✅ Good: studio recording

Key factors: low noise, good microphone, clear voice, controlled recording environment.

2. Transcript accuracy — must exactly match what was spoken.

Audio:      "I gonna go home"
Transcript: "I am going to go home"   ← MISMATCH — creates a problem

If the transcript doesn't match what's actually said, the model learns to produce something subtly different from the acoustic reality it's trained on — this corrupts the text→sound mapping in a way that's hard to detect just by skimming the transcript file.

3. Number of speakers — two fundamentally different dataset types:

4. Text diversity — the model needs exposure to short sentences, long sentences, numbers, names, and punctuation:

"Meeting starts at 10:30."  vs.  "Hello."

These stress very different aspects of prosody and pronunciation.

Building Persian TTS faces compounding difficulties:

• Limited dataset availability
• Limited speaker diversity
• Transcript quality issues

Building a usable Persian TTS system typically requires professional recording, diverse text sources, and several hours of clean audio — there's no large public dataset shortcut equivalent to what English has with LJSpeech/LibriTTS.

Text normalization — converting text into the form it should be read as:

Input:  "I have 10 apples"
Output: "I have ten apples"

The model needs to know how numbers (and dates, abbreviations, etc.) are actually pronounced, not just their written form.

Audio processing — resampling, trimming excess silence, normalizing volume.

Train/Validation/Test — with the same speaker-leakage trap as ASR:

❌ Bad:  Train: Ali's voice   Test: Ali's voice

The model might simply memorize Ali's voice rather than learning generalizable text-to-speech mapping.

Tacotron, for example, needs to learn Text → Mel Spectrogram — so the dataset must be structured to allow the model to actually learn that specific transformation cleanly.

1. A TTS dataset is Text + Audio pairs — the reverse orientation of ASR.
2. Recording quality and transcript accuracy directly determine how natural the output sounds.
3. Datasets are either Single Speaker (consistent, simple) or Multi Speaker (flexible, harder to train).
4. Well-known datasets: LJSpeech, LibriTTS, VCTK, Common Voice.
5. Text must be normalized (e.g. numbers → spelled-out words) before training.
6. In TTS, audio quality, speaker diversity, and text-audio alignment matter more than raw data volume.

Connecting to ASR: ASR asks the model to understand human speech; TTS asks it to produce human speech. In both, good data is the single most important driver of model quality — architecture choice is a distant second.

Just as ASR has Whisper and Wav2Vec2, TTS has its own family of ready-to-use pre-trained models — building one from scratch is rarely the right starting point.

Text → Pre-trained TTS Model → Speech

Building TTS from scratch is harder than it looks — the model must learn correct word pronunciation, natural speech rhythm, pausing, intonation, loudness dynamics, and speaker-specific characteristics, all of which require substantial data to learn well. The typical practical path:

Large TTS Model + Small Custom Dataset → Fine-tuned Voice

Text → Tokenizer → Text Encoder → Acoustic Model → Mel Spectrogram → Vocoder → Waveform

Some newer models fuse these stages into one end-to-end architecture rather than keeping them as separate trained components.

1. Tacotron 2 — the classic, highly influential architecture.

Text → Encoder → Attention → Decoder → Mel Spectrogram → WaveNet Vocoder → Audio

2. FastSpeech / FastSpeech 2 — built specifically to solve Tacotron's speed problem. Tacotron generates audio step-by-step; FastSpeech generates far more in parallel.

Text → Encoder → Duration Predictor → Mel Spectrogram → Vocoder → Speech

3. VITS (Variational Inference with adversarial learning for Text-to-Speech) — a modern, popular, fully end-to-end model.

Text → Encoder → Latent Representation → Decoder → Audio

Combines a text encoder, a latent-variable model, a vocoder, and GAN-based adversarial training into one system.

4. SpeechT5 — a multi-task Transformer from Hugging Face, capable of Text→Speech, Speech→Text, and Speech→Speech from one shared backbone.

Input → Shared Transformer → Task Output

The advantage: one model serves multiple tasks.

5. Bark — a generative audio model capable of speech, simple music, sound effects, and non-verbal vocalizations:

"laughing" → 😂 (generated laughter sound)

6. XTTS — focused on multilingual support and voice cloning:

Input:  Text + Short Speaker Sample
Output: Speech in that same voice

from transformers import pipeline

pipe = pipeline("text-to-speech", model="model_name")
output = pipe("Hello, how are you?")
# {"audio": waveform, "sampling_rate": 22050}

Pre-trained TTS (e.g. VITS) + Custom Voice Dataset (Text + Speaker Audio) → Custom TTS Model

Common goals: a Persian voice, a branded voice, or a specific person's voice.

Modern TTS goes beyond "read this text aloud" — the goal becomes "produce audio that sounds like this specific person." This typically requires:

Speaker Audio → Speaker Encoder → Speaker Embedding → TTS Model

1. Pre-trained TTS models let you go from text to speech without training from scratch.
2. Classic pipeline: Text → Acoustic Model → Spectrogram → Vocoder → Audio.
3. Key models: Tacotron 2 (classic foundation) · FastSpeech (speed) · VITS (quality, end-to-end) · SpeechT5 (multi-task) · XTTS (multilingual, voice cloning).
4. Custom/branded voices are typically built via fine-tuning or Speaker Embedding, not from-scratch training.
5. Good TTS doesn't just pronounce words correctly — it must sound natural, carry appropriate tone, and resemble genuine human speech.

Connecting back: ASR learns Audio → Representation → Text; TTS learns Text → Representation → Audio. Both ultimately learn the same underlying thing from opposite directions: the relationship between language and acoustic signal.

This section personalizes SpeechT5 for a specific voice, language, or domain — a concrete, step-by-step version of the fine-tuning concepts introduced earlier.

A Transformer-based, multi-task Hugging Face model capable of:

• Text → Speech (TTS)
• Speech → Text (ASR)
• Speech → Speech

Text → SpeechT5 Encoder-Decoder → Mel Spectrogram → Vocoder → Audio Waveform

The base model is trained on general-purpose data. You might need it to:

• Produce a specific voice
• Support a specific language better (e.g. English TTS → Persian TTS)
• Perform better in a specific domain (e.g. → Customer Support Voice)

{"text": "Hello everyone", "audio": "speaker001.wav"}

Model input: "Hello everyone" → Model target: the human recording of that exact sentence.

SpeechT5's defining feature is conditioning generation on who should be speaking:

Speaker Audio → Speaker Encoder → Speaker Embedding → SpeechT5 → Generated Voice

Step 1 — Prepare the dataset.

dataset/
 ├── audio/
 │     ├── 001.wav
 │     ├── 002.wav
 └── metadata.csv       # "001.wav | Hello world" / "002.wav | How are you"

Step 2 — Process the text. Tokenize: "Hello" → [39, 12, 5, 5, 8].

Step 3 — Process the audio. Standardize sampling rate, then convert to a Mel Spectrogram — the model predicts Mel Spectrogram, not raw waveform, directly:

Waveform → Mel Spectrogram

Step 4 — Train. Input = Text + Speaker Embedding. The model produces a Predicted Mel Spectrogram, which is compared against the Real Mel to compute loss, and weights update accordingly.

Step 5 — Apply the Vocoder.

SpeechT5 → Mel Spectrogram → Vocoder (commonly HiFi-GAN) → Audio

TTS is unusually sensitive to data quality:

10 hours of clean, single-speaker audio can outperform 100 hours of noisy, multi-speaker audio.

Sentences vary wildly in length:

"Hello"                              ← short
"This is a very long sentence..."    ← long

Batching requires uniform tensor shapes, so sequences must be padded to a common length — the Data Collator handles this automatically.

TTS evaluation is inherently harder than ASR evaluation, because there's no simple edit-distance ground truth for "how natural did that sound":

For large models, updating every weight is often unnecessary:

Freezing:

Encoder ❄️  frozen
Decoder 🔥  trained

Benefits: less memory, faster training.

PEFT methods like LoRA: train a small adapter instead of billions of parameters.

Text
 │
 ▼
Tokenizer
 │
 ▼
SpeechT5 ──── + Speaker Embedding
 │
 ▼
Mel Spectrogram
 │
 ▼
HiFi-GAN Vocoder
 │
 ▼
Speech

1. SpeechT5 is a multi-purpose Transformer for audio tasks (ASR, TTS, and more).
2. TTS fine-tuning requires: Text + Audio Recording pairs.
3. SpeechT5 uses Speaker Embedding to control voice identity in the output.
4. The model typically learns the path: Text → Mel Spectrogram → Vocoder → Audio.
5. Fine-tuning means adapting a general model to a specific language, domain, or voice — not building from zero.
6. In TTS, audio data quality matters even more than data volume.

This unit is effectively the last major building block before entering complete Voice AI systems: ASR to understand, an LLM to reason, and TTS to speak — which is exactly where Unit 7 picks up.

This unit is the synthesis of the entire course. Individually, you've now learned:

ASR                    → Speech → Text
TTS                    → Text → Speech
Audio Classification   → Audio → Label
Transformer architectures
Fine-tuning

The goal now: combine these pieces into real, working voice systems. Three canonical projects anchor this unit:

1. Speech-to-Speech Translation
2. Creating a Voice Assistant
3. Transcribing a Meeting

Goal: convert speech in one language into speech in another.

Input:  🎤 German speech
Output: 🔊 English speech

Classic cascade architecture — three models chained in sequence:

Speech → ASR → Text → Translation Model → Translated Text → TTS → Speech

i.e., Speech Recognition + Machine Translation + Speech Synthesis.

Worked example:

Input:        "Bonjour, comment allez-vous?"
ASR stage:    Bonjour, comment allez-vous?
Translation:  Hello, how are you?
TTS stage:    🔊 "Hello, how are you?"

Newer, more direct architectures (e.g. SeamlessM4T) skip the intermediate text entirely:

Speech → Speech-to-Speech Model → Speech

Trade-off between the two approaches:

Neither approach is strictly superior — the cascade's modularity (swap in a better translation model without retraining anything else) is a real engineering advantage even though the direct approach usually sounds better.

This is the Siri / Alexa / Google Assistant pattern:

🎤 User Speech → ASR → Language Model → TTS → 🔊 Response

Stage 1 — Speech Recognition.

User: "What's the weather today?"
ASR:  Audio → Text → "What's the weather today?"

Stage 2 — Understanding / Reasoning. A language model parses intent and extracts parameters:

Intent:   weather_query
Location: Berlin

Stage 3 — Speech Generation.

Response text: "The weather is sunny today."
TTS: Text → Speech

Latency. Users won't tolerate multi-second pauses — every component (ASR, LLM, TTS) needs to be fast enough that the combined pipeline still feels responsive.

Streaming. The system shouldn't wait for a complete sentence before beginning to process:

User (still speaking): "I want to know..."
                              ↓
                    Model already processing

Audio Stream → Streaming ASR → LLM → Streaming TTS

Wake word detection. The system listens continuously but only fully activates on a trigger phrase (e.g. "Hey Google") — a small, cheap, always-on classifier gates a much heavier downstream pipeline, so the expensive components only run when actually needed.

Goal: turn meeting audio into searchable, attributed text.

Input:  2-hour meeting audio
Output: Ali: We should improve the architecture...
        Sara: I agree...

System architecture:

Meeting Audio → Speaker Diarization → ASR → Transcript → Summary / Search

What Speaker Diarization actually is: determining who spoke when, entirely independent of what was said.

Audio:  [Person A] [Person B] [Person A]
Output: Speaker 1: Hello
        Speaker 2: Hi
        Speaker 1: Let's start

1. Audio Processing              → denoise, segment the file
2. Voice Activity Detection (VAD) → detect speech vs. silence
3. Speaker Diarization            → determine who spoke when
4. ASR                            → speech → text
5. Post-processing                → text correction, punctuation restoration, summarization

Voice Activity Detection, concretely:

Audio timeline: [Silence] [Speech] [Silence] [Speech]

VAD is what lets the system avoid wasting compute transcribing silence, and helps cleanly segment the file before diarization even starts.

                                Audio AI
                    ┌──────────────────┴──────────────────┐
                    ▼                                       ▼
                  ASR                                     TTS
             Speech → Text                            Text → Speech
                    │                                       │
                    └───────────────────┬───────────────────┘
                                        ▼
                              Voice Applications

ChatGPT Voice / general voice assistants:

User Voice → ASR → LLM → TTS → Assistant Voice

Simultaneous interpreter:

Language A Speech → ASR → Translation → TTS → Language B Speech

Meeting AI:

Audio → VAD → Diarization → ASR → Summary Model → Notes

1. Real audio products are pipelines of multiple specialized models — essentially never a single monolithic model.
2. A speech translator = ASR + Translation + TTS.
3. A voice assistant = ASR + LLM + TTS.
4. Meeting transcription requires VAD + Speaker Diarization + ASR (plus post-processing).
5. Real-system challenges that don't show up in isolated model benchmarks: Latency, Streaming, Memory, Accuracy, Multi-speaker handling.

                       Input Audio
                           │
                 ┌─────────┴─────────┐
                 ▼                   ▼
                ASR              Classification
                 │
                 ▼
                Text
                 │
                 ▼
             LLM / NLP
                 │
                 ▼
                TTS
                 │
                 ▼
           Output Speech

The single sentence that summarizes the entire field:

A complete voice AI system is a composition of Signal Processing + Deep Learning + Transformers + NLP — no single model does everything, and the real engineering skill lies in composing the right specialized components and managing the latency/accuracy/robustness trade-offs at every seam where they connect.

A single reference page for the comparisons that recur throughout this guide.

1. Every audio ML sample is audio.{path, array, sampling_rate} + a task-specific label — the model only ever sees the array, never the file itself.
2. Resampling to the model's training rate (usually 16 kHz) is mandatory, and it changes sample count, not perceived speed or pitch.
3. Streaming trades random access for near-zero disk/RAM cost — essential above a few hundred GB, unnecessary below it.
4. The universal backbone is Waveform → Features → Transformer Encoder → Task Head → Prediction; only the head changes across tasks.
5. Encoder-only vs. Encoder–Decoder is the single most predictive architectural fork — ask this first about any new model you encounter.
6. CTC vs. Seq2Seq is a speed/flexibility trade-off: CTC is fast but strictly monotonic; Seq2Seq is slower but can reorder, translate, and generate freely.
7. WER = (S+D+I)/N is necessary but not sufficient — RTF, latency, and robustness ultimately decide production viability, not benchmark leaderboard position.
8. Dataset quality dominates architecture choice in practice — a smaller, clean, well-labeled, well-matched-to-deployment dataset routinely beats a larger noisy one, in both ASR and especially TTS.
9. Speaker-identity leakage between train/test splits is a classic, audio-specific mistake that silently inflates apparent accuracy — watch for it explicitly.
10. In TTS and generative audio, data quality beats data volume far more decisively than it does in ASR — because the model must reproduce acoustic nuance, and any noise in training data becomes an artifact in every generated sample.
11. Real products are pipelines, not single models — speech-to-speech translation, voice assistants, and meeting transcription each compose 3+ specialized components, and the hardest engineering challenges (latency, streaming, diarization, wake-word gating) live at the seams between components, not inside any single one.
12. Bigger is not automatically better — model selection is a genuine trade-off between accuracy, latency, memory, and hardware target, not a single-axis optimization.

نسخه‌ی فارسی این مستند (با پشتیبانی کامل از راست‌به‌چپ) در حال آماده‌سازی است و در مرحله‌ی بعد به همین ریپازیتوری اضافه خواهد شد.

(A full right-to-left Persian-language version of this document is in progress and will be added to this repository in the next step.)

Found an inaccuracy, an outdated model reference, or want to add a section (e.g. a code-lab companion, diagrams-as-images, or benchmark numbers)? Issues and PRs are welcome.

This educational reference is shared under the MIT License — feel free to fork, adapt, translate, and extend it.