LLM architecture from scratch
Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.
Implemented
| Model | Model Size | Modality | Normalization | Positional Encoding | Attention | Mixture of Experts |
|---|---|---|---|---|---|---|
| GPT-2 XL | 1.5B | Text | LayerNorm | Learned Absolute | Multi-Head Self-Attention (causal) | ❌ |
| Llama3-8B | 8B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| Llama 3.2-3B | 3B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| OlMo 2 7B | 7B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| Phi-4 | 14B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| DeepSeek v3 | 671B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ✅ |
| Gemma 3 | 27B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ |
| xLSTM | 7B | Text | RMSNorm | RoPE | mLSTM | ❌ |
| Mistral 3.1 Small | 24B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ |
| Llama 4 Maverick | 400B | Text | RMSNorm | RoPE | GQA | ✅ |
| Qwen3 | 32B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
In progress
- Qwen3 (235B-A22B)
- SmolLM3
- Kimi K2 (1T)