Transformers & Pretraining
A 16-module path through transformers & pretraining, in study order from foundations to the frontier.
Modules
- Self-Attention Mechanism
- Scaled Dot-Product Attention
- Multi-Head Attention
- Positional Encodings
- Residual Connections & Layer Normalization
- Position-Wise Feed-Forward Networks
- Transformer Encoder Stack
- Masked Self-Attention
- Transformer Decoder Stack
- Full Transformer Architecture
- Tokenization at Scale
- BERT & Masked Language Modeling
- GPT & Autoregressive Language Modeling
- Pretraining-Finetuning Paradigm
- Transfer Learning in NLP
- Scaling Laws & the Compute-Data-Parameters Tradeoff