Transformers

The architecture that unified NLP (and now vision, audio, and multimodal AI). Full derivation of self-attention, positional encoding, and the Transformer block. Then: BERT, GPT, T5, and beyond.

Table of contents

1. Motivation: attention without recurrence
2. Scaled dot-product attention (full derivation)
3. Multi-head attention
4. Transformer block
5. Positional encoding
6. The full Transformer (Vaswani et al., 2017)
7. Masking
8. BERT (encoder)
9. GPT (decoder)
10. T5 (encoder-decoder)
11. Efficient transformers
12. Vision Transformer (ViT)
13. Parameter-efficient fine-tuning (LoRA)

1. Motivation: attention without recurrence

1.1 Problems with RNNs for long sequences

1. Sequential computation: $\mathbf{h}_t$ depends on $\mathbf{h}_{t-1}$ → cannot parallelize across the sequence. Training on long sequences is slow.
2. Long-range dependencies: gradient path from position $T$ to position 1 is $T$ steps long → vanishing gradients even with LSTM.
3. Fixed context vector: seq2seq bottleneck, even with attention (attention is add-on; RNN still processes sequentially).

1.2 The transformer idea (Vaswani et al., "Attention is All You Need", 2017)

Remove recurrence entirely. Process all positions simultaneously with self-attention.

Self-attention allows every position to directly attend to every other position — path length 1 regardless of sequence length. No vanishing gradient across positions.

Tradeoff: $O(T^2)$ memory and compute for attention (vs $O(T)$ for RNNs). For very long sequences this is costly.

2. Scaled dot-product attention (full derivation)

2.1 Queries, Keys, and Values

Given an input sequence $\mathbf{X} \in \mathbb{R}^{T \times d}$ (T tokens, d-dimensional), project to:

where $\mathbf{W}^Q, \mathbf{W}^K \in \mathbb{R}^{d \times d_k}$ and $\mathbf{W}^V \in \mathbb{R}^{d \times d_v}$ are learned projection matrices.

Intuition:

• $\mathbf{Q}$ (Query): what am I looking for?
• $\mathbf{K}$ (Key): what do I contain?
• $\mathbf{V}$ (Value): what do I actually provide?

2.2 Attention computation

Step 1 — Raw attention scores:

Entry $A_{ij}$ = dot product between query $i$ and key $j$ = similarity between position $i$'s query and position $j$'s key.

Step 2 — Scale:

Why scale by $\sqrt{d_k}$? For random $\mathbf{q}, \mathbf{k} \in \mathbb{R}^{d_k}$ with unit-normal components:

Large $d_k$ → large variance → extreme dot products → softmax saturates → tiny gradients. Dividing by $\sqrt{d_k}$ keeps variance at 1.

Step 3 — Softmax over keys:

Row $i$ of this matrix = attention distribution of position $i$ over all positions.

Step 4 — Weighted sum of values:

Complete formula:

2.3 Complexity

• Time: $O(T^2 d_k)$ for the matrix product $\mathbf{Q}\mathbf{K}^\top$.
• Memory: $O(T^2)$ to store the $T \times T$ attention matrix.

For $T = 512$: $\approx 262K$ attention weights per head — manageable. For $T = 100K$ (long documents): impractical with standard attention.

3. Multi-head attention

3.1 Motivation

A single attention head computes one set of attention patterns. Different heads can attend to different aspects:

• Head 1: local patterns (adjacent tokens).
• Head 2: syntactic dependencies (subject-verb agreement).
• Head 3: coreference (pronouns → nouns).

3.2 Definition

With $H$ heads, each with dimension $d_k = d_v = d_\text{model}/H$:

where $\mathbf{W}^Q_h, \mathbf{W}^K_h \in \mathbb{R}^{d_\text{model} \times d_k}$, $\mathbf{W}^V_h \in \mathbb{R}^{d_\text{model} \times d_v}$, $\mathbf{W}^O \in \mathbb{R}^{Hd_v \times d_\text{model}}$.

Parameters: $H$ attention heads, total params = $4d_\text{model}^2$ (since $Hd_k = d_\text{model}$): three projection matrices $\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V$ each $d_\text{model} \times d_\text{model}$, plus output $\mathbf{W}^O$.

3.3 Cross-attention (encoder-decoder)

In the encoder-decoder Transformer, decoder queries attend to encoder keys and values:

This is the generalization of seq2seq attention.

4. Transformer block

4.1 Standard block (Post-LN, original)

Input x
  │
  ▼
Multi-Head Attention(x, x, x)   ← self-attention
  │
  + x  ← residual connection
  │
  ▼
LayerNorm
  │
  ▼
Feed-Forward Network (FFN)
  │
  + x  ← residual connection
  │
  ▼
LayerNorm
  │
Output

4.2 Pre-LN variant (more stable training)

Apply LayerNorm before each sub-layer (inside the residual branch):

x + MultiHead(LayerNorm(x), ...)
x + FFN(LayerNorm(x))

Pre-LN allows larger learning rates and is used in GPT-2, GPT-3, LLaMA.

4.3 Feed-Forward Network (FFN)

Two linear layers with a nonlinearity (ReLU or GELU):

$\mathbf{W}_1 \in \mathbb{R}^{d_\text{ff} \times d_\text{model}}$, $d_\text{ff} = 4d_\text{model}$ (standard expansion ratio).

FFN operates position-wise (same weights applied to each token independently). It is where most of the model's knowledge storage happens.

Parameters per block: $\approx 12d^2$ (4 for attention, 8 for FFN). For $d=768$: $\approx 7M$ params per block.

4.4 Gated FFN variants

SwiGLU (used in LLaMA, PaLM):

where $\text{Swish}(z) = z \cdot \sigma(z)$.

Empirically outperforms standard RELU FFN.

5. Positional encoding

5.1 Why needed

Self-attention is permutation-equivariant: the output of attention does not depend on token order (only on which tokens attend to which). Without positional information, "cat sat on mat" and "mat sat on cat" give the same output.

5.2 Sinusoidal positional encoding (original Transformer)

For position $t$ and dimension $i$ (out of $d_\text{model}$):

Added to token embeddings: $\tilde{\mathbf{e}}_t = \mathbf{e}_t + \text{PE}(t)$.

Properties:

• Each position has a unique encoding.
• The encoding for position $t + k$ is a linear function of the encoding at position $t$ (enables relative attention).
• Fixed (not learned); generalizes to sequences longer than training.

5.3 Learned positional embeddings

Learn a separate embedding $\mathbf{p}_t$ for each position up to a maximum length. Used in BERT, GPT-2.

Limitation: cannot extrapolate to longer sequences than seen in training.

5.4 Relative positional encoding (RoPE, ALiBi)

RoPE (Rotary Positional Encoding): encode relative position by rotating query and key vectors. Used in LLaMA, PaLM, many modern LLMs. Naturally handles longer sequences.

ALiBi (Attention with Linear Biases): add a linearly decaying bias $-m|i-j|$ to attention scores (no trainable position parameters). Generalizes well beyond training length.

6. The full Transformer (Vaswani et al., 2017)

6.1 Architecture

Encoder (stack of $N_E$ identical blocks):

Input tokens → Embedding + PE
  → [Self-Attention → Add&Norm → FFN → Add&Norm] × N_E
  → Encoder output (T × d)

Decoder (stack of $N_D$ identical blocks):

Output tokens (shifted right) → Embedding + PE
  → [Masked Self-Attention → Add&Norm
     → Cross-Attention(encoder output) → Add&Norm
     → FFN → Add&Norm] × N_D
  → Linear → Softmax → Output probabilities

6.2 Original hyperparameters (base model)

• $d_\text{model} = 512$, $H = 8$, $d_k = d_v = 64$.
• $d_\text{ff} = 2048$.
• $N = 6$ encoder layers, 6 decoder layers.
• Total parameters: $\approx 65M$.

6.3 Training

Loss: cross-entropy over target tokens (teacher forcing):

Optimizer: Adam with warmup + inverse square root decay:

where $W$ = warmup steps (e.g., 4000).

7. Masking

7.1 Padding mask

Prevents attending to padding tokens. Set attention scores for padding positions to $-\infty$ (→ softmax → 0).

7.2 Causal (autoregressive) mask

For decoder self-attention: position $t$ should only attend to positions $\leq t$ (no future leakage).

Mask matrix $\mathbf{M}$: upper triangle = $-\infty$, lower triangle + diagonal = 0.

8. BERT (encoder)

8.1 Architecture

Bidirectional Encoder Representations from Transformers (Devlin et al., 2018).

Encoder-only Transformer. Sees the full sequence (no causal mask). $\text{BERT}_\text{base}$: $L=12, H=768, A=12$, 110M params. $\text{BERT}_\text{large}$: $L=24, H=1024, A=16$, 340M params.

8.2 Pre-training objectives

Masked Language Modeling (MLM): randomly mask 15% of tokens; predict them. Of masked tokens: 80% replaced with [MASK], 10% with random token, 10% unchanged (prevents mismatch from [MASK] token not appearing at fine-tuning).

Next Sentence Prediction (NSP): predict if sentence B follows sentence A. (Later shown to be less useful; RoBERTa drops it.)

8.3 Fine-tuning

Add a task-specific head on top of BERT's [CLS] token embedding (for classification) or token embeddings (for token-level tasks). Fine-tune with small learning rate ($2\times10^{-5}$) for few epochs.

from transformers import BertForSequenceClassification, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

inputs = tokenizer("Hello, how are you?", return_tensors='pt')
outputs = model(**inputs, labels=torch.tensor([1]))
loss = outputs.loss
logits = outputs.logits

9. GPT (decoder)

9.1 Architecture

Decoder-only Transformer with causal mask (left-to-right autoregressive language model). Trained to predict the next token.

GPT-1 (2018): 117M params. GPT-2 (2019): 1.5B. GPT-3 (2020): 175B. GPT-4 (2023): unknown scale.

9.2 Training objective (language modeling)

Trained on massive corpora (books, web). Next-token prediction is a simple objective that forces the model to learn grammar, facts, reasoning, and world knowledge.

9.3 In-context learning (ICL)

GPT-3's key finding: large models can perform tasks without gradient updates by showing examples in the prompt:

Example: "Translate English to French: sea otter → loutre de mer"
Example: "translate cheese → fromage"
Test: "steak → "

Zero-shot / few-shot / many-shot: 0, few, or many examples in prompt. Performance improves with scale.

9.4 Instruction following (GPT-3.5, GPT-4)

Reinforcement Learning from Human Feedback (RLHF):

1. Fine-tune on demonstrations (supervised fine-tuning / SFT).
2. Train reward model on human preference comparisons.
3. RL fine-tuning (PPO) to maximize reward while not deviating from SFT model (KL penalty).

10. T5 (encoder-decoder)

10.1 Architecture

Text-to-Text Transfer Transformer (Raffel et al., 2019). Encoder-decoder Transformer where every task is framed as text-to-text:

• Translation: "translate English to German: The house is wonderful."
• Summarization: "summarize: ..."
• Classification: "mnli premise: ... hypothesis: ... entailment/contradiction/neutral"

Unified objective: standard language modeling cross-entropy loss for all tasks.

10.2 Span corruption pre-training

Mask contiguous spans of tokens (avg span length 3, 15% of tokens) and predict them. More efficient than MLM.

11. Efficient transformers

11.1 The $O(T^2)$ bottleneck

Standard attention has $O(T^2)$ memory for the attention matrix. For $T = 16384$ (16K tokens): $16384^2 = 268M$ values per head — multiple GB for a single layer.

11.2 Sparse attention

Sliding window attention (Longformer): each token attends to a window of $w$ neighbors. $O(Tw)$ complexity. Plus global tokens that attend everywhere (for [CLS]).

Local + global (BigBird): combines local attention, global tokens, and random attention. Provably as expressive as full attention.

Strided attention: attend to every $s$-th position. Combine with local to cover full range.

11.3 Linear attention

Rewrite attention without explicit $T\times T$ matrix. Kernel-based: approximate $\exp(\mathbf{q}^\top\mathbf{k}/\sqrt{d}) \approx \phi(\mathbf{q})^\top\phi(\mathbf{k})$. Reduces to $O(T)$ via associativity:

11.4 FlashAttention

FlashAttention (Dao et al., 2022): exact attention, reordered to minimize HBM (memory bandwidth) access via tiling. Uses on-chip SRAM. Same output as standard attention; $2\text{–}4\times$ faster; sub-quadratic memory.

FlashAttention-2/3: further optimizations, now standard in modern LLM training.

12. Vision Transformer (ViT)

12.1 Patching images

Dosovitskiy et al. (2020): apply a pure Transformer to images by treating image patches as tokens.

1. Split image $H \times W \times C$ into non-overlapping patches of size $P \times P$.
2. Number of tokens: $T = HW/P^2$. For $224\times224$, $P=16$: $T = 196$ patches.
3. Linearly project each flattened patch to $d_\text{model}$: $\mathbf{e}_t = \mathbf{W}_e \text{flatten}(\text{patch}_t)$.
4. Prepend [CLS] token; add positional embeddings.
5. Apply standard Transformer encoder.
6. Use [CLS] token embedding for classification.

12.2 Key observations

• ViT needs large pre-training data (ImageNet-21K or JFT-300M) to match CNNs. CNNs have stronger inductive biases (translation equivariance, locality) that help with small data.
• With enough data, ViT surpasses CNNs. Modern architectures (DeiT, Swin) close the data gap.

12.3 Swin Transformer

Hierarchical ViT: computes attention within local windows (reducing $T^2$ cost), then shifts windows between layers for cross-window connectivity. Produces feature maps at multiple scales (like CNNs) → useful for detection/segmentation.

13. Parameter-efficient fine-tuning (LoRA)

13.1 Motivation

Fine-tuning all 175B parameters of GPT-3 for each downstream task is expensive. PEFT methods adapt pretrained models with far fewer trainable parameters.

13.2 LoRA (Low-Rank Adaptation)

Hu et al. (2021): for a pretrained weight matrix $\mathbf{W}_0 \in \mathbb{R}^{d \times k}$, instead of learning $\Delta\mathbf{W}$, constrain it to be low rank:

where $\mathbf{B} \in \mathbb{R}^{d \times r}$, $\mathbf{A} \in \mathbb{R}^{r \times k}$, and rank $r \ll \min(d,k)$.

Initialization: $\mathbf{A} \sim \mathcal{N}(0, \sigma^2)$ (random), $\mathbf{B} = 0$ (so $\Delta\mathbf{W} = 0$ at start — same as pretrained model).

Parameters: $r(d+k)$ vs $dk$. For $d=k=768, r=8$: $8 \times 1536 = 12288$ vs $589824$ ($\approx 48\times$ reduction).

Why low rank? Weight updates during fine-tuning have low intrinsic dimensionality — the task-relevant changes lie in a low-dimensional subspace of the weight space.

At inference: merge $\mathbf{B}\mathbf{A}$ into $\mathbf{W}_0$ — no extra latency:

from peft import get_peft_model, LoraConfig, TaskType

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,               # rank
    lora_alpha=32,      # scaling factor alpha/r
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: ~4M || all params: ~7B || trainable%: 0.057%

13.3 Other PEFT methods

*File: notes/15_transformers.md — next: notes/16_generative_models.md*