A dense reference for attention, the transformer block, parameter, FLOP, and memory counting, decoding, fine-tuning methods, retrieval-augmented generation, and evaluation. Derivations and runnable implementations are in the chapters.
\(\operatorname{softmax}\!\left(QK^\top/\sqrt{d_k} + M\right)V\), softmax over each row
Why \(\sqrt{d_k}\)
For unit-variance components, \(\operatorname{Var}(\mathbf{q}^\top\mathbf{k}) = d_k\); dividing restores variance 1 and keeps the softmax out of saturation
Mask \(M\)
\(0\) where allowed, \(-\infty\) where blocked; causal mask blocks \(j > i\)
Self- vs cross-attention
Self: \(Q, K, V\) from the same sequence. Cross: \(Q\) from the decoder, \(K, V\) from the encoder
Multi-head
\(\operatorname{Concat}(\text{head}_1, \ldots, \text{head}_h)W_O\), each head attends in a \(d/h\) subspace
MHA parameters
\(4d^2 + 4d\) with biases, independent of \(h\)
Cost
\(O(T^2 d)\) time, \(O(T^2)\) memory for scores (FlashAttention avoids storing them)
Grouped-query attention
Several query heads share one key-value head: smaller KV cache
importmathimporttorchimporttorch.nn.functionalasFdefattention(Q,K,V,mask=None):# mask: True = may attendscores=Q@K.transpose(-2,-1)/math.sqrt(Q.shape[-1])ifmaskisnotNone:scores=scores.masked_fill(~mask,float("-inf"))returnscores.softmax(-1)@VQ=K=V=torch.randn(2,4,5,8)# (batch, heads, T, d_k)causal=torch.tril(torch.ones(5,5,dtype=torch.bool))asserttorch.allclose(attention(Q,K,V,causal),F.scaled_dot_product_attention(Q,K,V,is_causal=True),atol=1e-6)
Opposite mask conventions
F.scaled_dot_product_attention: boolean True = may attend. nn.MultiheadAttention and nn.Transformer: boolean True = blocked.
\(x \leftarrow x + \operatorname{MHA}(\operatorname{LN}(x))\), then \(x \leftarrow x + \operatorname{FFN}(\operatorname{LN}(x))\); a final LayerNorm before the output head.
Attention moves information between positions; the FFN processes each position.
Pre-LN trains stably without long warmup; post-LN, \(\operatorname{LN}(x + f(x))\), was the original.
Common variants: RMSNorm instead of LayerNorm, SwiGLU FFN (three matrices, hidden size about \(\frac{8}{3}d\)), no biases.
KV cache: \(2 \times L \times T \times d_{kv} \times\) bytes, per sequence (\(d_{kv} = d\) without GQA). \(L = 32\), \(d = 4096\), \(T = 4096\), bf16 → about 2.1 GB per sequence.
Activations (GPT layer, 16-bit, no recomputation, Korthikanti et al.): \(\approx sbh(34 + 5as/h)\) bytes per layer. Checkpointing each layer keeps about \(2sbh\).
Keep the smallest set with total probability \(\ge p\)
Adapts to confidence; common default (\(p \approx 0.9\))
Min-p
Keep tokens with \(p \ge p_{\min} \cdot p_{\max}\)
Stable at high temperature
Speculative decoding
Draft \(k\) tokens with a small model; accept with \(\min(1, p/q)\), else resample from \(\max(0, p - q)\)
Faster, exactly the target distribution
Perplexity\(= \exp(\text{mean NLL per token})\), the effective number of choices. Compare across tokenizers with bits per character\(= \text{NLL}_{\text{token}} / (\text{chars per token} \cdot \ln 2)\). A uniform model over \(V\) tokens has perplexity \(V\); check that step-0 loss \(\approx \ln V\).
Group-relative advantage \((r_i - \bar{r})/\operatorname{std}(r)\), no critic
importtorch.nnasnnclassLoRALinear(nn.Module):def__init__(self,base,r=8,alpha=16):super().__init__()self.base,self.scale=base,alpha/rbase.requires_grad_(False)self.A=nn.Parameter(torch.randn(r,base.in_features)/base.in_features**0.5)self.B=nn.Parameter(torch.zeros(base.out_features,r))defforward(self,x):returnself.base(x)+(x@self.A.T@self.B.T)*self.scaledefdpo_loss(logp_w,logp_l,ref_logp_w,ref_logp_l,beta=0.1):# summed token log-probs per responsereturn-F.logsigmoid(beta*((logp_w-ref_logp_w)-(logp_l-ref_logp_l))).mean()
Choosing: try prompting first; fine-tune for format, style, and task behavior; use LoRA or QLoRA when memory is tight or you need many task variants; use RAG, not fine-tuning, for facts that change or must be cited; use DPO or RLHF to align with preferences.
Position, verbosity, and self-preference bias; validate against humans
Task-specific eval set
What you actually need
Build it early: a few hundred real inputs with references or rubrics
Retrieval metrics
Whether RAG finds the evidence
Measure separately from generation
Safety and red-teaming
Harmful outputs, jailbreaks, injection
Layer defenses; least-privilege tools
Hallucination: plausible but unsupported output. Mitigate with grounding and citations, allowing abstention, automatic verification, and human review for high stakes. Temperature 0 makes outputs repeatable, not true.
Missing \(\sqrt{d_k}\) or a wrong mask convention: training stalls, or the loss is suspiciously low because the model sees the future. Test causality by perturbing a future token and checking that earlier outputs don't change.
Initial loss far from \(\ln V\): bad initialization or a label-shift bug. Fix it before tuning anything else.
Comparing perplexities across tokenizers: convert to bits per character or byte first.
Forgetting the position offset with a KV cache: every generated token gets position 0. Compare cached and uncached greedy outputs.
Sampling with greedy decoding on open-ended text: repetition loops. Use temperature with top-p or top-k.
SFT without loss masking or without an end-of-turn token: the model learns to write user turns, or never stops.
LoRA with an unfrozen base: print the trainable parameter count before training.
16-bit weights without fp32 master copies: small updates round to nothing and learning silently stops.
Gradient accumulation without dividing the loss by \(k\): an effective learning rate \(k\) times too large.
Quantizing without evaluating: check task accuracy and per-layer error; keep embeddings, the output head, and norms in higher precision.
Evaluating RAG only end to end: measure retrieval separately; most failures start there.
Optimizing a reward model without a KL penalty: reward hacking and collapsed diversity.