Two eras: pretraining, then post‑training
If you've heard of GPT‑2 or GPT‑3, that's pretraining — the classical language‑modeling paradigm: model all of the internet, one next token at a time. If you've heard of ChatGPT, that's post‑training — the much more recent shift from "a model of internet text" to "an assistant that answers your question instead of continuing it." Every model you interact with today is both: a pretrained base model, then aligned on top of it.
The lecture's opening claim is worth sitting with: academia has spent most of its effort on architecture and training algorithms — new attention variants, new losses, new optimizers — because that's the part that produces publishable, attributable novelty. In practice, the three unglamorous components — data, evaluation, and systems — are what separates a mediocre model from a frontier one. That imbalance is why the rest of this page is heavy on the last three and light on the first.
Five components determine whether an LLM works: architecture, training objective, data, evaluation, systems. This lecture skips the first almost entirely — it's covered elsewhere — and spends its time on the other four, in roughly the order they happen: objective → data → scale → alignment → evaluation → systems.
Language modeling is just the chain rule
A language model is a probability distribution over token sequences, p(x₁, …, x_L).
"The mouse ate the cheese" should get higher probability than "The the mouse ate cheese" (broken syntax)
or "The cheese ate the mouse" (broken semantics) — a good model has learned both.
Autoregressive models factor that joint distribution with the chain rule of probability — no approximation, just one particular way of decomposing it:
Training just predicts the real next token and compares it to what actually happened (teacher forcing); sampling is the same model run in a loop — tokenize → forward pass → softmax over the vocabulary → sample a token → detokenize → feed it back in as new context. The tokenize/sample/detokenize steps only exist at inference time; training only needs the probability assigned to the real token.
Watch it sample, one token at a time
Four illustrative decoding steps for the lecture's own example, "She likely prefers ___". Each step shows the top‑5 candidates the model assigns probability to before a token is drawn — press Next to sample and extend the context.
One structural cost of this design: generation is an inherently sequential for‑loop — token t needs token t−1 — so a longer completion simply takes longer to produce. That's the whole reason inference latency is its own subfield.
Tokenization: the one decision you can't take back
Why not just use words? Because a typo produces a "word" with no token, and languages like Thai don't even use spaces to mark word boundaries — tokens have to be more general than words. Why not just use raw characters, then? Because attention cost grows quadratically with sequence length, and character sequences are long. Byte Pair Encoding (BPE) is the practical compromise, landing around 3–4 letters per token: start every character as its own token, then repeatedly merge whichever adjacent pair is most frequent across the training corpus, until you hit a target vocabulary size.
Asked in lectureDoes the output layer's size have to equal the number of tokens? Yes — the vocabulary is literally the width of the model's final softmax. That's why tokenizer design is close to a permanent decision: growing the vocabulary later means resizing (and retraining part of) the output layer.
Train a tiny BPE tokenizer, merge by merge
A simplified corpus of three related words — the merges are applied mechanically (most‑frequent adjacent pair wins), the same way a real BPE trainer works, just at toy scale.
Applying the trained tokenizer always greedily prefers the largest known token — so once "token" exists as a single token, the tokenizer never falls back to spelling it out letter by letter. Untrained pairs are simply left as smaller pieces, which is exactly what gives the tokenizer graceful degradation on typos and rare words.
Tokenization sounds like a footnote and isn't: because digits aren't tokenized digit‑by‑digit
(327 is often one opaque token), models can't decompose arithmetic the way a human does
by working digit‑by‑digit — one reason LLM math is unreliable. And whitespace matters more than it
looks: GPT‑4's tokenizer changes to how it handles the leading four‑space indentation in Python code
were a deliberate fix, because naive tokenizers had been mangling code structure.
Evaluating a base model
Perplexity is validation loss made interpretable: exponentiate the average per‑token negative log‑likelihood, so it lives on a scale of "how many tokens is the model hesitating between," bounded between 1 (perfect prediction) and the vocabulary size (total confusion):
On standard benchmarks that number fell from roughly 70 plausible next words in 2017 to under 10 by 2023 — models went from wavering between 70 tokens to wavering between fewer than 10. But perplexity isn't used to compare different models anymore, because it's tokenizer‑dependent: a model with a 100,000‑token vocabulary has a higher theoretical worst‑case perplexity than one with a 10,000‑token vocabulary, so the numbers aren't apples‑to‑apples across labs.
What replaced it is aggregating many task benchmarks — HELM (Stanford) and the Hugging Face Open LLM Leaderboard are the two most common. MMLU is the standard example: a four‑choice question (say, in astronomy), scored either by asking which answer the model assigns highest likelihood to, or by constraining decoding to just the tokens "A/B/C/D" and reading off the most likely one.
Same model, different harness
Llama‑2‑65B scored 63.7% on MMLU under HELM's evaluation harness and 48.8% under a different one — same model, same benchmark name, a 15‑point gap purely from scoring‑methodology choices (never mind prompt wording).
Catching train/test contamination
One clever check: benchmark datasets online usually aren't shuffled. If a model assigns higher likelihood to a benchmark's examples in their published order than to the same examples shuffled, that's evidence the ordering was memorized during pretraining — indirect proof the test set leaked into training data.
Turning a petabyte of internet into a corpus
"Train on all of internet" undersells the amount of engineering involved. Common Crawl — the standard web crawl everyone starts from — holds around 250 billion pages, roughly 1 petabyte. A random page from it is not Wikipedia; it's often a half‑rendered HTML fragment with an unfinished sentence. Getting from that to a training corpus is most of the actual work:
Asked in lectureWhy filter out toxic content instead of training on it with a supervised signal that says "don't do this"? Both happen — but not at this stage. Pretraining is trying to model how people write generally; the "here's what not to say" signal comes later, in post‑training.
Two numbers worth keeping: this pipeline is genuinely secretive — competitively (nobody wants to hand competitors their recipe) and legally (nobody wants to publicly confirm they trained on copyrighted books). And it's a bigger team than you'd guess: on a ~70‑person model team, something like 15 people work on data alone — plus a large, separate CPU compute budget just for filtering at this scale.
Scaling laws: predicting the future from small models
The empirical finding (OpenAI, 2020) that reshaped the field: plot compute, data, or parameter count against test loss on log‑log axes, and the relationship is close to a straight line. That's a genuinely strange and useful fact — it means you can predict how much a model will improve from 10× more compute before you've spent the 10×.
Illustrative reconstruction of the shape described in the lecture (based on Kaplan et al. 2020's compute‑scaling result) — not the original digitized curve. The point isn't the exact numbers, it's that both slope and intercept are comparable across architectures: this is literally how you'd settle a "transformer vs. LSTM" argument without training either at full scale.
This flips the standard ML workflow. Old pipeline: hyperparameter‑search directly on the model you'll ship — if you have 30 GPU‑days, that might mean 30 one‑day runs, and the model you actually deploy only got one day of training. New pipeline: find a scaling recipe (e.g., "learning rate should shrink as model size grows"), tune hyperparameters on several small models across a range of scales, fit a scaling law to those points, extrapolate to the target size — then spend nearly all the budget on one large final run.
Chinchilla: how many tokens per parameter?
If more data helps and more parameters help, and compute is the shared constraint, what's the optimal split? Training‑loss‑optimal ("Chinchilla‑optimal") lands near 20 tokens per parameter. But that ignores inference cost — a smaller model trained on more tokens costs less to serve for its entire deployed lifetime — so production models in practice sit closer to 150 tokens per parameter.
"The only thing that matters, in the long run, is the leverage on computation." — paraphrasing Rich Sutton's Bitter Lesson (2019), the lecture's framing for why clever architecture tweaks mostly nudge the intercept, not the slope, of these curves — and why compute, data, and systems keep winning against hand‑engineered inductive bias. Cited in lecture as the reason academia's architecture‑centric incentives point the wrong way.
What a frontier run actually costs — Llama 3 405B
Back‑of‑envelope math the lecture walked through live, using the shorthand
FLOPs ≈ 6 · params · tokens:
| Parameters | 405 B |
|---|---|
| Training tokens | 15.6 T |
| Tokens / parameter | ≈ 38 |
| Compute (6·P·D) | ≈ 3.8 × 10²⁵ flops |
| Regulatory note | Just under the 10²⁶‑flop threshold that triggers U.S. executive‑order scrutiny |
| Hardware | 16,000 × H100 |
| Wall‑clock time | ≈ 70 days |
| GPU‑hours | ≈ 26–30 M |
| Rental cost (@ ~$2/GPU‑hr) | ≈ $52 M |
| Team (≈50 people, 1 yr) | ≈ $25 M |
| Total, rough order | ≈ $75 M |
| Carbon (CO₂‑equivalent) | ≈ 4,000 t (~2,000 JFK↔London round trips) |
The 38 tokens/parameter ratio sits deliberately between the Chinchilla‑optimal 20 and the inference‑optimal 150 — a middle path. And each new model generation has roughly multiplied compute by 10× over the last few — constrained only by how much energy and hardware a lab can actually secure.
Post‑training I: supervised fine‑tuning
A pure pretrained model isn't an assistant — ask GPT‑3 to "explain the moon landing to a six‑year‑old" and it's liable to continue with another question, because on the internet a question is usually followed by more questions, not an answer. Post‑training's job is closing that gap.
Supervised fine‑tuning (SFT) uses the exact same next‑token cross‑entropy loss as pretraining — the only thing that changes is the data (curated instruction → human‑written‑answer pairs) and the hyperparameters (much higher learning rate, vastly less data). It's best understood as behavioral cloning: the pretrained model already contains many "types of writers" — one that writes bullet‑point continuations, one that writes an actual answer — and SFT just reweights toward the assistant‑shaped one.
Surprisingly, SFT needs very little data: the LIMA paper found scaling from 2,000 to 32,000 curated examples barely moved performance. The intuition matches the "reweighting, not teaching" framing — you're not injecting new knowledge, just telling the model which latent persona to commit to, so a few thousand well‑chosen examples saturate quickly.
Alpaca: bootstrapping instruction data with an LLM
Human‑written instruction data is slow and expensive to collect. Alpaca
started from 175 human‑written seed examples, asked a strong model
(text‑davinci‑003) to generate 52,000 more in the same style, then fine‑tuned
LLaMA‑7B on that synthetic set to get Alpaca‑7B — an early, low‑cost academic replication of the
SFT step behind ChatGPT, and one of the first widely‑used examples of "use an LLM to generate the
data that trains the next LLM."
Asked in lectureDoesn't purely synthetic data eventually collapse in on itself across generations? Probably, after a few generations of pure self‑bootstrapping — the more promising direction is human‑in‑the‑loop: let the model draft, have a human edit rather than write from scratch. Edits are far cheaper than fresh writing, and still inject genuinely new information, unlike pure LLM‑to‑LLM bootstrapping.
Post‑training II: from preferences to RLHF and DPO
SFT has three concrete problems. It's bounded by human generation ability — a human labeler can often judge that one essay is better than another without being able to write the better one themselves. It plausibly causes hallucination: if a human‑written reference answer contains a fact the pretrained model never actually saw, SFT teaches the model that the right move is to confidently assert plausible‑sounding claims regardless of whether it actually knows them. And full human‑written gold answers are expensive to collect at scale.
RLHF sidesteps all three by asking for something much cheaper than a perfect answer: given two model‑generated completions, which one is better? A raw ±1 preference signal is too sparse to learn much from, so instead you train a reward model — a classifier fit with the Bradley–Terry loss, where the reward assigned to each completion is literally the logit of a two‑way softmax over the preferred vs. dispreferred pair. Then PPO optimizes the policy against that learned reward, with a KL penalty back to the SFT model so the policy can't wander into blind spots the reward model rewards but a human wouldn't ("reward over‑optimization").
Worth flagging: once a model is trained with PPO it's no longer a calibrated language model — it's a policy optimized to generate one high‑reward answer, with nothing pushing it to preserve a realistic distribution over plausible answers. That's part of why perplexity stops being a meaningful number for aligned models (more in the next topic).
| — | PPO | DPO |
|---|---|---|
| Separate reward model? | Yes | No — folded into one loss |
| RL rollout loop? | Yes — sample, score, update, repeat | No — a single supervised‑style pass |
| Can exploit unlabeled data? | Yes, via the reward model labeling it | No — needs labeled pairs directly |
| Implementation complexity | High — clipping, rollouts, notoriously finicky | Low — closer to ordinary fine‑tuning |
| Empirical performance | Strong | Matches PPO in practice |
| Where it's used now | Original ChatGPT recipe | Open‑source default; used in industry too |
Asked in lectureIf DPO is simpler and performs the same, why did anyone start with PPO? Path dependence — the team that built the first RLHF pipeline included the authors of PPO itself; reinforcement learning was the tool they already trusted. RL does keep one theoretical edge (a trained reward model can label unlabeled data for further training, which DPO can't do directly) — but that edge hasn't translated into a practical advantage.
Evaluating an assistant
Every base‑model evaluation trick breaks down for an aligned model. There's rarely one correct answer to score against; PPO'd policies aren't calibrated distributions, so perplexity is meaningless; and PPO vs. DPO checkpoints don't even share a comparable loss to begin with. The fix is the same preference‑pair idea RLHF trains on, repurposed for measurement: show a labeler two models' answers to the same real question, ask which is better.
ChatBotArena does this at real scale — hundreds of thousands of blind, crowdsourced votes from actual users (skewed toward technical questions, since that's who shows up to compare chatbots for fun). AlpacaEval is the cheap proxy: have GPT‑4 itself judge which of two answers is better, average into a win‑rate. It correlates 98% with ChatBotArena's rankings, and a full run costs under $10 and takes under 3 minutes — which is why it's become the everyday development loop metric.
The length‑bias trap
GPT‑4's win rate against itself, varying only a system‑prompt instruction:
Nothing about answer quality changed — only length. Humans have the same bias but self‑correct ("five pages for a simple question? no thanks"); an LLM judge trained against this bias keeps exploiting it indefinitely, which is exactly why RLHF'd models tend to get longer the more alignment they go through. Length‑controlled variants of these benchmarks use regression/causal adjustment to strip the effect back out.
A sobering baseline underneath all of this: on binary preference judgments, humans agree with the majority vote of other humans only about 66–68% of the time — even five co‑authors calibrating for three hours before labeling only reached that range. Well‑tuned LLM judges now match the human‑majority vote more often than an individual human does, at roughly 1/50th the cost — which is the actual argument for LLM‑as‑judge, not just convenience.
Systems: where the compute actually goes
GPUs are throughput machines, not latency machines — many simple cores running the same instruction over different data, tuned hardest for matrix multiplication (routing a computation through matmul is routinely ~10× faster than any alternative). The catch: raw compute has been improving faster than memory bandwidth and inter‑GPU communication, so naively‑written training code leaves GPUs idle waiting on data far more often than people expect. The standard metric is Model FLOP Utilization — observed throughput over theoretical peak — and ~50% is considered a good result; Meta reported Llama‑class training around 45%.
Mixed precision
Fewer bits per number means less data to move, and deep nets tolerate the added rounding noise fine (SGD is already noisy). The standard split:
Full fp32 stays only where it matters — the master weights — so a tiny learning‑rate step doesn't get rounded away to nothing.
Operator fusion
Naive eager execution ships each tensor from GPU memory to the
compute cores and back per line of code. A fused kernel — what torch.compile
does automatically — rewrites a whole chain of operations into one round trip instead of one per
op.
x.cos().cos(): naive execution is
memory→core→memory→core→memory (4 trips); a fused kernel does memory→core→memory (2 trips) —
roughly a 2× speedup, for a one‑line change.
The lecture ran out of time before tiling, multi‑GPU parallelism strategies, and mixture‑of‑experts routing — flagged here as open threads rather than skipped silently.
What actually determines whether an LLM is good
Run the whole lecture back through one lens and it's a single argument, made five different ways: the next‑token objective, the tokenizer, the data pipeline, the scaling laws, and the RLHF→DPO simplification are all cases of a boring, general method winning over a clever, specific one, purely because it scales with compute. That's the Bitter Lesson, and it's why the lecturer — an architecture researcher for a chunk of their own career — spent most of this lecture everywhere except architecture.
Left out entirely here, by the lecture's own admission: architectural detail, inference‑time optimization, multimodality, whether the internet even contains enough remaining data, and the legal exposure of how it was collected. The lecturer pointed toward three follow‑on courses for those: CS224N for grounding and history, CS324 for a deeper treatment of everything covered here, and CS336 for building an LLM from scratch.