Previous slide Next slide Toggle fullscreen Toggle overview view Open presenter view
Lecture 17: Large language models
What they are, and how they are trained
Systems and Toolchains for AI Engineers
A chatbot, from the inside
What we did: asked a real chat model one question, on a laptop: Qwen2.5-0.5B-Instruct, a small open model from Alibaba (0.49 billion parameters), built like a GPTWhat you see: the chat app (left), the text the model really reads and continues token by token (middle), its top guesses (right). Press playThe gray tags, <|im_start|> and <|im_end|>: special tokens that open and close each message, so the model knows who speaks. The app hides them
Systems and Toolchains for AI Engineers
Today
1
From generative models to text Text, one token at a time
2
Tokens What the model reads, and what it costs
3
The model A transformer that only looks back
4
Training Predict the next token, everywhere
5
Data and scale From our notes to trillions of tokens
6
Sampling Temperature, and why fluent is not true
Systems and Toolchains for AI Engineers
Today, our running example: a tiny GPT
Our notesLectures 1 to 16
→
Train2,000 steps
→
A tiny GPT842,881 parameters
GPT , generative pretrained transformer: the model family behind ChatGPT. Ours is tiny, and we train it from scratch
It reads and writes one character at a time
Same recipe as GPT-3, about 200,000 times smaller. We trained it for this lecture, and it appears on many slides
Before training , it writes:The model l₂4zé'-(u↑Ly°⁷)aQ8d¹₆−wZ²t,>zü4i{₂YGIkt└TR>P×−t_ …
After training , given the start of one term from each lecture, it finishes 15 of 16 : Parquet regression Bayesian optimization physics-into a s
Systems and Toolchains for AI Engineers
From generative models to text
Systems and Toolchains for AI Engineers
From generative models to text, where Lecture 16 left us
Datax
→
Learn the distributionp(x)
→
Fix what you know,
Lecture 16: a generative model learns the distribution of the data, then samples new data from it
There, x was a point in two dimensions: Gaussian mixtures, flows, diffusion
Today, x is a text: a sequence of tokens A large language model (LLM) is a generative model of text. Your prompt is the part you fix; it samples the rest
Systems and Toolchains for AI Engineers
From generative models to text, one variable at a time
Left, Lecture 16: draw a whole point (x, y) at once
Right, the same distribution, one variable at a time: first draw x, then y given that x. Same result: p(x, y) = p(x) p(y | x)Text has too many possibilities to draw at once. A 10-token text has 10 slots, each holding any of 50,257 tokens: 50,257 × 50,257 × … ten times = 50,257¹⁰ ≈ 10⁴⁷ texts. One token at a time, each step is one choice among 50,257
Systems and Toolchains for AI Engineers
From generative models to text, the chain rule
The whole text How likely this exact text is, all T tokens together
One factor per token Multiply: the 1st token, times the 2nd given the 1st, times the 3rd given the first two, and so on to the last
One next-token probability How likely token t is, given all the tokens before it. This is the only thing an LLM learns
Example: we ran GPT-2 (OpenAI, 2019, 124 million parameters) on "The pump stopped" and read its probability for each token, given the ones before it:
p("The pump stopped")-6 × 0.0019 = 5.9 × 10-10
Systems and Toolchains for AI Engineers
Language model
Language model : a model that gives the probability of the next token,
Large language model (LLM) : a language model with billions of parameters, trained on trillions of tokens.
Systems and Toolchains for AI Engineers
From generative models to text, the generation loop
1
Predict the last factor of the chain rule, for every possible next token:
"The pump stopped because the" →
2
Pick One token from that distribution
Here, the most likely: ␣water . Other ways to pick: Sampling section
3
Append Add it to the end of the text
"The pump stopped because the water "
4
Repeat Predict again, from the longer text
→ ␣was → ␣too → ␣hot → .
The pump stopped because the ␣water ␣was ␣too ␣hot .
Each red token is one trip round the loop: GPT-2 predicted it, it was appended, and the next prediction started from the longer text. A model that uses its own outputs as its next inputs is autoregressive .
Systems and Toolchains for AI Engineers
From generative models to text, autoregressive as in Lecture 8
Lecture 8's AR(10), reactor pressure, 2 hours ahead on one test run
Lecture 8: AR(10) An LLM Predicts the next pressure the next token From the last 10 values all the text so far Output one number a probability for every token Model 11 coefficients billions of parameters Long outputs feed each forecast back in append each token: the same
Same weakness: both feed their own outputs back in, so an early mistake becomes an input for every later step. Lecture 8: after 2 hours, the forecast was worse than the plain mean. An LLM: one wrong token steers the rest of the answer
Systems and Toolchains for AI Engineers
Tokens
Systems and Toolchains for AI Engineers
Tokens, what the model actually reads
Token : a piece of text the model reads and writes: a character, a word, or part of a word.
Vocabulary : the list of every token the model knows, each with a number, its token id .
Tokenizer : the program that cuts text into tokens and looks up their ids.
Text The pump stopped. ↓ the tokenizer cuts the text into tokens Tokens The ␣pump ␣stopped . ↓ and looks up each token in the vocabulary Token ids 464 8901 5025 13
GPT-2's vocabulary, real ids (50,257 in all)
0 !
1 "
2 #
⋮
13 .
⋮
464 The
⋮
5025 ␣stopped
⋮
8901 ␣pump
⋮
Systems and Toolchains for AI Engineers
Tokens, three ways to cut the same text
Characters
T h e ␣ p u m p ␣ o v e r h e a t e d ␣ a t ␣ 8 5 ␣ ° C .
Words
The pump overheated at 85 °C.
Subwords
The ␣pump ␣overhe ated ␣at ␣85 ␣° C .
Characters One token per letter, digit, space or symbol. A tiny vocabulary (our tiny GPT: 129 ), but long sequences
Words One token per word. The vocabulary needs every word ever written, and a new one has no token: a pump tag like P-101A, a steel grade like AISI 4140
Subwords Pieces of words. Common words stay whole ("␣pump"), rarer ones split ("␣overhe" + "ated"), so any new word can be built. GPT-2: 50,257 tokens
Systems and Toolchains for AI Engineers
Tokens, byte-pair encoding
Byte-pair encoding (BPE) : how a subword vocabulary is built. Find the pair of neighboring tokens that appears most often, merge it into one new token, repeat.
start: one token per character a a a b d a a a b a c 11 tokens
1. most frequent pair: a + a (4 times) aa a b d aa a b a c 9 tokens
2. most frequent pair: a + b (2 times) aa ab d aa ab a c 7 tokens
3. most frequent pair: aa + ab (2 times) aaab d aaab a c 5 tokens
Why: fewer tokens Frequent pieces become one token. Shorter sequences: more text fits in the context window, fewer steps to write
Why: nothing is unknown A new word can always be built from smaller pieces, down to single characters
Why: learned from data The merges come from the training text, so its frequent words become single tokens
Example from Wikipedia, Byte-pair encoding . The algorithm is Gage's (1994), adapted to building subword vocabularies by Sennrich et al. (2016) ; GPT-2 starts from bytes and stops after 50,000 merges.
Systems and Toolchains for AI Engineers
Tokens, byte-pair encoding learned from our notes
What we did: learned 600 merges by BPE from our notes (Lectures 1 to 16), then applied them, in order, to one sentence. Press play. The last row is GPT-4o's tokenizer, learned the same way from far more text
The takeaway: each merge makes tokens longer and fewer. 35 characters become 18 tokens with our 600 merges, 10 with GPT-4o's. Fewer tokens: more text fits in the context window, and writing it takes fewer steps
Systems and Toolchains for AI Engineers
Tokens, numbers and units cost more than words
Same length, very different cost: "heat exchanger" is 14 characters and 2 tokens . "Q = 1.5e-3 m³/s" is 15 characters and 12 tokens
The model reads numbers in pieces: the tokenizer cuts "4140" (a steel grade) into two tokens, "414" and "0", the way it cuts a word into pieces. The model never sees the quantity four thousand one hundred fortySo numbers, units and part numbers (P-101A: a pump tag) are where LLMs slip. Check them, and do the arithmetic in code
GPT-4o's tokenizer (o200k_base, in tiktoken).
Systems and Toolchains for AI Engineers
Context window
Context window : the most tokens a model can take in at once: your prompt, any documents, and everything it has written so far. Text beyond it is not seen at all.
128
characters: our tiny GPT
128,000
tokens: Llama 3 (
Llama Team, 2024 )
a long report, but not all our notes:
164,779 tokens
Too long to fit: cut it, summarize it, or search for the part that matters.
GPT-3: "a context window of nctx = 2048 tokens". Llama 3: "a context window of up to 128K tokens". Our notes counted with GPT-4o's tokenizer (tiktoken, o200k_base).
Systems and Toolchains for AI Engineers
The model
Systems and Toolchains for AI Engineers
The model, the big picture
The case: we gave GPT-2 the prompt "The plant shut down for maintenance" and asked for the next token. Watch "plant": alone, it could be a tree or a factory. The final probabilities are GPT-2's; the vectors and the map are a sketch. Press play
In Lecture 12 the tokens were cycles of a sensor window; the machinery is the same.
Systems and Toolchains for AI Engineers
The model, attention looks back
1. Score The word "too" asks every earlier word: how useful are you for guessing what comes next?
2. Mix It takes a weighted average of what those words carry. Here, mostly "water"
3. No peeking Later words are hidden: the causal mask . In training the next word is the answer, so it must stay hidden
In symbols, as in Lecture 12: softmax(QKᵀ/√dk + M) V. QKᵀ/√dk are the scores; the mask M is −∞ for later words, which sets their weight to 0. The weights in the picture are a sketch.
Systems and Toolchains for AI Engineers
Training
Systems and Toolchains for AI Engineers
Training, tuning the dials
One training step: show the model some text with the next character hidden; it guesses; compare its guess with the true character; nudge every dial a little so the true one gets more likely. Repeat 2,000 timesWhat you see: our tiny GPT on one example, "…learns the proba?". The true answer is "b". Press play
A straight-line fit (Lecture 9) has 2 dials; our tiny GPT has 842,881; GPT-3 has 175 billion. The nudge is a gradient step, with the gradient from backpropagation (Lecture 11).
Systems and Toolchains for AI Engineers
Training, the objective
Training makes real text likely. By the chain rule, a text's log-probability is a sum over its tokens. We want it large, so we minimize its negative
log p of the true next token How much probability the model gave the token that really came next. θ: the dials
The minus, and the average The minus makes it a loss: near 0 when right and sure, large when surprised (the plot). 1/T Σ: the average over T positions. This is the cross-entropy of Lecture 10
Systems and Toolchains for AI Engineers
Training, every position at once
What you see: one sentence of 10 tokens turned into 9 guesses. Guess 1: from "The" alone, guess the next token. Guess 2: from "The pump". Each answer is just the next token of the text: nobody labels anything. Press play
The point: one sentence gives 9 training examples, all done at once, in one pass: self-supervised learning. Each guess sees only the tokens before it (the causal mask); otherwise the answer would be in plain sight
Systems and Toolchains for AI Engineers
Training, an optimization problem
Decision variables The parameters θ: 842,881 in our tiny GPT, 175 billion in GPT-3
Objective The loss over all N predictions: the same kind of data loss we minimized in Lecture 13
Method Gradient-based, by Lecture 15's map. AdamW (Lecture 12) for our tiny GPT and Llama 3 ; Adam for GPT-3
Systems and Toolchains for AI Engineers
Training, reading the loss
At the start, the dials are random: each of the 129 characters gets about the same chance, 1/129. A blind guess has a loss of −ln(1/129) = 4.86: the dotted line
To read a loss, undo the ln. The loss is the average of −ln p, so e−loss is the typical p. At the end, e−1.51 = 0.22: the true next character typically gets 22%
loss typical p = e−loss
4.86 a blind guess, 1/129
1.51 our tiny GPT, at the end
0.69 a coin flip
0.00 sure and right
Systems and Toolchains for AI Engineers
Training, what it writes as it learns
What you see: the loss (top), what it writes after "The model " at that step, and what changed. Press play. Trained only on our notes, it copies their style, LaTeX included: far more text and a bigger model turn style into meaning
Systems and Toolchains for AI Engineers
Training, what our tiny GPT learned
Give it the first 60% of one term from each lecture; it picks the most likely next character until the term ends.
L1 toolchain
L2 environment
L3 database
L4 Parquet
L5 dataframe
L6 validation
L7 steady state
L8 forecast
L9 regression
L10 classification
L11 gradient
L12 convolution
L13 physics-into a s
L14 uncertainty
L15 Bayesian optimization
L16 distribution
15 of 16 right: it learned our spelling and our vocabulary. The miss: "physics-in" → "to a s", because "into a" appears 58 times in our notes and "physics-informed" only 8
No sentences with meaning: ask for one and it writes "and the same step of the same samples". Meaning takes far more parameters and data, next
Systems and Toolchains for AI Engineers
Data and scale
Systems and Toolchains for AI Engineers
Data and scale, in human terms
Their data Mostly filtered web text, plus books, code and papers. Llama 3: "about 15T multilingual tokens"
Our data The notes of Lectures 1 to 16: 668,651 characters, each seen about 27 times in 2,000 steps
Reading it Our notes: about 7 hours at 250 words a minute. GPT-3's training text: about 1,300 years , non-stop
Computing it Our tiny GPT needed about 8 × 10¹³ operations: 2 minutes on a laptop GPU (about 700 billion a second), but 23 hours at a billion a second. GPT-3 needed 3.14 × 10²³: about 10 million years at a billion a second
Systems and Toolchains for AI Engineers
Data and scale, after pretraining
1. Pretraining
Learns to continue text, from trillions of tokens
"What is the boiling point of water?" → "The boiling point of water is the point at which the water boils. …"
→
2. SFT
Supervised fine-tuning : more training on examples of good answers, written by people
→
3. RLHF or DPO
People pick the better of two answers, and the model learns their preference: by reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO)
→
Result: a chat model
The same network, now answering
"In one sentence, what does a heat exchanger do?" → "A heat exchanger transfers heat from one medium to another."
Still how it is done: each round of Llama 3's post-training was SFT, then DPO (Llama Team, 2024 ). InstructGPT "hired a team of about 40 contractors" to write and rank answers (Ouyang et al., 2022 ). Examples: GPT-2, a pretrained model, and Qwen2.5-0.5B-Instruct, a chat model, both run by us.
Systems and Toolchains for AI Engineers
Sampling
Systems and Toolchains for AI Engineers
Sampling, temperature
Temperature : a knob for how adventurous the model is when it picks the next token.
Low : it almost always takes the most likely token.
High : less likely tokens get a real chance.
z: the logits The model's raw score for each token, before the softmax turns them into probabilities
τ (tau): the temperature Divide every logit by τ, then softmax. τ = 1: the model's own probabilities
Two logits, 2 and 1. τ = 1: 73% and 27%. τ = 0.5 doubles the gap: 88% and 12%. τ = 2 halves it: 62% and 38%.
We ran GPT-2 on "…the water was too": "␣hot" gets 88% at τ = 0.5, 42% at τ = 1, 4% at τ = 2.
"Using a higher value for T produces a softer probability distribution over classes" (Hinton, Vinyals and Dean, 2015 ).
Systems and Toolchains for AI Engineers
Sampling, spin the wheel
What you see: GPT-2's next token after "…the water was too" as a wheel, one slice per token. Picking a token is a spin. Change τ, then spin
τ changes the wheel, the spin stays fair: at τ = 0.5 "␣hot" takes 88% of it; at τ = 2, only 4%
Why "temperature": the softmax is Boltzmann's distribution from thermodynamics.
A logit
"In statistical mechanics, the softmax function is known as the Boltzmann distribution" (Wikipedia , after LeCun et al., 2006 ).
Systems and Toolchains for AI Engineers
Sampling, the loop with a temperature knob
What you see: GPT-2's ten most likely next tokens at each step of the loop. Red: the one appended. Step through, then move τ
τ = 1: GPT-2's own probabilities. At step 1, "␣water" gets 3.7%: many words could follow
Slide τ to 0.1: "␣water" jumps to 89% . Low temperature: nearly the same text every time
Slide τ to 2: every bar flattens. At step 4, even "␣hot" falls to 4%: almost anything can come next
Systems and Toolchains for AI Engineers
Sampling, our tiny GPT at three temperatures
τ = 0.3 A Gaussian process in the data and a different statistical models.
### A file section and layers
A **fitted of stead of a column store and a strain of the same data is the same offf the same way th … Safe and repetitive: "the same … the same …", and a Markdown heading copied from our notes
τ = 0.8 A Gaussian process and the default does not exact the band file coefficients the wide, vector plant. The discipline for a strong by an Lecture 9.
The log quantition and can we course at have you been … Varied words in our style, with no meaning
τ = 1.5 A Gaussian process cructor arNive," and a brieful mini-bassed flrecash uncaged defaularing most-is carryied outue: chan `y[desy''](h✓dds#p.01├x, lipproduce.
- [Slagáni lipyndbof/Jagimaus.ipynb) answer … Broken words and rare symbols (✓, ├): the tail of the distribution
All three start from the same prompt, "A Gaussian process", and the same model. Only τ changed.
Systems and Toolchains for AI Engineers
Hallucination
Hallucination : fluent, confident text that is false: an invented number, unit, citation or fact.
What is the boiling point of water at sea level? The boiling point of water at sea level is 100 degrees Fahrenheit.
After "…100 degrees", the next token:
␣Fahrenheit 58.6%
␣Celsius 40.8%
␣F 0.3%
What happened: we ran a chat model on a laptop (Qwen2.5-0.5B-Instruct, 0.49 billion parameters, trained on 18 trillion tokens) and asked. It got the number right and the unit wrong: after "…100 degrees" it gave Fahrenheit 58.6% and Celsius 40.8%, and took the more likely one
Why: the training loss rewards the likely next token. Nothing in it checks whether a sentence is trueSo: treat every number and unit from an LLM as unchecked. Check it against a measurement, a handbook or a calculation
Systems and Toolchains for AI Engineers
Limitations and trade-offs
What goes wrong
What to do
Invents sources and packages. 5.2% of packages in GPT models' code do not exist; 21.7% for open models (
Spracklen et al., 2025 )
Check that each source and package exists before you cite or install it
Coding agents act on their guesses: they edit files, install packages and run commands
Review the diff and run the tests before you merge; keep secrets out of reach
Knows only its training data: nothing after its cutoff date, nothing about your plant
Put the documents it needs in the prompt. Do not ask it for current values
Send only what matters: retrieve the relevant parts, ask it to quote its evidence, repeat the main instructions
Numbers and units: it reads 4140 as two text pieces, "414" and "0", never as a quantity
Do arithmetic and unit conversions in code
Not reproducible: the same prompt can give another answer, even at temperature 0
Save the output you used, with the model's name and the date
Your prompt leaves your machine for the provider's servers
Check your company's rules first, or run an open model locally
Systems and Toolchains for AI Engineers
Recap
→
→
Transformer →
→
Sample, append, repeat ␣water ␣was …
temperature τ
A generative model of text
One token at a time, like Lecture 8's recursive forecast
Trained by
At scale
Billions of parameters, trillions of tokens
Fluent is not true
␣about 1,000 degrees Fahrenheit
GPT-2 on water's boiling point (true: 212 °F). Check every number that matters
Systems and Toolchains for AI Engineers
Resources
Systems and Toolchains for AI Engineers
Before next time
Final project : released today. Team agreement due Friday 10-30, proposal due Wednesday 11-04Practice module for this session, for participation credit
Systems and Toolchains for AI Engineers
Picture a movie script with the AI's reply torn off, and a machine that predicts the next word. Run it again and again and the reply writes itself. That is all a chatbot is. The model was trained on chats written with these tags, so it expects them. How this was made: Hugging Face transformers, the model taking its most likely token at every step; the numbers come from figures/make_figures.py (group chat).
GPU: graphics processing unit, the graphics card. 116 s on an Apple-silicon laptop. On Colab, choose a GPU runtime.
The chain rule is exact: no assumption about which earlier tokens matter. ␣ marks a space: GPT-2's tokens carry their leading space. The product is tiny because so many texts are possible; "pump" right after "The" is the surprise, 8 in a million.
How this was made: we ran GPT-2 (124 million parameters, Hugging Face transformers) on "The pump stopped because the" and, five times, took its single most likely next token and appended it. The probabilities on the cards are GPT-2's. The code: figures/make_figures.py, group gpt2. Click once per token: each click is one trip round the loop. The Sampling section steps through the same loop with GPT-2's real probabilities, and a temperature slider.
The analogy is between Lecture 8's autoregressive forecaster and an LLM writing text: same word, same idea. It is not a claim about RNNs, nor that LLMs lose accuracy over the years. Bengio et al. (2015) described the same effect for RNN text generators: trained on true previous tokens, run on their own, "errors that can accumulate quickly along the generated sequence".
Step 2 is a tie: "aa" + "a" also appears twice. Wikipedia's example takes "a" + "b". A real tokenizer learns its merges once, from a large text, and then applies them in the same order to any new text: the next slide.
The first merges our notes learn: " t", " a", "he", "in", "re". More merges, longer tokens, fewer of them.
Attention: the vectors talk to each other. MLP: each vector, alone, through the same small network. Repeat. Only the last vector predicts the next token. The next two slides open the two boxes: the attention equation and the final softmax.
How this was made: we trained the tiny GPT for 2,000 steps and saved its guesses every 250 steps; the bars and the chart are those checkpoints. The dials are a drawing.
The reactor of Lecture 15 had 2 knobs and hours per experiment: Bayesian optimization. Our tiny GPT has 842,881 knobs and 58 ms per step: gradient descent. The map is Lecture 15's, with the tiny GPT added.
"Typical" is the geometric mean of the probabilities given to the true characters. The blue curve, the training batches, ends lower (1.22) than the red, held-out text (1.51): it predicts text it has seen better than new text, the overfitting of Lecture 9.
Random symbols, then letters and spaces, then short words, then the course's vocabulary, then LaTeX copied from our notes. It learns the style long before any meaning.
The GPT-4 technical report "contains no further details about the architecture (including model size)". Anthropic's model cards give no parameter counts. The mixture-of-experts models (DeepSeek, Kimi) only use a fraction of their parameters for each token: the hollow dots.
GPT-3's reading time uses our notes' 0.58 words per GPT-2 token (GPT-3 uses the same vocabulary) for its 300 billion tokens. GPT-3's compute is Table D.1 of Brown et al. (2020); ours is about 6 operations per parameter per token, 6 × 842,881 × 16.4 million. Same objective and same kind of model: Llama 3 has about 500,000 times the parameters of ours.
For chemical engineers this is exactly the Boltzmann distribution, with the logit as minus an energy in units of k_B T.
The first two rows go together: an agent that installs what it writes can install a package an attacker registered under a hallucinated name (Spracklen et al. describe this attack).