Lab, Step 1 (part 3): scale your tiny GPT without fooling yourself

A loss of 1.47 sounds like a result. It isn’t, not by itself. Which tokeniser? How many tokens seen? On what compute? Strip those away and the number is a mood.

Parts 1 and 2 gave you a model and a loop that behaves. Part 3 is about the habit that separates a lab exercise from engineering: never read a loss without knowing what it cost and what it counts.

flowchart LR
  T[BPE tokeniser] --> D[tokens D]
  S[size dial N] --> C["compute ~ 6 x N x D"]
  D --> C
  C --> L[loss at that compute]
  L --> F[fit curve, pick next run]

Do it (about 90 minutes)

Same tiny GPT and loop as parts 1 and 2. Three changes: the tokeniser, the size dial, the bookkeeping.

Rung 9: a BPE tokeniser (25 min)

Byte-pair encoding, introduced for subword units in translation by Sennrich et al. (2015), starts from raw units and repeatedly merges the most frequent adjacent pair into a new token. Byte-level BPE starts from the 256 byte values, so nothing is ever out of vocabulary. Karpathy’s minbpe (MIT) is the readable reference: a BasicTokenizer, a RegexTokenizer that splits text by category first so merges don’t cross letters, digits and punctuation, and a GPT4Tokenizer that reproduces tiktoken’s cl100k_base. Its example: with “aaabdaaabac” and 3 merges, aa becomes 256, ab 257, and their pair 258, giving [258, 100, 258, 97, 99].

from minbpe import RegexTokenizer   # clone minbpe next to your notebook

tok = RegexTokenizer()
tok.train(open("input.txt").read(), vocab_size=1024)   # 256 bytes + 768 merges
ids = tok.encode("To be, or not to be")
assert tok.decode(ids) == "To be, or not to be"
print(len(ids))

Re-tokenise your corpus, save the ids as an array, and set vocab_size and block_size in your model config. Two checks before training: the encode/decode round trip is exact, and the untrained loss is near ln(vocab_size) (you used the same check for characters in part 1).

For a ready-made vocabulary, nanoGPT’s OpenWebText prep uses the GPT-2 BPE via tiktoken (50,257 tokens), stored as uint16 ids. With that vocabulary the embedding table alone is 50,257 x 768, about 38.6M parameters, roughly 31% of GPT-2 small’s 124M. At small scale the vocabulary is a big slice of your model.

Rung 10: three sizes, one dial (30 min)

Pick three configurations that differ by roughly 2–3x in parameter count, changing depth and width together (nanochat’s lesson: one dial, the rest derived). For example:

small:  {n_layer: 4,  n_head: 4, n_embd: 128}
medium: {n_layer: 6,  n_head: 6, n_embd: 192}
large:  {n_layer: 8,  n_head: 8, n_embd: 256}

For each, count parameters (sum(p.numel() for p in model.parameters())), and train with the same loop, seed and data. Use the schedule from part 2, scaled to each run length. Write down the loss at fixed token counts, not fixed steps, so the three runs line up on the same x-axis.

Rung 11: price it with 6ND (15 min)

The Chinchilla analysis uses the standard approximation that training FLOPs are about 6 x parameters x tokens: 2ND for the forward pass and twice that for the backward. nanoGPT’s scaling-laws notebook checks it against the exact count for six of the paper’s models and finds the ratio between about 0.99 and 1.10. Good enough for planning, not for billing.

Fill in your three runs. Do the estimate before you train, then compare with wall-clock time. If you’re off by more than 3x, find out why: data loading, a small batch, or a GPU you didn’t know was shared.

Rung 12: plot loss against compute (20 min)

On log-log axes, plot val loss against 6ND for your three runs. If your sizes and token counts are sensible, you get the start of a power-law-looking curve and, more usefully, the point where a bigger model stops paying for itself at your budget.

Expect noise. At this scale the curves won’t be clean, and Kaplan’s neat power laws were measured over many orders of magnitude, not three tiny models. If your points are scattered, say so. A fit you’d bet on needs more runs than a Sunday allows.

The shape to notice: the small model flattens early, the big one starts worse and overtakes late. Which one is cheapest for a given loss depends on where you stop.

What to observe

• Where each model’s curve flattens, and whether the flat part is the model running out of capacity or the data running out of variety.
• Whether training longer or growing the model buys more loss per unit of compute at your budget.
• How much of the small model’s parameter count is the embedding table.
• The gap between your 6ND time estimate and the real clock.

How you know the step is (nearly) complete

Without looking at the code, you can:

• Train a byte-level BPE tokeniser and say why the round trip must be exact.
• Explain why losses from two tokenisers can’t be compared directly.
• Estimate training FLOPs and GPU-hours for a run with 6ND and a utilisation guess.
• State Chinchilla’s rule in one sentence, and one reason real teams train past it.

Part 4, if the author hasn’t marked the step done, covers ablations: removing one architectural piece at a time and measuring what it was worth.

Try this

Take any model you’ve sized recently, for training or for serving, and write down four numbers: parameters, training tokens, the tokeniser, and the cost per million served tokens. If you can’t fill one in, that’s the gap to close before the next “should we go bigger” meeting.

Half a ladder rung is still progress. The curves will wait.