Deep Dives · Every Sunday
LabLab · Step 1, part 4
Part 4 of Step 1: ablate a tiny GPT one component at a time, measure the damage against a seed-noise baseline, and learn which pieces are load-bearing and which are decoration at this size.
LabLab · Step 1, part 3
Part 3 of Step 1: swap characters for a BPE tokeniser, grow the model in steps, and read loss against compute using the Chinchilla and Kaplan scaling papers as the yardstick.
LabLab · Step 1, part 2
Part 2 of Step 1: read nanoGPT's train.py line by line, then rebuild its warmup, cosine decay, weight-decay groups, gradient clipping and eval loop around your own tiny GPT, and break each one on purpose.
LabLab · Step 1, part 1
Step 1 of a public research ladder: grow a character-level GPT from a bigram table to a small transformer, watching the loss at each rung, with nanoGPT as the reference.