Building GPT-2 from scratch

A complete GPT-2 training system — tokeniser, forward pass, backprop, CUDA kernels, multi-GPU ZeRO-1, checkpointing — rebuilt one layer at a time, in C and CUDA, with no framework underneath.

Start with A0 All exercises Source on GitHub

---

What this is

llm.c by Andrej Karpathy trains GPT-2 in plain C and CUDA, with no PyTorch at runtime. It is one of the clearest existing answers to the question what is a training loop actually doing? — but reading a finished implementation and being able to write one are very different skills, and reading it is the easy one.

So I took the codebase apart. Every function body that contains a real idea has been replaced with a stub:

void manual_seed(mt19937_state* state, unsigned int seed) {
    // TODO (A0) - Initialize the Mersenne Twister PRNG state from a seed value
    // HINT: Set MATRIX_A constants, then fill state_ array using a linear
    //       congruential recurrence
    ...
}

What is left is the scaffolding: struct layouts, function signatures, the call graph, the build system, the tests. The architecture is given; the substance is not. Filling it back in is the exercise.

Why stubbing, specifically

A from-scratch rewrite spends most of its time on things that teach you nothing — argument parsing, file formats, Makefile incantations, which of your four bugs is the one making the loss curve flat. Stubbing keeps the parts of the system that are merely work and removes only the parts that are ideas. Every hour goes into the fused classifier kernel rather than into plumbing.

It also gives you something a rewrite cannot: a reference that is bit-exact. Your randfloat32 is not “close enough” — it either reproduces PyTorch’s MT19937 stream draw for draw, or it does not.

How it is graded

The repo ships a grader. ./grade A0 compiles your headers, runs them against golden vectors captured from the reference implementation, and scores each function separately:

A0 — Data Pipeline and PRNG
  PASS Code compiles cleanly (no warnings)                    10 / 10
  PASS rand.h  manual_seed()      seeded state + counters       2 / 2
  PASS rand.h  next_state()       twist, wraparound, last word  3 / 3
  FAIL rand.h  randint32()        golden draws + twist boundary 0 / 4
  ...
  SCORE: 74 / 100

One row per function, in the order the functions appear in the header, because they build on each other — when six rows fail at once, the topmost one is the bug and the rest are downstream. Checks that need hardware you do not have (A4 wants two GPUs) report SKIP rather than FAIL, so the score stays honest on a laptop.

The six exercises

	Topic	GPU	The idea
A0	Data pipeline and PRNG	none	MT19937 bit-exact against PyTorch, BPE decoding, sharded batching
A1	Forward pass	none	The transformer, in C, from the matrix multiply up
A2	Backprop and AdamW	none	Every gradient derived and written by hand
A3	CUDA kernels	1	Vectorised loads, warp reductions, chasing peak bandwidth
A4	Multi-GPU	2+	NCCL AllReduce and ZeRO-1 optimizer sharding
A5	Training infrastructure	1	LR schedules, grad clipping, bit-exact checkpoint resume

A0–A2 run on any laptop. A3 and A5 want one NVIDIA GPU; A4 wants two.

Working through them yourself

git clone https://github.com/anupriyamranjit/llm.c
cd llm.c
pip install -r requirements.txt
./dev/download_starter_pack.sh   # checkpoint, tokenizer, tinyshakespeare
./grade A0                       # 0 / 100, to start with

Then grep -rn "TODO (A0)" and begin.

Solutions

My own solutions live in a separate repository, one branch per exercise, each a single commit on top of the stubs — so a solution reads as a diff against exactly what you start from.

They are kept out of this repo on purpose. Clone this one and the exercises are still unsolved, which is the only thing that makes it worth cloning. Go look at the answers when you want them, not by accident.

---

The underlying implementation, build system and tests are llm.c by Andrej Karpathy, MIT licensed. The stubbing, the exercise text and the grader are mine.