Exercises — overview

A set of self-study exercises that rebuild a complete GPT-2 training system from the ground up, starting with tokenisation and ending with multi-GPU training. Each one is a layer of the stack; later ones depend on earlier ones.

The codebase is llm.c by Andrej Karpathy. Function bodies have been replaced with // TODO (A?) stubs, each followed by a // HINT: line. The job is to fill them in, and ./grade tells you whether you got it right.

The exercises

Exercise Topic GPU? Files
A0 Data Pipeline and PRNG no llmc/rand.h, llmc/tokenizer.h, llmc/dataloader.h, llmc/sampler.h
A1 GPT-2 Forward Pass (CPU) no train_gpt2.c
A2 Backpropagation and AdamW (CPU) no train_gpt2.c
A3 CUDA Kernels 1 GPU llmc/encoder.cuh, llmc/layernorm.cuh, llmc/attention.cuh, llmc/matmul.cuh, llmc/gelu.cuh, llmc/fused_classifier.cuh, llmc/adamw.cuh, train_gpt2.cu
A4 Multi-GPU with NCCL/ZeRO-1 2+ GPUs llmc/zero.cuh, train_gpt2.cu
A5 Training Infrastructure 1 GPU llmc/schedulers.h, llmc/logger.h, llmc/outlier_detector.h, llmc/global_norm.cuh, train_gpt2.cu

Each file appears in exactly one exercise, and every TODO tag in a file matches it. train_gpt2.cu is the one exception — it is touched by A3, A4, and A5, but each of its stubs carries its own tag, so grep "TODO (A4)" train_gpt2.cu gives you exactly that part.

Learning Objectives

By the end you will be able to:

  1. Explain and implement the GPT-2 transformer architecture from first principles
  2. Derive and implement backpropagation for each component of the network
  3. Write CUDA kernels that approach hardware-peak memory bandwidth using vectorized loads and warp-level reductions
  4. Scale training across multiple GPUs using NCCL AllReduce and ZeRO-1 optimizer sharding
  5. Build production training infrastructure: LR schedules, gradient clipping, structured logging, and bit-exact checkpoint resume

Prerequisites

  • Fluency in C (pointers, arrays, structs, malloc/free)
  • Linear algebra: matrix multiply, dot products
  • Calculus: partial derivatives, chain rule
  • A0, A1, and A2 need no GPU. A3 and A5 need one NVIDIA GPU (RTX 3080 or better recommended); A4 needs at least two, plus NCCL. Without the hardware, ./grade reports those checks as SKIP rather than failing them.

Environment Setup 

# from the repo root

# Python deps, for data prep and generating the PyTorch reference
pip install -r requirements.txt

# Fetch the GPT-2 checkpoint, tokenizer, debug state, and tinyshakespeare in one go
chmod u+x ./dev/download_starter_pack.sh && ./dev/download_starter_pack.sh

# ...or regenerate everything yourself:
#   python dev/data/tinyshakespeare.py
#   python train_gpt2.py

# CPU build (A0-A2)
make train_gpt2
make test_gpt2
./test_gpt2

# CPU-side unit tests (A0 dataloader, A5 outlier detector)
cd dev/test && make && ./test_dataloader && ./test_outlier_detector

# CUDA build (A3-A5), requires nvcc
make train_gpt2cu
make test_gpt2cu
./test_gpt2cu

All builds must be warning-free. If make prints ✗ OpenMPI is not found, MULTI_GPU is undefined and A4’s code is not compiled at all.

Command-line flags

train_gpt2cu uses single-character flags, not GNU long options — the parser reads argv[i][1]. --input-bin, --num-steps, --zero-stage and friends do not exist and will be ignored. Run the binary with no arguments to print the full list. The ones you will use most:

Flag Meaning
-i / -j train / val data shard glob
-b / -t micro-batch size B / sequence length T
-d total batch size in tokens (drives gradient accumulation)
-x total steps
-v validate every N steps
-o output log / checkpoint directory
-n / -y checkpoint every N steps / resume from newest checkpoint
-l / -u / -q / -k peak LR / warmup / final LR fraction / scheduler type
-z ZeRO stage

Checking your work

One command checks your work against the scoring table in each exercise:

./grade A0

Run it with no arguments (./grade, or make grade) to score everything that can be checked on your machine.

A0 — Data Pipeline and PRNG
  PASS Code compiles cleanly (no warnings)                             10 / 10
  PASS randint32: golden vectors (seeds 137, 42) + twist boundary      10 / 10
  PASS randfloat32 golden vector (24-bit mantissa)                      5 / 5
  FAIL sample_softmax + xorshift* PRNG                                  0 / 10
  SKIP Tokenizer init, versions, decode                                -- / 15   gpt2_tokenizer.bin missing
  ...
  subtotal                                                        60 / 85   (15 points not checkable here)

Three things to know about it:

  • SKIP is not zero. A check that cannot run here — no GPU, no NCCL, a missing .bin file — is excluded from the denominator, not failed. The summary lists everything that was skipped and why.
  • It cannot check everything. Performance targets (A3 MFU) and the multi-node init paths depend on hardware and setup the script cannot inspect; those stay SKIP and are yours to verify with ncu and the scripts in scripts/multi_node/.
  • It builds the test harnesses at -O0 with ASan/UBSan. At -O3, an unimplemented function leaves buffers uninitialised, the reads become undefined behaviour, and the compiler is entitled to delete the very checks being graded. That shows up as a false pass, so the grader does not use it.

The lines it prints correspond one-to-one with the Scoring table in each exercise. Where the harness genuinely cannot separate two functions, the line says so — see A2’s (inferred) rows.

Assignment Tag Reference

Each stubbed function is tagged with its assignment, followed by a hint:

void encoder_forward(float* out, int* inp, float* wte, float* wpe, int B, int T, int C) {
    // TODO (A1) - Add token and position embeddings into the output tensor
    // HINT: out[b,t,c] = wte[inp[b*T+t]*C+c] + wpe[t*C+c]
}

To find all functions for a given assignment:

grep -rn "TODO (A3)" llmc/ train_gpt2.cu

Where this handout and the in-file HINT disagree, the code is authoritative. Please report any discrepancy — the handouts are checked against the source, but errors survive.

Testing Philosophy

Each assignment is verified numerically against a reference:

Exercise Harness Reference
A0 dev/test/test_dataloader, MT19937 golden vectors PyTorch’s MT19937
A1, A2 ./test_gpt2 gpt2_124M_debug_state.bin, exported from PyTorch
A3 ./test_gpt2cu the same debug state
A4 loss equality across 1/2/4 GPUs your own single-GPU run
A5 dev/test/test_outlier_detector, checkpoint resume analytic values / your own run

test_gpt2 and test_gpt2cu compare your analytic gradients against PyTorch’s analytic gradients on a fixed batch. They are not finite-difference checks — there is no (loss(θ+ε) - loss(θ-ε)) / 2ε in the harness.

Tolerances enforced by test_gpt2.c:

Check Tolerance
Logits, element-wise (first V columns only) 1e-2 absolute
Loss, at step 0 and each of 10 steps 1e-2 absolute
Every gradient tensor, element-wise (check_tensor) 2e-2 absolute

The pass condition is the final overall okay: 1. Note the harness checks the model end to end, not layer by layer — there are no [layernorm_forward] OK lines. If you are within tolerance your implementation is correct regardless of the algorithm used (different loop ordering is fine).

Generating the reference yourself requires python train_gpt2.py, which writes gpt2_124M.bin, gpt2_124M_bf16.bin, gpt2_124M_debug_state.bin, and gpt2_tokenizer.bin.

A note on the reference implementation

The original llm.c is public, and reading it while working through these is the fastest way to learn nothing. Three sets of files were removed from this repo for exactly that reason — each contained working implementations of functions the exercises ask you to write:

Removed Contained
train_gpt2_fp32.cu, test_gpt2_fp32.cu reference versions of most A3 kernels, same names
dev/cuda/ (22 files) standalone versions of the A3 kernels, plus global_norm.cu (A5) and nccl_all_reduce.cu (A4)
doc/layernorm/, dev/cpu/matmul_forward.c layernorm_forward, layernorm_backward, matmul_forward

They are all still in upstream llm.c whenever you want them — the point of removing them here is only that you have to reach for them deliberately rather than trip over them.