Skip to content

Training a head

Platform requirement

The feature extraction, head training/evaluation, and chess LoRA workflows require MLX on Apple silicon. The PyTorch backend supports scoring and serving on Windows/Linux; it does not port these training workflows or load MLX adapters.

Zero-shot scoring is a good default, but softmaxed next-token likelihoods are not calibrated judgements. When you have labelled rows, you can freeze Gemma and train a small cross-attention head on top of its features — jevlike's Route A, ported to MLX.

The background is in Per-task fine-tuning with Gemma.

The three steps

make features    # one prefix-shared pass per example, cached to runs/feats/{train,validation,test}.npz
make train       # jevlike's cross-attention head in MLX, listwise cross-entropy -> runs/head.safetensors
make eval-head   # top-1/top-3, ECE, and the shuffled-context control on the test split

make features expects $(DATA_DIR)/{train,validation,test}.jsonl (data/synthetic by default) and writes one .npz per split into $(FEATS) (runs/feats). make train reads the train and validation caches and writes $(HEAD) (runs/head.safetensors). make eval-head scores $(HEAD) against the test cache.

1. Cache the features

.venv/bin/openjev features DATA --out F.npz

This stops Gemma before the LM head and keeps two things per example: every context token's final hidden state, and the masked-mean of each option's tokens (float16).

Options are encoded on their own, as in jevlike, so the head has to do the matching itself.

--contextual invalidates the control

--contextual encodes options as continuations of the context instead. The features are stronger, but the match leaks into the option vectors and the shuffled-context control below stops meaning anything.

2. Train the head

.venv/bin/openjev train runs/feats/train.npz \
    --validation runs/feats/validation.npz --out runs/head.safetensors

The defaults are AdamW at 5e-4, weight decay 1e-4, gradient clipping at 1.0, 8 epochs, batch 64, keeping the best validation epoch. The objective is listwise cross-entropy over each row's options.

Why 5e-4

2e-3 diverges on Gemma features, whose norms are around 115.

The checkpoint is head.safetensors plus a head.json recording the rank, hidden size and training config.

3. Evaluate, and check the control

.venv/bin/openjev eval-head runs/head.safetensors runs/feats/test.npz

This reports what jevlike-eval reports: top-1, top-3, 10-bin expected calibration error, and the same metrics with every example paired with another example's context.

The shuffled-context control is the important one. If the shuffled number does not collapse toward chance, the head is reading option priors rather than the state, and the headline accuracy is not measuring what you think it is.

Reference result

On the synthetic split (2000 train / 400 validation / 400 test, 2026-09-17). Feature extraction ran at 77 ms per example; head training took about 15 s for 8 epochs.

top-1 top-3 ECE
head on frozen Gemma features, test 0.970 1.000 0.027
same head, shuffled-context control 0.258 0.698 0.724
jevlike tiny byte encoder, test 0.998 1.000 0.008
Gemma zero-shot, --norm sum, test 1.000 1.000 n/a

The control sits at chance (about 4.5 options per row), so the head really does match options to the state, and its ECE of 0.03 is what a calibrated confidence looks like.

Zero-shot or trained?

The synthetic task above is easy for both, so it does not settle the question. The real validation is your own labelled rows: run openjev eval on them zero-shot, and compare against a jevlike-train / jevlike-eval run on the same split.

  • If Gemma zero-shot is close to the trained head, Route B (zero-shot likelihood scoring) is enough.
  • If not, train a head with make features && make train && make eval-head.

Validating against jevlike

jevlike can be installed into the same venv (uv pip install -e ../../vinnylarouge/jevlike), so both CLIs read the same JSONL. Generate its synthetic menu set, then score it both ways:

.venv/bin/jevlike-data synthetic --output data/synthetic
.venv/bin/openjev eval data/synthetic/test.jsonl --norm sum --sep $'\nChoice: '
.venv/bin/jevlike-train data/synthetic/train.jsonl --validation data/synthetic/validation.jsonl \
    --output runs/synthetic-tiny.pt --device mps
.venv/bin/jevlike-eval runs/synthetic-tiny.pt data/synthetic/test.jsonl --device mps

Results on the 400-row synthetic test set (2026-09-16):

scorer training top-1 top-3 median latency / example
openjev, Gemma 3 4B zero-shot, --norm sum none 1.000 1.000 0.086 s
openjev, --norm mean none 0.988 1.000 0.086 s
openjev, --norm pmi none 0.988 1.000 0.153 s
jevlike tiny byte encoder + head 2000 rows, 8 epochs 0.998 1.000 well under 10 ms