Per-task fine-tuning with Gemma: what jevlike does, what openjev must add¶
Written 2026-09-16. Companion to one-pass-option-scoring.md.
Today openjev answers System One questions zero-shot: it renders a prompt and reads Gemma's
next-token likelihoods. That gives the right decision often but meaningless probabilities
(see the README comparison: 1.00 where Jev says 0.84). Turning it into a per-task, calibrated
scorer means training something on labelled examples. This document lists exactly what has to
change, task by task, and what can be copied from jevlike.
1. What jevlike trains, and what it does not¶
| Piece | jevlike (PyTorch) | Status in openjev (MLX) |
|---|---|---|
| Encoder | Frozen AutoModel (default Qwen 0.5B) or a tiny byte encoder |
Gemma 3 4B via mlx-lm, used only through its LM head |
| Features | Context: all token hidden states. Option: mean-pooled hidden states | Not extracted; only logits are read |
| Head | One cross-attention layer, width --rank: option vector queries context tokens, dot product gives a logit per option |
None |
| Loss | Listwise softmax cross-entropy over the options of one example | None |
| Data | JSONL {"context", "options", "label"} |
Same format accepted by openjev eval |
| Calibration | Falls out of cross-entropy; measured as ECE in jevlike-eval |
None |
| Caching | None: encoder re-run every epoch | Prefix-cached inference only |
| Checkpoint | Head weights + encoder name | n/a |
Everything below is the delta.
2. Feature extraction from Gemma (new)¶
Add a hidden_states=True path to OptionScorer that returns the final-layer hidden state
for every token instead of logits. In mlx-lm this is model.model(tokens, cache=cache)
before the LM head; shape (batch, tokens, 2560) for Gemma 3 4B.
- Reuse the prefix-shared batched pass: context once, options batched. The features come from the same forward pass that scoring already does, so extraction is not slower than inference.
- Pool options by masked mean, as jevlike does. Keep all context tokens.
- Encode options on their own, not as continuations of the context. If option tokens
attend to the context (prefix-shared cache), each pooled option vector already contains the
match: the head reaches 100% in one epoch and the shuffled-context control also reads 100%,
so nothing can be verified. Standalone options cost the same and keep the control honest.
(Implemented:
openjev features, with--contextualas the leaky alternative.) - Cache to disk as
.npzper example:context_h (Lc, 2560),options_h (N, 2560),label. One pass over the dataset, then head training never touches Gemma again. At about 100 ms per example, 5,000 examples take under 10 minutes. - Which layer: start with the last. If the head under-performs, try a middle layer (roughly two thirds of the depth); intermediate layers often carry better task features than the final, LM-tuned one.
- Decide
-itvs-pthere, once. Base weights usually give softer, more general features.
3. The head (port from jevlike, ~60 lines of MLX)¶
Same architecture, in mlx.nn:
LayerNorm(context), LayerNorm(options) # fp32
q = Linear(2560 -> rank)(options) # one query per option
k, v = Linear(2560 -> rank)(context tokens)
attn = softmax(q @ k.T / sqrt(rank), masked by context padding)
ctx_vec = attn @ v # (N, rank), one per option
logit_i = (q_i * ctx_vec_i).sum() / sqrt(rank) # (N,)
rank 256 worked for Qwen in jevlike; start there for Gemma. Padded option slots get
-inf. Trainable parameters: about 2 M, so it fits in any memory and trains in seconds per
epoch on cached features.
4. Per-task targets: one head per question type¶
The System One API has three question types. Each needs its own target and loss, but they share the encoder features and the head architecture. Train one head per task (or per question id when a task has its own labelled set); the encoder pass is shared.
| Question type | Training example | Head output | Loss | Answer fields |
|---|---|---|---|---|
choice |
state + option names/descriptions, label = index |
one logit per option | listwise cross-entropy | choice = argmax, probabilities = softmax, confidence from the distribution |
score |
state + ordered level texts, label = level index |
one logit per level | cross-entropy over levels (ordinal variant: CORAL or squared error on the expected index) | score = sum(i * p_i), probabilities, legend |
noul |
state + true/false descriptions, label in |
one logit for "yes" | binary cross-entropy | noul = sigmoid |
Notes per type:
- choice: this is exactly jevlike's setting. Options are the rendered option name plus description, so the head sees the same text the zero-shot prompt shows.
- score: levels are options too, so the same head applies. Plain cross-entropy trains
well; if you need the score to respect ordering, add a small penalty on
|expected_index - label|. - noul: the two options are the
trueandfalsedescriptions (or the literals "yes" and "no" when no criteria are given). Binary cross-entropy on the "yes" logit.
Rendering must be identical at train and serve time. Factor the prompt/option renderers
in systemone.py so the feature extractor and the endpoint call the same functions.
5. Training loop (port train.py, drop the encoder)¶
- AdamW, weight decay 1e-4, grad-norm clip 1.0, 8 epochs, batch 64, as in jevlike, but lr 5e-4, not 2e-3: Gemma's final hidden states have norms around 115 and the higher rate diverges on some seeds (loss spikes to about 75 in epoch 1, val top-1 stuck at chance).
- Batches are padded feature tensors from the cache, not text.
- Validation each epoch; keep the best checkpoint.
- Checkpoint: head weights as
.safetensorsplus a JSON config with encoder name, layer, rank, task type, and the renderer version. The encoder is not stored.
6. Evaluation (port eval.py)¶
Report what jevlike reports so results are comparable:
- top-1 and top-3 accuracy;
- shuffled-context control: score each option set against a different example's context. If accuracy does not drop, the head is reading option priors, not the state;
- ECE (10-bin expected calibration error) so
confidenceandprobabilitiesmean something. This is the number that separates a trained head from zero-shot softmax.
Add these to openjev eval so zero-shot and trained heads are compared with one command.
7. Serving¶
/v1/systemone gains a mode per question, or a server-wide default: zero-shot (today) or
head:<name>. In head mode the endpoint runs the same feature pass, applies the loaded head,
and fills the answer fields from the head's softmax. Latency is unchanged: one prefill of the
state plus one batched pass over the options, then microseconds for the head.
8. When to unfreeze Gemma (Route C)¶
Only after a head on frozen features plateaus and you have thousands of examples. mlx-lm has LoRA built in. Apply to attention and MLP projections, backbone lr about 10x lower than the head, warmup, and the same listwise loss. Keep the frozen-head checkpoint as the baseline you must beat.
9. Data you need¶
Per task, a JSONL of {"context", "options", "label"} where context is the rendered
state plus instructions and options are the rendered option texts. Rough guide from
jevlike's numbers: a few hundred examples show whether the signal is there, a few thousand
give a usable head. Labels can come from human review, from downstream outcomes (which
department actually resolved the ticket), or from a strong model used as a judge, in which
case the head distils that judge into a 100 ms local scorer.
10. Order of work¶
- Feature extraction with disk cache (section 2). Verify shapes on 10 examples.
- Head in MLX plus training loop on the jevlike synthetic set (sections 3 and 5); it must reach about 99% top-1 like jevlike's own head does.
- Eval with shuffled control and ECE (section 6).
- Per-task datasets and heads for choice, score, noul (section 4).
- Serve heads behind
/v1/systemone(section 7). - LoRA only if needed (section 8).