Skip to content

CLI reference

Backend and device selection

Scoring commands (score, eval, bench, check) and serve accept:

Flag Default Behavior
--backend auto mlx on Apple silicon, torch elsewhere; can be set explicitly.
--device auto PyTorch: CUDA if available, then MPS, then CPU. Accepts cpu, cuda, cuda:N, or mps.

Use uv run openjev in place of .venv/bin/openjev in examples to run them on Windows or Linux. After installing a custom GPU PyTorch build, use uv run --no-sync openjev to retain it.

uv run openjev score --backend torch --device cpu --context "The capital of France is" --option " Paris" --option " Berlin"
uv run --no-sync openjev serve --backend torch --device cuda --batch-size 2

An explicitly selected unavailable GPU produces an error. MLX uses Metal and does not support PyTorch device selection. MLX adapters are unsupported with PyTorch. features, train, and eval-head still require MLX on Apple silicon.

The package installs one entry point, openjev:

openjev {score,eval,bench,check,features,train,eval-head,serve} ...

In a make setup checkout it lives at .venv/bin/openjev.

Common flags

score, eval, bench, check and features all accept the same scoring options:

flag default meaning
--model MODEL models/gemma-3-4b-it local dir or HF repo id
--batch-size N 8 options per forward pass
--norm {mean,sum,pmi} mean how option log-probs are normalised
--chat off wrap the context in Gemma's chat template; options score as the reply
--sep SEP "" string inserted between context and option

serve accepts --model and --batch-size with the same defaults, but none of the other scoring flags.

Normalisation (--norm)

norm score use when
mean (default) sum of option-token log-probs divided by token count options differ in length
sum total log-prob options are the same length or you want raw likelihood
pmi sum minus the option's unconditional log-prob (BOS-only context) options differ in base-rate plausibility; costs one extra batched pass

Let options carry their own leading space

Passing --sep " " makes the tokenizer emit a lone space token, which almost never occurs mid-sentence in training data. Every option then scores around -20 nats and the ranking becomes noise. Put the leading space inside the option string instead (" Paris", not "Paris").


score

Score options for one context.

openjev score [common flags] --context CONTEXT
              [--option OPTION ...] [--options-file OPTIONS_FILE] [--json]
flag meaning
--context CONTEXT required; the context to score against
--option OPTION repeat for each option
--options-file FILE text file with one predefined option per line
--json emit JSON instead of the table
# Rank options for one context (prints probability, score, raw sum, token count)
.venv/bin/openjev score --context "The capital of France is" \
    --option " Paris" --option " Berlin" --option " Lyon"

# Chat template (context as user turn, options scored as the reply) + PMI normalisation
.venv/bin/openjev score --chat --norm pmi --context "..." --option "..." --option "..."

# Predefined options: one per line in a text file, reused for every context
.venv/bin/openjev score --options-file options.txt --context "..."

There is also a Makefile wrapper:

make score CONTEXT="The capital of France is" OPTIONS="--option ' Paris' --option ' Berlin'"

eval

Top-k accuracy on jevlike-style JSONL rows, {"context": ..., "options": [...], "label": 0}.

openjev eval [common flags] [--limit LIMIT] [--fixed-options FILE] [--verbose] data
argument meaning
data positional; path to the JSONL file
--limit N stop after N rows (0, the default, means all)
--fixed-options FILE text file of options used for every row lacking an options field
--verbose per-row output
# Top-1 / top-3 accuracy
.venv/bin/openjev eval data.jsonl --norm mean

# Rows need only {"context": ...}; add "label" for accuracy
.venv/bin/openjev eval contexts.jsonl --fixed-options options.txt

Or through the Makefile, which passes --model and --norm for you:

make eval DATA=data/synthetic/test.jsonl

bench

Latency of prefix-cached batched scoring versus naive re-encoding, on a synthetic workload.

openjev bench [common flags] [--context-tokens N] [--options N]
              [--option-tokens N] [--repeat N] [--tol TOL]
flag default
--context-tokens N 200
--options N 8
--option-tokens N 30
--repeat N 5
--tol TOL 0.5
.venv/bin/openjev bench --context-tokens 200 --options 8 --option-tokens 30
make bench

check

Verify that cached batched scores match naive full-sequence scores. It takes the same flags as bench; --tol is the maximum absolute log-probability difference allowed before the check fails.

.venv/bin/openjev check
make check

features

Cache frozen-Gemma features for a JSONL file into an .npz. See Training a head for how the cache is used.

openjev features [common flags] --out OUT [--limit LIMIT] [--contextual] data
argument meaning
data positional; jevlike-style JSONL
--out OUT required; destination .npz
--limit N stop after N rows (0, the default, means all)
--contextual encode options as continuations of the context (leaks the match into option features)
.venv/bin/openjev features data/synthetic/train.jsonl --out runs/feats/train.npz
make features

train

Train the cross-attention head on cached features.

openjev train --validation VALIDATION [--out OUT] [--rank RANK] [--epochs EPOCHS]
              [--batch-size BATCH_SIZE] [--learning-rate LEARNING_RATE] [--seed SEED] train
argument default meaning
train — positional; the training .npz
--validation FILE required validation .npz
--out OUT runs/head.safetensors checkpoint path
--rank RANK 256 head rank
--epochs EPOCHS 8 training epochs
--batch-size BATCH_SIZE 64 examples per step
--learning-rate LR 5e-4 AdamW learning rate
--seed SEED 7 random seed
.venv/bin/openjev train runs/feats/train.npz \
    --validation runs/feats/validation.npz --out runs/head.safetensors
make train

Note

train reads cached features, not the model, so it does not take the common scoring flags.

eval-head

Top-k, ECE and the shuffled-context control for a trained head.

openjev eval-head checkpoint features

Both arguments are positional: the checkpoint written by train, and the test feature .npz.

.venv/bin/openjev eval-head runs/head.safetensors runs/feats/test.npz
make eval-head

serve

HTTP server with the model loaded once. Exposes GET /health, POST /score and POST /v1/systemone.

openjev serve [--model MODEL] [--batch-size BATCH_SIZE] [--host HOST] [--port PORT]
flag default
--model MODEL models/gemma-3-4b-it
--batch-size BATCH_SIZE 8
--host HOST 127.0.0.1
--port PORT 8000
.venv/bin/openjev serve --port 8000
make serve

Set OPENJEV_API_KEY before starting the server to require Authorization: Bearer <key> on /v1/systemone.


Makefile targets

make help lists them. In full:

target what it does
setup create the venv and download the model
venv uv sync --extra torch
model download $(HF_REPO) into $(MODEL) if missing
serve run the HTTP server
health curl the running server's health endpoint
request example scoring request against the running server
doom play Doom in the terminal, the running server picks every action
systemone TypeSafe quickstart example against the running server
score one-off CLI scoring
check verify prefix-cached batched scores match naive re-encoding
bench latency: cached and batched vs naive
eval zero-shot top-k accuracy on $(DATA)
features cache frozen-Gemma features for $(DATA_DIR)/{train,validation,test}.jsonl
train train the attention head into $(HEAD)
eval-head top-k, ECE and shuffled-context control of $(HEAD)
clean remove caches and run artefacts, keeping the venv and model

Overridable variables: MODEL, HF_REPO, HOST, PORT, NORM, DATA, API_KEY, DATA_DIR, FEATS, HEAD, SEP.