Getting started¶
Requirements¶
- Windows or Linux for PyTorch inference (CPU or a supported GPU), or macOS on Apple silicon for MLX/Metal. Head training and feature extraction require MLX.
- uv for the Python 3.12 environment.
- A Hugging Face login with access to
google/gemma-3-4b-it, which is a gated repository.
Windows and Linux¶
From the repository root, these commands work in PowerShell and Linux shells:
uv sync
uv run hf auth login
uv run hf download google/gemma-3-4b-it --local-dir models/gemma-3-4b-it
uv run openjev serve --backend torch --device auto --port 8000
Accept Google's Gemma license on Hugging Face before downloading. You can instead
pass --model google/gemma-3-4b-it to load from the Hugging Face cache. No Make or
virtual-environment activation is required.
GPU setup¶
Install your GPU driver and the appropriate PyTorch build using the
official PyTorch installer. Run its
installation command inside this project's environment, replacing pip3 install
with uv pip install. Use uv run --no-sync afterward to preserve that build.
uv run --no-sync python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
uv run --no-sync openjev serve --backend torch --device cuda --batch-size 2
uv run --no-sync openjev check --backend torch --device cuda
--device auto chooses CUDA, then MPS, then CPU according to availability.
--device cuda requires a working GPU installation and reports an error otherwise.
Use cuda:1 to select another GPU or cpu to force CPU inference. Supported AMD
ROCm installations on Linux also use PyTorch's cuda device name.
Allow roughly 8 GB for the 4B model's 16-bit weights, plus memory for activations and option caches. Reduce batch size or context length if GPU memory is tight. CPU inference uses float32 and needs more RAM. PyTorch requires original Hugging Face weights; MLX quantized checkpoints and MLX LoRA adapters are unsupported.
Score and verify¶
uv run openjev score --context "The capital of France is" --option " Paris" --option " Berlin"
uv run openjev check
uv run openjev bench
Use uv run --no-sync in these examples if you installed a custom GPU build.
Scoring, evaluation, benchmarking, correctness checks, /score, and
/v1/systemone support both backends. Feature extraction, head training/evaluation,
and chess LoRA training remain MLX workflows on Apple silicon. The Doom terminal
UI has separate platform dependencies.
Apple silicon setup¶
Xcode licence
make on macOS needs the Xcode licence accepted
(sudo xcodebuild -license accept), or use Homebrew's gmake.
Setup¶
make setup # uv sync (arm64 Python 3.12 venv) + download google/gemma-3-4b-it into models/
make setup runs two targets:
| target | what it does |
|---|---|
make venv |
uv sync --extra torch into .venv/ |
make model |
hf download google/gemma-3-4b-it --local-dir models/gemma-3-4b-it if it is missing |
The optional torch extra is installed because make setup asks for it; it is
used only for the jevlike comparison and Hugging Face cross-checks, not for
scoring.
Both the model directory and the Hugging Face repo id are overridable:
make setup MODEL=models/my-gemma HF_REPO=google/gemma-3-4b-it
First run¶
Rank three options for one context:
.venv/bin/openjev score --context "The capital of France is" \
--option " Paris" --option " Berlin" --option " Lyon"
Each option prints with its probability, normalised score, raw log-probability sum, and token count. Note the leading space inside each option: options carry their own leading whitespace rather than relying on a separator.
Check that the prefix-cached batched path agrees with naive re-encoding, then benchmark it:
make check # verify cached batched scoring against naive re-encoding
make bench # latency benchmark
Run the server¶
make serve # = .venv/bin/openjev serve --host 127.0.0.1 --port 8000 --model models/gemma-3-4b-it
In another terminal:
make health # curl /health
make request # example curl against /score
make systemone # TypeSafe-style request against /v1/systemone
POST /score takes a context and a list of options:
curl -s localhost:8000/score -H 'content-type: application/json' -d '{
"context": "Customer: my order arrived broken. Agent:",
"options": [" I am sorry, I will send a replacement.", " Please read our returns policy."],
"norm": "mean", "chat": false, "sep": ""
}'
The response has best, best_index, per-option probability /
logprob_sum / n_tokens, and timing.
TypeSafe System One¶
POST /v1/systemone implements the request/response shape documented at
docs.typesafe.ai: a state (string, object or
array) plus a map of typed questions, answered against that state.
make systemone # sends examples/systemone-quickstart.json
question type |
request criteria |
answer fields |
|---|---|---|
choice |
map of option name to description (string, object, array or null); up to 255 options | choice, probabilities (sum to 1), confidence |
score |
ordered array of level descriptions | score (probability-weighted mean of level index), probabilities keyed "0".."n-1", legend, confidence |
noul |
optional {"true": ..., "false": ...} |
noul = probability of yes |
The response is
{"model", "answers": {id: answer}, "usage": {"input_tokens", "output_tokens"}}.
Set OPENJEV_API_KEY before make serve to require
Authorization: Bearer <key> on this route.
Each question is rendered to a plain-text prompt (state, instructions, options
or levels, then Answer: and a newline), and the option names, level numbers
or yes/no are scored as continuations in one prefix-shared batched pass per
question. confidence is 1 minus the normalised entropy of the distribution;
TypeSafe does not publish its formula, so treat it as an approximation.
usage.output_tokens counts the candidate-label tokens that were scored.
Zero-shot probabilities are not calibrated
On the docs' quick-start request, Gemma 3 4B zero-shot reaches the same routing decision as Jev but is markedly over-confident and disagrees on judgement calls. Softmaxed next-token likelihoods are not calibrated judgements — closing the gap means labelled data and a trained head.
Evaluate on your own data¶
make eval DATA=data/synthetic/test.jsonl
Rows are jevlike-style JSONL: {"context": ..., "options": [...], "label": 0}.
See the CLI reference for the full set of flags.
Cleaning up¶
make clean # remove runs/ and __pycache__, keeping the venv and the model
Next steps¶
- CLI reference for every subcommand.
- Training a head when zero-shot is not accurate or calibrated enough.
- Design notes for why this works.