Introducing TOD: A Multimodal Extractive Tool Calling Decision Model

Summary

TOD (Task-Oriented Dialogue) is a decision model for the decisions an agent actually makes: which tool to call, with which arguments, or whether to hand the turn to a bigger model or a human. It reads text and images, and it never generates. The action is selected from the candidate calls and replies in front of it, and each argument is selected from the real entities in the context, so it cannot invent an order id, a catalog item, or a payment method. Every pick carries a calibrated probability that your code can threshold.

On public JevBench, held fully out of training, TOD scores 84.4% (195 of 231). With the image it answers 97% of chart questions, up from 70% for the untrained base model and 79% without the image. Fine-tuned to τ2-bench retail customer support, it matches the parent model’s solve rate at 23–34% lower cost per episode, with each pick taking about 0.22 s instead of the 2.4 s it takes to generate the same call.

Public JevBench

84.4%

of 231 held-out decisions

Easy 100% · original 91.7% · hard 73.0%. Never trained or calibrated on.

Chart questions

97%

correct with the image

70% for the untrained base. 79% for TOD without the image.

Support cost

−23–34%

per episode vs. parent alone

96/114 with Gemini Pro and 100/114 with Flash on τ2-bench retail.

Latency per pick

0.22 s

one discriminative read

Versus ~2.4 s to generate the same tool call.

Most agent decisions do not need a language model, but they do need tool calling

Decision models are small models that produce bounded, typed outputs cheaply, quickly, and consistently, so they can sit in the hot path wherever a workflow needs a decision. Jev and Clef answer questions like “is this ticket urgent?” and “which team should handle it?” with a label and a probability, and that is genuinely useful. But it is not what an agent spends most of its turns deciding. An agent decides to call cancel_order on order W2378156 and refund to the original card, or to look the customer up by zip code before doing anything, or to say nothing and ask. A classifier that returns a category still leaves the actual tool call to a language model, and the tool call is where the latency, the cost, and the hallucinated arguments live.

TOD closes that gap by extraction rather than generation. The action is one pick over the enumerated candidates for the turn: every constructible tool call plus the reply options. If the action is a tool call, each argument is then its own pick over the real entities present in the conversation and the customer record: an actual order id, an actual item, an actual payment method or address. Nothing is decoded token by token, so the chain structurally cannot produce an argument that does not exist. When a pick’s confidence is below the operating threshold, the whole turn is handed to the parent model unchanged.

That shape buys four things. Latency: one discriminative read per pick, about 0.22 s, instead of an autoregressive decode loop, about 2.4 s for the same call. Cost: a flat per-decision rate that does not grow with reasoning or context length, $0.30 per 1,000 decisions against $0.40 for Jev. Calibration: confidences are trained with a proper (Brier) score, and TOD ships at a raw calibration error of 0.8% on JevBench before any temperature correction. Abstention: a calibrated confidence per pick is exactly what you threshold to act or defer.

Results on public JevBench

JevBench-231 is the public decision benchmark: 231 items across three difficulty tiers (easy, original, hard) and the three primitive types (choice, yes/no, score). It is excluded from TOD’s training mix by tier and by an explicit denylist, and temperatures are fit on our own validation rows, never on the benchmark. TOD scores 84.4% (195 of 231): 100% on easy, 91.7% on original, 73.0% on hard, with an expected calibration error of 3.3% (0.8% before temperature correction).

One number we report that nobody would volunteer: the untrained Gemma 4 12B base that TOD is built on, scored with the same readout, gets 86.2%. Training on our decision corpus gave back four items on public JevBench, all in general-reasoning families, in exchange for the things the base model cannot do and the benchmark does not measure: read images, choose among hundreds of options, and report confidence that is honest out of the box. We ship the trained model.

Any number of options

Every open decision model on the board reads its answer as a single letter, which caps a decision at 26 options. The Jev contract allows 255, and real tool slates in a support conversation routinely run past 26 once every constructible call is enumerated. TOD uses a two-stage read: a small trained retriever shortlists the slate, and the 12B scores the shortlist in one forward pass, with the result mapped back onto the full slate. It is exact for small slates and degrades gracefully for large ones.

The public board, the sealed split, and leaked data

Public JevBench is not the whole benchmark. The board operator holds a sealed split of 308 items that no one can train on. On that split, every one-pass decision model collapses toward chance (29.3%). Jev 1.13.0 goes from 86.6% public to 36.7% sealed. Plumb-4B goes from 89.6% to 38.0%, Cygnet from 87.9% to 33.8%, and the class of 45 open one-pass models averages 29%. The same items that look solved in public re-collapse when they are new: the trap family runs about 98% public and 48.5% sealed for one-pass systems, and paraphrase robustness, which is simply rewordings of items the systems have seen, sits at 39.5%.

The most parsimonious explanation is that the public scores were learned from the public items. Several board entries say so outright: one trains on the test splits of the standard benchmarks it is evaluated against, another ships eight thousand rows built to mirror the sealed family names. Jev does not publish its training data, but a model that scores 100% on public trap, adversarial, and hard-routing items and then drops to 41.7% on unseen ones of the same kind has pattern-matched those items, not learned the skill. Its public number should be read as a ceiling on memorization, not a measure of generalization.

The sealed items are not unsolvable. Reasoning language models score 89–96% on them, and the same DiffusionGemma weights double their sealed accuracy, from 29.9% to 60.1%, when allowed to compute at inference time. What collapses is a single pass over memorized patterns. TOD’s training mix excludes JevBench entirely by denylist, and its calibration is fit on our own validation rows, so its public number is clean by construction. TOD has not yet been scored on the sealed split; when it is, that number will not be a contamination artefact.

It reads the image, and training taught it to

Jev is text-only. TOD takes images in the state, which matters for support (a photo of a damaged parcel), for documents (a chart or a form), and for any decision whose evidence is not in the transcript. Two claims need separate evidence: that TOD actually relies on the image, and that the vision capability is something our training produced rather than something the base model had anyway.

First, take the image away. On 100 held-out questions per set, TOD drops from 97% to 79% on charts (ChartQA), from 87% to 62% on diagrams (AI2D), and from 90% to 73% on science questions (ScienceQA). Every 95% confidence interval excludes zero. The model is reading the pixels, not guessing from the surrounding text.

Second, compare against the untrained base model with the image, scored the same way. The base gets 70% on charts, 80% on diagrams, and 83% on science questions; TOD gets 97%, 87%, and 90%. A 27-point gain on charts is not a prompting effect. Brier scores improve on every set as well, so the probabilities on image decisions got more honest, not just more accurate. The adapter was trained on licence-clean decision rows that include image questions; this is the value the training added on top of the base.

How a turn is resolved

In the customer-support deployment, one turn is a chain of picks. A small retriever shortlists the enumerated candidate slate (every reply option plus the top 16 tool calls, capped at 26). A collapse step merges each write with its read-back so they count as one candidate. TOD then scores the shortlist in a single forward pass and maps the chosen option back to the original slate with a calibrated probability.

If the action is a write, an argument chain fills each argument with its own pick, selecting a real order, item, payment method, or address from the slate. A confidence head decides whether the turn is certified; if not, the whole turn goes to the parent model unchanged. Every write is offered once as a read-back (“I’m about to cancel order #W… — confirm?”) and is not fired until the customer’s next reply is scored: yes executes it once, no drops it for the episode, and conditional assent (“only if you refund to X”) counts as not confirmed. The only two decisions TOD never certifies are calls that cannot be constructed and transfers to a human agent.

Selecting arguments beats generating them

On 20 held-out support turns we compared the base model’s native function calling against TOD’s pick plus argument chain. TOD’s only errors were picking the wrong action; when the action was right, its arguments were right 15 of 15 times. Generation’s errors were arguments and policy: hallucinated order ids and item swaps, skipping a look-up before acting, and narrating a number instead of calling a tool. Giving the generator the correct action to work from barely helped, because the failure is in constructing the arguments, not in choosing the tool.

How we trained TOD

TOD starts from the open Gemma 4 12B instruction model. The general decision adapter is a rank-16 LoRA trained for two epochs on 30,360 licence-clean decision rows under our 2026-09-24 data policy, with every benchmark we report on held out by denylist, plus a ModernBERT-base retriever trained with contrastive loss and hard negatives to build the shortlist. Confidence temperatures are fit with a Brier objective on 602 of our own validation rows. The base model is frozen and referenced by its Hugging Face id; the adapter is 82 MB.

The support deployment fine-tunes the same recipe on retail decisions: 33,796 supervised rows (22,442 action and 11,354 argument picks, including 6,293 human-generated), a rank-128 adapter, and one reinforcement-learning round on 1,180 live-rollout decisions. On a held-out set of 916 action and 694 argument decisions it agrees with the Gemini Pro teacher on 65% of actions and 94% of arguments.

Fine-tuned to a domain: τ2-bench retail

Fine-tuned on retail-support decisions, TOD runs as the front end of a customer-support agent on τ2-bench retail: 114 held-out tasks, held out by customer. A solve requires both the correct database write and the correct natural-language read-back. When confidence on a pick is below the operating threshold, the turn goes to the parent model, Gemini Pro or Gemini Flash.

Two operating points: serve-less (a more conservative threshold) and serve-more. With either parent, TOD matches or nearly matches the parent’s solve rate at 23–34% lower cost per episode. Jev, which can only substitute replies and cannot make the tool calls, costs more than Pro alone at its best solve rate.

What’s next

A decision model does not have to stop at classifying a ticket. TOD picks the tool, picks every argument from what is actually there, reads the image when there is one, and tells you how sure it is, so your agent can act on the cheap decisions and escalate the rest. Read more at https://daseinai.ai/tod.