Cost-Efficient Agent Serving on τ-bench (Retail)

Summary

We evaluated a small, domain-fitted decision model placed in front of a frontier "parent" LLM in a customer-service agent. On each turn the small model either handles the turn itself or defers to the parent. On a held-out retail benchmark it delivered the same task-completion rate as the parent alone while cutting per-episode cost by about 28% and parent-model calls by about 41%.

Benchmark

Benchmark: τ-bench (Sierra), retail domain — https://taubench.com/. τ-bench runs an agent against a simulated customer over a tool-equipped retail environment (order lookup, returns, exchanges, address and payment changes, and so on).

What counts as a "solve": task completion is judged by the benchmark's own check on the final database state (plus the benchmark's response assertions where a task defines them) — not by any judgment of ours.

Evaluation set: 114 held-out retail tasks. These tasks were never seen during training of the decision model.

Training sources: the decision model was trained on airline and telecom agent episodes plus about 500 synthetic retail tasks. It did not train on any of the 114 evaluation tasks.

The comparison

Both arms were run on the identical environment and the identical simulated customer; the only difference is whether the small decision model sits in front of the parent.

What the numbers mean

The two arms complete tasks at the same rate within the benchmark’s run-to-run variation (a single trial on 114 tasks moves by roughly ±6 solves from sampling noise alone, so a 99-vs-92 gap is not a meaningful difference in capability). At that matched completion rate, our system makes about 41% fewer parent-model calls and costs about 28% less per episode. Because the savings come from turns the small model handles itself, they scale with the share of turns it takes on.

Methods

Scoring. Both arms run on the same environment, the same tool implementations and policy, and the same customer simulator. A task is scored solved by the benchmark’s own database-state check. Cost is the parent model’s token usage billed at pinned public per-token prices. Parent-call counts and the share of turns handled by our model are read from the serving ledger.

Held out. The 114 retail evaluation tasks were excluded from all training. The decision model was trained only on the airline, telecom, and synthetic-retail sources above, and never saw the evaluation tasks.

Also observed

Parent-agnostic. The same front-end approach works with different parent models; it is not tied to one vendor. The same setup runs unchanged on top of a faster, cheaper parent (which alone solves 100/114 at $0.057 per episode).

Better than fine-tuning a small model to do the whole task. We also tried supervised fine-tuning of small models to act as the agent directly on this task. That did worse: a fine-tuned small model handling the majority of turns solved only 50/114 and lost to the parent on cost per solve. Handing a share of turns to a small decision model while deferring the rest to the parent is the stronger approach.

What’s next

Applying the same method to your own agent workflows to measure the cost reduction available at your task-completion bar.