Reducing Agent Costs: Where the Tokens Actually Go

Why an agent costs 5 to 30x what a chat does

A chat sends 1 prompt and reads 1 reply. An agent runs a loop: plan, call a tool, read the result, decide, repeat. Gartner’s estimate puts agentic workloads at 5x to 30x the tokens of a chatbot interaction for the same task, and a typical agent fires 10 to 20 model calls per user action. Three things drive the multiple, and all three land on the input side of the bill.

History is re-sent every turn. Turn 20 carries turns 1 through 19 as input. Input per turn grows linearly, so input over the whole run grows with the square of the turn count.

Tool output lands in context raw. A file read, a search result, or a query result set can run 5,000 to 20,000 tokens, and it stays in the window for every later turn whether or not the agent needs it again.

Retries multiply both. A failed tool call does not cost one more call. It costs the full accumulated context one more time.

Output tokens are the smaller line in nearly every agent trace. Input is where the money goes, and input is what the 4 levers below attack.

Lever 1: Carry less context per turn

Most of what an agent carries forward is dead weight by the next turn. A tool result that has already been acted on, a planning step that resolved, a file the agent will not touch again. Pruning drops it before the next call. Compression shrinks what stays. Two ways to do it:

1. Rule-based pruning. Drop tool outputs older than N turns, truncate results above a size cap, summarize resolved subtasks. Cheap to build, blunt in effect, and prone to dropping the one thing the agent needed 6 turns later.

2. Learned pruning. A model scores each span of context on its likelihood of being needed and drops the low scorers. This is what Parsec runs on every turn. On typical agent traces it carries about 40% less context per turn than the raw history.

The quality check is the same either way. Run your eval suite with pruning on and off. If task completion holds, the savings are real. If it drops, you pruned something that mattered, and the fix is in the scorer, not in the idea.

This lever has the highest return because it compounds. Every token removed at turn 5 is a token not paid for at turns 6 through 30.

Lever 2: Stop paying to recompute the prefix

Every turn, the model re-reads the system prompt, the tool definitions, and the full history before it generates a single new token. The attention state it builds over that prefix, the K/V cache, is identical to what it built last turn. Recomputing it is pure waste, and it is billed as full-price input.

Provider-side prompt caching recovers some of this. It works when the prefix is byte-identical to the last call and the whole run stays on 1 model. It breaks in 2 common cases: the context was pruned or compressed between turns, so the prefix changed and the cache missed; or the run switched models mid-task, and the new model has no cache for this conversation.

That means Lever 1 and provider caching fight each other. Prune the context and you lose the cache. Keep the cache and you carry the dead weight.

Parsec resolves this with a universal K/V cache that persists across turns and transfers between models. The cache is pruned along with the context, so the 2 levers compound instead of cancelling. When the router (Lever 3) hands a turn to a different model, the cache goes with it. 5-bit compression keeps the transfer small enough to move between clusters without eating the savings.

Lever 3: Route per turn, not per project

Most teams pick 1 model for the whole agent. The frontier model that handles the hard planning step also handles the step where the agent formats a date. The cheapest step in the loop gets billed at the most expensive rate.

Gartner calls the fix inference tiering: route each part of a workflow to the model its difficulty requires, and hold the expensive model for the work that needs it. In an agent, the unit of routing is the turn. The pattern:

A small decision model looks at the current turn (task state, the tool just called, what the next step has to produce) and picks the model for that turn.

Routine turns go to a small language model. Tool-call formatting, extraction, classification, summarizing a result, deciding whether a result is good enough to move on.

Hard turns go to a frontier model. Planning, ambiguous reasoning, recovery from a failed step.

Parsec’s router is an RL-trained decision model that makes this call on every turn, with the K/V cache from Lever 2 following the turn to whatever model receives it. The share of turns an SLM can take depends on the workload, and the only way to know yours is to measure it, which is Lever 4.

Lever 4: Measure cost per task, not per token

Per-token price is the wrong unit. A cheaper model that needs 3 retries costs more per completed task than the expensive one that got it right the first time. Gartner’s own read is that the cost of completing a workflow is the measure that matters, not the price of the tokens inside it. Four numbers to instrument before you touch Levers 1 through 3, so you can tell which one paid:

1. Cost per completed task, by task type. The only number finance cares about, and the one that catches a cheap model that fails often.

2. Input tokens per turn, plotted over the run. A straight rising line is history bloat. A line that bends flat is pruning working.

3. Loop depth and retry count per task, with a hard cap on both. Gartner recommends token thresholds and monitoring for exactly this; a runaway loop is the single most expensive thing an agent can do.

4. Cache hit rate per turn. If it drops when you turn pruning on, your cache and your pruner are fighting (see Lever 2).

Set a per-task token budget. When an agent blows through it, escalate to a human or fall back to a cheaper path. That is standard in cloud cost control and should be standard here.

A worked example

Take a 30-turn coding agent run. The system prompt plus tool definitions come to 8,000 tokens and never change. Each turn adds an average of 3,000 tokens of history: the model’s step plus the tool result it read.

With full history, input at turn t is 8,000 + 3,000 × t. Summed over 30 turns that is 1,635,000 input tokens. With carried history pruned by 40%, input at turn t is 8,000 + 1,800 × t, and the sum is 1,077,000 tokens. Same task, 34% fewer input tokens, before any other lever is touched.

Now stack the others. Cache reuse (Lever 2) bills the surviving prefix at the cached rate instead of full price. Routing (Lever 3) bills a share of those turns at SLM prices instead of frontier prices. Each lever multiplies against the token count the one before it left behind. Your trace shape sets the real numbers; this model exists so you can plug in your own.

Where Parsec fits

Parsec is an inference runtime. It is not a cloud and not an agent framework. Your agent stays as it is, Parsec sits between it and the models, and it runs all 4 levers on every turn.

Automatic cache pruning, about 40% less carried context per turn.

A universal K/V cache that persists across turns and transfers between models, with 5-bit compression for cross-cluster moves.

An RL-trained decision model that routes each turn to the model it needs, with SLMs taking the turns that do not need a frontier model.

Per-turn visibility into which model took each turn and how much context it carried, so the savings show up in your own numbers.

It is agent-agnostic. A LangGraph graph, a hand-rolled loop, or a vendor framework all look the same from the runtime’s side.

To see what it does on your stack: pull a representative set of agent traces, run them through Parsec with pruning and routing on, and compare cost per completed task against your eval suite. If the number does not hold for your workload, you will see that too. Details at https://parseclab.ai

Sources

Agentic Inference Costs to Rise Fivefold Even as Tokens Get Cheaper, Gartner Says, Techstrong.ai, August 17, 2026. Gartner’s 5x forecast through 2028, the at-least-5x cost of an agentic reasoning model versus a basic chatbot, inference tiering, token thresholds, and cost per completed workflow as the right measure.

Token Prices Fell 67% This Year. Your AI Bill Is Going Up Anyway, Rework, June 3, 2026. The roughly 67% blended price drop from Q1 2025 to Q1 2026, 10 to 20 model calls per agentic user action, and per-task token budgets.

LLM Token Cost Explained, Prem AI, August 18, 2026. Gartner’s 5x to 30x token estimate for agentic workloads versus a standard chatbot interaction.

The 40% figure for carried context and the K/V cache, compression, and routing details are Parsec’s own. The 30-turn example is a model built from stated assumptions, not a measured benchmark.