Stop Using Fixed Top-K

TLDR: Every RAG stack picks a top-K and pays for it on every query. Half the time the answer's at rank 1 and the rest is noise the LLM has to read past. Dasein predicts K per query, cuts 30–60% of context tokens, and keeps R@1 intact. The portable predictor is free and works in front of any stack. Visualization of a result stack shrinking from 10 records to a per-query K, with the rank-1 result staying on top. There's a math problem hiding in every agentic RAG product. Top-K is fixed at 10 because that's what BEIR defaults to and what every LangChain example ships with. Tokens per record × records per hop × hops per task × daily tasks = the line item your CFO started asking about this quarter. This is how that number got so big, why fixed K is the wrong default, and what changes when K is predicted per query. The arithmetic: A typical agentic research call pulls top 10 records per hop. Each record averages 250–300 tokens after chunking. Four hops for a real research agent. That's ~11,000 tokens of retrieved context per task, before the LLM's output, before tool calls, before re-reads. At GPT-class pricing ($2.50–$5.00 per 1M tokens) it's $0.027 to $0.055 per task. At 100,000 daily tasks that's six- to seven-figure annual spend on retrieved context alone. The CFO's right to ask why. Why fixed K is wrong: K = 10 was a compromise from when first-stage retrieval was unreliable — pre-BERT dense, pre-Cohere v3, pre-Voyage. You wanted enough candidates that the right answer was probably in the top-K, and you paid for the noise downstream because recall was the bottleneck. It's wrong in both directions now. Easy queries get over-fetched. On most queries the answer's at rank 1 and ranks 2–10 are filler. K = 10 over-fetches by nine records on the majority of queries. Hard queries don't all need K = 10. When the right answer's at rank 5 or rank 12, you need K = 5 or K = 12. A fixed cutoff is wrong on both ends. The fix is per-query K. The hard part is doing it without losing R@1. What Dasein does: Dynamic Top-K reads the query and the top candidates' scores and predicts the right K for that query. It always keeps the rank-1 record, so R@1 is preserved by construction. The cuts come from trimming records 2 through K on queries where the answer was already in the kept prefix. The SDK exposes it on hybrid indexes as a per-query toggle. Pair dynamic_top_k=True with dynamic_hybrid=True, which it requires because the cutoff was trained on the α-fused ranking, and the trim happens server-side. Your top_k is the hard ceiling; Dasein only clips down. Python example: from dasein import Client. client = Client(api_key="dsk_..."). index = client.create_index("my-docs", index_type="hybrid", model="bge-large-en-v1.5"). results = index.query("AAPL Q3 2025 earnings call", top_k=10, mode="hybrid", dynamic_hybrid=True, dynamic_top_k=True). Under agentic_search=True it applies to every hop — see the agentic search post for the compounding savings. For a bring-your-own retrieval stack, the same predictor returns both K cutoffs, dense and hybrid, alongside α in one call. Python example: qvec = my_encoder.encode("who founded apple?"). p = client.predict_dynamic("who founded apple?", query_vector=qvec). kept_hybrid = rrf_fuse(my_dense, my_bm25, alpha=p.alpha)[:p.top_k_hybrid]. kept_dense = my_dense_hits[:p.top_k_dense]. The receipts: From dynamic_topk_summary.md. Eval pool of 239,395 queries portable and 223,763 Dasein-native, pooled across FiQA, FEVER, SciFact, NQ, and ten encoders. Portable — what your stack would see this week: Dense top-10 — R@1 0.7109, R@10 0.8162, average records 10.00, average tokens 2,756. Dynamic Hybrid plus Dynamic Top-K — R@1 0.7107, R@10 0.8717, average records 6.92, average tokens 1,545. 30.8% fewer records. 43.9% fewer tokens. R@10 up 5.55 percentage points. R@1 down 0.0002 — noise. Dasein-native — compression plus per-query α plus per-query K composed end-to-end: Dense top-10 — R@1 0.7606, R@10 0.8771, average records 10.00, average tokens 2,859. Dynamic Hybrid plus Dynamic Top-K — R@1 0.8129, R@10 0.9494, average records 3.65, average tokens 905. 63.5% fewer records. 68.4% fewer tokens. R@10 up 7.23 percentage points. R@1 up 5.23 percentage points. Fewer than four records on average. Recall higher at every cutoff. Compression and per-query α cover the upstream stages. K distribution across queries — most need K ≤ 4. Token savings outpace record savings, −68.4% vs −63.5%, because lower-ranked confusers tend to be longer records — more boilerplate, more semantic smearing. The predictor cuts those first and the byte savings compound. Per-corpus — where this works hardest: FEVER — Dasein-native token reduction vs dense top-10 −70.1%, R@10 +9.06 percentage points. NQ — token reduction −39.9%, R@10 +14.26 percentage points. SciFact — token reduction −50.6%, R@10 −0.24 percentage points. FiQA — token reduction −27.5%, R@10 −2.82 percentage points. FEVER and NQ are lexically rich — per-query α lifts the answer to rank 1 or 2 reliably and Dynamic Top-K cuts everything below. SciFact and FiQA are dense-dominant; the FiQA R@10 dropped 2.82 percentage points, which is real and worth flagging. If your production corpus is dense-dominant, such as financial filings, clinical literature, or anything where every retrieved record is broadly plausible, measure on your data before turning the cutoff on across the board. Token reduction by corpus: FEVER, NQ, SciFact, FiQA on portable and Dasein-native. What this saves at your volume: At 10K daily tasks, the top-10 retrieval-context bill is approximately $120K, with Dasein-native approximately $38K, for approximately $82K annual savings. At 18K daily tasks, the top-10 bill is approximately $217K, with Dasein-native approximately $69K, for approximately $148K annual savings. At 100K daily tasks, the top-10 bill is approximately $1.2M, with Dasein-native approximately $380K, for approximately $820K annual savings. Assumptions: $3 per 1M input tokens, substitute your model, 4 hops per task, approximately 275 tokens per record, retrieval context only. Portable savings are roughly 60% of those: approximately $50K, $90K, and $500K. Bigger chunks mean bigger absolute savings. Annual savings at 10K, 18K, and 100K daily tasks, portable vs Dasein-native. Try it: Drop the portable SDK on your retrieval path, run last quarter's queries through it with logging, measure the token delta. The savings hold at the recall you ship today — no change to your encoder, your index, or your reranker. Six to seven figures of recovered margin from one config flip.