NVIDIA says adding a reranker cut agent token cost 13-14%

NVIDIA says that putting a reranker between an agent's search tool and its context window reduced the tokens the agent spent per query by roughly 14%. The claim is in a post on the Hugging Face blog dated 21 October 2026 (updated 29 October) that launches the Nemotron 3.5 retrieval models: Embed in 8B and 1B sizes, Rerank 8B, and a 631M-parameter document parser called Parse. All figures below are NVIDIA's, from its own evaluation.

What was tested

The authors built an agent that queries a document set through a search tool, inspects what comes back and refines its next search. They ran it on three retrieval benchmarks: ViDoRe v3 (text plus images), BRIGHT and BrowseComp-Plus (text only). The LLM, tools, tasks, corpus and execution policy stayed fixed. Only the search tool changed: embedding alone, embedding plus reranking, or different model pairings.

What NVIDIA reports

• Adding Nemotron 3.5 Rerank cut agentic token cost per query by about 14% with the 8B embedder and 13% with the 1B embedder, while improving retrieval accuracy in both.
• With the 8B embedder, its reranker scored about 0.9 nDCG@10 points higher than Qwen3-VL-Reranker-8B at a similar token cost. nDCG@10 is a standard ranking score for how well the top ten results match the relevant documents.
• On the ViDoRe v3.1 leaderboard, as of 20 October 2026, Embed 8B scored 61.17 nDCG@10, first among dense retrievers and 8.94 points above NVIDIA's previous 1B model. The 1B model scored 58.38, 95.4% of the 8B.
• The reranker scored 60.49 on combined image-and-text inputs against 59.17 for Qwen3-VL-Reranker-8B. NVIDIA launched the reranking leaderboard with the MTEB team, so it is both a participant and a co-organiser.

Serving and fine-tuning

For throughput, NVIDIA reports the 1B NVFP4 (4-bit) embedder reaches 128.8 documents per second on a GB200 GPU against 101.3 for the BF16 version, a 27.1% gain, with 99.85% of the accuracy retained. On an RTX PRO 6000 Blackwell server GPU it says its NIM container handled 6,031 query requests per second against 2,099 for a comparable vLLM setup, about 2.9 times.

The authors also describe a fine-tuning recipe: generate synthetic queries from your own documents, mine hard negatives, then train with NeMo AutoModel. On 8,202 chart and infographic images, one epoch lifted the 1B model's nDCG@10 from 37.34 to 44.12 and the 8B's from 41.44 to 48.98, both about 18% relative, measured on 1,719 held-out synthetic queries. Synthetic queries are written by a model, so real users' questions may behave differently.

One detail matters for anyone with an existing index. Because the language component was frozen in training, the post says text-only embeddings stay identical to Nemotron 3 Embed, so current text vectors can be reused.

What the post leaves out

• The agent LLM is not named in the text I read, and there are no absolute token or dollar figures, only relative savings.
• The reranker adds its own compute per query; the post reports agent tokens saved but not the cost of running the reranker.
• No customer production result is reported for the token saving; the headline number comes from three public benchmarks.
• Comparisons are against selected open baselines, chosen by NVIDIA.

The useful idea is cheap to test. Agents that retrieve poorly search again, which costs tokens. If your support or knowledge agent shows long search loops, log tokens per resolved task, add a reranker over the top 50 candidates on a sample of real queries, and compare.