Stream RAG — Searching While You're Still Speaking.

Arnab Maiti's pick is the most product-shaped paper in the batch. Stream RAG (Siddhant Arora and 16 co-authors, with Cornell and Meta AI among the institutions) tackles a problem anyone who has used a voice assistant has felt: it either answers instantly and sometimes makes things up, or it pauses awkwardly to look something up and kills the conversation. Stream RAG's move is to do the lookup while you're still talking.

Don't wait for the user to finish, then search, then answer. Predict what to search for in parallel with their speech — so by the time they stop, the facts are already on the way back.

The tension: natural vs factual

End-to-end speech dialogue systems — speech in, speech out, no intermediate transcription pipeline — are wonderfully natural and fast. But they share the LLM curse: without grounding, they hallucinate. Ask for a fact they don't reliably know and they'll confidently invent one.

In text systems, the fix is well established: RAG. Let the model call a tool — web search, a knowledge base — and ground its answer in what comes back. But naively bolting tool use onto a speech system breaks the thing that made it good: tool calls add latency, and in a spoken conversation a multi-second pause to go search is jarring. You've traded hallucination for awkward silence.

The idea: overlap the search with the speech

Stream RAG's insight is that a conversation gives you free time you're not using. While the user is still speaking, the system already has a growing partial transcript — often enough to guess what they'll need. So it predicts and issues the tool query in parallel with the incoming speech, before the user even finishes. The retrieval happens during the part of the turn that was dead time anyway.

By the moment the user stops talking, the retrieved results are already arriving. The grounding is ready, and the reply can be both factual and prompt. The latency didn't disappear — it got hidden behind the user's own talking.

What it took to build

Making that work required more than a clever trigger. The paper contributes a few pieces:

• A post-training pipeline that teaches when to call a tool mid-speech — the model has to learn to predict, from a partial utterance, both that a lookup is needed and what to search for, without waiting for the full question.
• Spoken-summary generation by audio–text fusion — the reply is produced by fusing the original spoken query with the retrieved text, so the assistant speaks a grounded summary rather than reading documents aloud.
• A modality-agnostic design — the same approach works for typed and spoken inputs, so it isn't a speech-only hack.
• AudioCRAG — a speech version of the CRAG QA benchmark, created to measure spoken tool-using assistants where none existed.

The results

The payoff lands on both axes that were in tension:

The accuracy jump is the headline — tripling factual QA accuracy is the difference between a toy and something you'd ship. But the latency reduction is what makes it usable: it shows the streaming approach doesn't just add grounding, it does so while cutting the time cost that made naive tool use unacceptable in voice.

Why it matters for builders

Voice agents are having a moment, and this is a missing primitive. Everyone wants assistants that talk naturally and get facts right; the latency of grounding has been the blocker. Stream RAG is a concrete recipe for hiding that latency in plain sight — and the modality-agnostic framing means the same trick (predict the tool call before the input is complete) applies to typed chat, agent loops, and any setting where you can act on a partial request.

• The general principle: start the slow work as early as you have enough signal to guess at it, in parallel with the rest of the input, rather than strictly after it.
• For voice products specifically, this reframes tool use from "a pause the user notices" to "work that finishes by the time they stop talking."
• AudioCRAG also matters quietly — you can't optimize spoken tool use without a benchmark for it, and now there is one.

The honest caveats

Predicting the query before the user finishes means sometimes predicting wrong — the user's sentence can take a turn the partial transcript didn't imply, wasting a search or grounding on the wrong thing. The system has to handle mispredictions gracefully (re-issue, or fall back). And speculative early tool calls cost real API/compute even when discarded. As with all speculative execution, the win is statistical: most early guesses are right enough to pay for the few that aren't.

The hard part: learning when to fire

The deceptively difficult contribution is the when. A system that searches too eagerly — firing a query off every few words — wastes compute and grounds on half-formed questions. One that waits too long gives back the latency it was trying to hide. So the post-training pipeline has to teach the model a genuinely new skill: from a partial utterance, judge both whether external facts will be needed and what to look up, and time the call so the result lands as the user finishes.

This is a different objective from anything a text RAG system learns. Text RAG sees the whole question before deciding to retrieve. Stream RAG has to decide under uncertainty, mid-sentence, and be right often enough that the speculative calls pay for themselves. That's why it needs a dedicated training stage rather than a prompt — the model is learning a policy over incomplete input.

The same idea as speculative decoding

It's worth naming the through-line to Part 1. Speculative decoding speeds up generation by guessing future tokens with a cheap model and verifying them in parallel — doing speculative work that's usually right and cheap to check. Stream RAG is the same instinct applied to tool use: speculatively start the retrieval before you're certain what's being asked, in parallel with the input, because most guesses are right and a wrong one is cheap to discard. Across the two batches, a pattern emerges — much of recent systems work is about finding dead time and filling it with speculative work.

The takeaway

Stream RAG turns the dead time while a user speaks into retrieval time, so a voice assistant can be grounded and fast at once — tripling QA accuracy while cutting tool latency. The transferable idea is older than speech: speculatively start the expensive step as soon as you can guess it, in parallel with the input. It's the same instinct as speculative decoding from Part 1, applied to tool use in a conversation.