Multi-Turn Chat History and Query Rewriting

A single-turn RAG system has a simple contract: the user asks a complete question, that question gets embedded, and the embedding gets matched against an index. Multi-turn chat breaks that contract, because natural conversation is full of references that only make sense next to what was already said. A user asks "What's the refund policy for annual plans?", gets an answer, and then asks "What about monthly ones?" — a fragment that, read on its own, contains no verb and no clear subject. A retriever handed that fragment literally will search for documents about "monthly ones" in the abstract, which is not the same search as "what is the refund policy for monthly plans," and the retrieved chunks will reflect that mismatch.

Query rewriting is the step that sits between the user's raw turn and the retrieval call, and its job is narrow: take the current turn plus enough of the preceding conversation, and produce one self-contained question that carries the same meaning the user intended but does not depend on anything outside itself. "What about monthly ones?" becomes "What is the refund policy for monthly subscription plans?" — a question any retriever, run with zero conversation context, would search correctly. The rewrite does not answer the question; it only repairs the query so that the downstream retrieval and generation steps can do their jobs on inputs that actually mean what they appear to mean.

This matters specifically for RAG-backed chat, because RAG's entire value proposition (covered in full in 07-09, the complete RAG pipeline stage by stage) depends on the retrieval step finding the right documents. A generation model without retrieval can sometimes limp through an ambiguous follow-up using its own attention over the chat history — it has the whole conversation in its context window and can often infer the referent. A retrieval step run in isolation cannot: unless the system deliberately reconstructs the reference, the vector search has no conversational memory at all, and the failure is silent — it doesn't error, it just returns confidently wrong context.

L1 — The intuition: retrieval has no memory of its own

Picture calling a librarian on a phone that resets after every question — each call is a brand-new librarian who has never spoken to you before and can only hear the words in this one call. If your second call is "what about the paperback?", that librarian has no idea what book you mean; they will confidently go fetch information about paperbacks in general. A human assistant sitting next to you the whole time would understand instantly, because they remember the first question. A vector index is the phone-reset librarian: every query it receives is evaluated purely on the text handed to it, with zero awareness that a conversation came before. Query rewriting is the step that, before dialing, restates the question in full — "what is the paperback price for the book we discussed a moment ago" — so the librarian on the other end of the reset phone can actually help.

L2 — The mechanism: history-conditioned rewriting

A working rewriting step has three moving parts:

1. A window of prior turns. Not the entire conversation history necessarily — enough of the most recent turns (and often the most recent retrieved-and-answered exchange specifically) to resolve the current turn's references. Section 3 below covers the budgeting decision for how much history to include.
2. A rewriting call. Typically a small, fast LLM call (sometimes the same model that will later generate the answer, sometimes a smaller and cheaper one dedicated to this one job) is prompted with the recent turns and the new fragment, and asked to produce a single self-contained question. The prompt is deliberately narrow — "rewrite the final user message as a standalone question, resolving any pronouns or references using the conversation above; do not answer it" — because a rewriting call that drifts into answering, elaborating, or adding assumptions not present in the conversation introduces its own failure mode.
3. The rewritten query replaces the raw turn for retrieval only. The original raw turn is still what's shown in the chat UI and still what the final generation step sees as "what the user actually typed" — the rewrite is an internal artifact used exclusively to drive the retrieval call. This distinction matters: rewriting corrupts nothing the user sees, it only repairs what the retriever is asked to search for.

A concrete trace: turn 1, user asks "What's the refund policy for annual plans?" — the system retrieves the annual-plan refund section and answers correctly. Turn 2, user asks "What about monthly ones?" The rewriting step receives turn 1's question, turn 1's answer (or at least its topic), and turn 2's fragment, and produces: "What is the refund policy for monthly subscription plans?" That rewritten question — not the two-word fragment — is what gets embedded and searched. The retriever now finds the monthly-plan refund section, because it was given a query that actually names the thing it needs to find.

L3 — Failure modes when rewriting is skipped or done badly

Skipping rewriting entirely produces the failure described above: literal fragments retrieve nonsense, and because the retrieval step never errors — it always returns something, just the wrong something — the system appears to work while quietly degrading every follow-up question's answer quality. This is a particularly dangerous failure mode precisely because it's silent: a demo built entirely on first-turn questions looks perfect, and the defect only appears once real users start asking natural follow-ups, which is nearly always.

Rewriting done badly has its own failure modes. A rewrite that hallucinates detail not present in the conversation — inventing a specific plan name or number that was never mentioned — poisons retrieval with a confident but wrong query, which can be worse than the honest ambiguity of the unrewritten fragment. A rewrite that includes too much accumulated context from turns far earlier in a long conversation can conflate two different topics the user has since moved past, producing a query that resolves the wrong reference. And a rewrite that is too aggressive — rewriting a question that was already self-contained — can needlessly alter its meaning and retrieve for a subtly different question than the one the user actually asked. The rewriting prompt has to be scoped tightly enough to resolve genuine ambiguity without over-editing turns that didn't need it.

Three different strategies for handling multi-turn context are easy to confuse because all three "use the conversation" in some sense, but they solve different problems and fail in different ways.

The distinguishing insight: raw history passing fixes the generation model's ability to understand context (because it has the whole conversation in its attention window) but does nothing for the retrieval step, which only ever sees whatever string it's handed to embed. Query rewriting is the only approach in this table that specifically repairs the retrieval query itself, which is why it is the standard fix for RAG-backed multi-turn chat rather than a bespoke workaround — it addresses the actual mechanism of the failure (retrieval's statelessness) instead of a symptom.

Consider a support chatbot for a software product, backed by RAG over a documentation corpus. This is a constructed illustrative trace, not a captured real transcript, built to show the mechanics turn by turn.

Turn 1 — user: "How do I export my project as a PDF?" No rewriting is needed; the question is already self-contained. Retrieval embeds it as-is and finds the PDF-export documentation. The system answers correctly.

Turn 2 — user: "Can I do that from the mobile app too?" Read alone, "that" and "too" are unresolved — the retriever has no idea what action is being asked about. The rewriting step is given turn 1's question, the topic of turn 1's answer (PDF export), and turn 2's fragment, and produces: "Can I export my project as a PDF from the mobile app?" That rewritten question is embedded and retrieves the mobile-app export documentation specifically — a different page from turn 1's desktop-focused article. Without the rewrite, the raw fragment "Can I do that from the mobile app too?" would likely retrieve generic mobile-app documentation with no connection to PDF export at all.

Turn 3 — user: "What about as a Word doc instead?" This fragment references two things from history: the export action (from turns 1–2) and a shift in target platform is ambiguous — does "instead" mean instead of PDF (same platform, different format) or instead of mobile (back to desktop, PDF)? A good rewriting prompt handles this by keeping the most recent unresolved dimension — format — as the one being changed, since "instead" most naturally attaches to the immediately preceding noun ("as a PDF from the mobile app"). A careful rewrite: "Can I export my project as a Word document from the mobile app?" This is exactly the kind of ambiguity where rewriting quality varies — a weaker rewriting prompt might drop the mobile-app qualifier and produce "How do I export my project as a Word document?", silently reverting to the desktop context from turn 1. The fix, covered in the next section's mistakes table, is to make the rewriting prompt explicitly carry forward all unresolved dimensions from recent turns, not just the most syntactically local one.

The retrieval-quality delta. In this constructed example, turn 1 needs no rewrite (retrieval already correct), turn 2's rewrite corrects what would otherwise be a wrong retrieval (mobile docs unrelated to export), and turn 3's rewrite is where rewriting quality genuinely matters and can go either way depending on how carefully the prompt tracks multiple unresolved references at once. The general pattern holds across real deployments: the first turn of a session rarely needs rewriting, and the value of rewriting compounds as a conversation gets deeper into follow-ups.

The decision rule: rewrite whenever a retrieval step sits between the user's message and the answer, because that retrieval step is the part of the pipeline with zero built-in memory. If there is no retrieval — a pure chat model answering from its own context window — rewriting is usually unnecessary work, since the generation model can already resolve references from the history in its prompt. The one exception is very long conversations, where even the generation model benefits from an explicit, bounded restatement rather than trusting attention to find the right thread across dozens of turns.

Every turn of history added to a prompt costs tokens, and tokens cost both money (per 12-09) and latency (per 12-10, since a longer prompt means a longer prefill and a larger KV cache to maintain through decode). A multi-turn system therefore has to make an explicit decision about how much history it carries forward, and there are two standard strategies:

• Truncation. Keep only the most recent N turns (or the most recent K tokens of history) and drop everything older. This is simple and cheap, and works well when conversations are short or when older turns genuinely stop mattering — a support chat that's moved from "how do I export a PDF" to "how do I change my billing address" doesn't need the export discussion carried forward. Its failure mode is dropping a reference the user still expects to be remembered: "go back to what you said about the mobile app" fails if that turn has already scrolled out of the truncation window.
• Summarization. Periodically compress older turns into a running summary — a short paragraph capturing what's been established — and keep that summary plus the most recent few turns verbatim. This preserves long-range continuity at a fraction of the token cost of keeping full history, at the price of an extra summarization call (itself adding latency and cost) and the risk that the summary drops or distorts a detail that later turns depend on.

Most production multi-turn systems combine both: a fixed window of recent turns kept verbatim for fine-grained reference resolution, backed by a periodically-refreshed summary of everything older for long-range continuity. The query-rewriting step then draws on whichever of these — recent verbatim turns, the running summary, or both — it needs to resolve the current fragment. This is the same budgeting discipline 2.02-style chatbot-building material addresses at the application layer; this lesson is specifically about the retrieval-facing consequence of that budgeting decision, which is that whatever history survives the truncation or summarization policy is the only material the rewriting step has available to resolve a reference against.

Because the retrieval step embeds and searches exactly the text it is handed, and a follow-up like "what about monthly ones?" contains no explicit subject on its own — the retriever has no access to the earlier turns that would tell it "monthly ones" refers to subscription plans discussed a moment ago. It searches for whatever the literal fragment superficially resembles, which is rarely the document the user actually needs, and it does so with full confidence because retrieval never signals "I don't understand this query" — it always returns its best match for whatever string it was given.

Passing chat history to the generation model lets that model use its own attention mechanism to understand references across turns when it produces the final answer — but it does nothing for the retrieval step, which is a separate component that only ever sees the string it's asked to embed and search. Query rewriting specifically repairs that string before it reaches the retriever, producing a standalone question that means the same thing the ambiguous fragment meant in context. The two are complementary: history in the generation prompt helps the model write a coherent answer, and a rewritten query helps the retriever find the right source material to answer from.

Enough recent turns to resolve the references actually present in the current message, which in practice is usually the last two to five turns for typical follow-up patterns, plus a running summary of anything older if the conversation runs long. Carrying the entire conversation into every rewrite call is rarely necessary and adds cost and latency (Section 10 in 12-09 and 12-10 cover why token count matters here) without improving resolution quality past what recent turns already provide — the failure mode of too little history (dropped references) is more common in practice than the failure mode of too much.

• Query rewriting: reconstructing an ambiguous conversational turn into a self-contained question before it is sent to a retriever.
• Referent resolution: determining what a pronoun or implicit reference ("that," "the second one," "instead") points to using prior conversation.
• Truncation (memory): keeping only the most recent N turns of history and discarding the rest.
• Summarization (memory): compressing older turns into a running summary rather than discarding or keeping them verbatim.
• Stateless retrieval: the property that a vector index or search engine evaluates each query independently, with no memory of prior turns unless the query itself carries that context.

Retrieval has no memory of its own — it evaluates exactly the text it's handed, turn by turn — so a multi-turn chat system built on RAG needs an explicit step that reconstructs each ambiguous follow-up into a self-contained question before it reaches the retriever. Passing raw chat history into the generation prompt helps the model write a coherent answer but does not fix retrieval, because those are two different components with two different inputs. The rewriting step itself needs careful scoping — resolve references without inventing detail, track all unresolved dimensions from recent turns rather than just the most recent one, and keep it as an internal artifact rather than something the user sees — and it should draw on a deliberately budgeted window of history, whether that's a fixed number of recent turns, a running summary, or both.

Next: 12-12 picks up the other half of this paired session: once the retriever is getting a well-formed query, the next question is whether the index it's searching is actually current — how a vector index stays fresh as the underlying corpus changes, and why switching to a new embedding model is a full-corpus migration rather than a config change.