Context assembly and the lost-in-the-middle problem in RAG

Context assembly is the construction of the final prompt from retrieved material. Its inputs are the reranked candidate list, the user's question, any conversation history, and your instructions; its output is one string (or message array) sent to the model. The decisions it makes:

Lost-in-the-middle is a positional effect in how models use long contexts. The reported shape is a U-curve: accuracy at retrieving and using a fact is highest when the fact appears near the beginning of the context, nearly as high when it appears near the end, and lowest when it appears somewhere in the middle. The effect is well enough established to design around, and it gets more pronounced as the context gets longer.

Two clarifications that matter, because both are exam-relevant:

It is not a hard limit. The model does not fail to see the middle; it uses it less reliably. This is a probabilistic degradation, which makes it exactly the kind of bug that passes your manual testing and shows up as a mysterious accuracy dip in production.

It is not fixed by a bigger context window. A 200k-token window does not make position irrelevant — if anything, longer contexts make positional effects more consequential, because there is more middle. A model advertising a large context window is advertising capacity, not uniform attention across that capacity.

I am stating the U-curve as a reported, widely reproduced empirical finding about long-context behaviour, and I am deliberately not quoting specific accuracy percentages for it, because the numbers vary by model, task, and context length and I have no single sourced figure to give. The design implication — put your best material at the edges — does not depend on the exact magnitude.

L1 — The intuition you can carry into an exam

The model reads your context like a person skim-reading a long document: the opening lands, the ending lands, the middle blurs. So put the most important chunk first, the second most important last, and let the weaker material occupy the middle where its lower value matches its lower influence.

Three facts to carry:

• Lost-in-the-middle is a U-curve: beginning best, end nearly as good, middle worst.
• Best chunk first, second-best last is the standard mitigation.
• Fewer, better chunks beat more, weaker chunks. Context is a budget.

L2 — The mechanism: assembly order and the token budget

The standard ordering strategies, and what each is for:

Note the document-order row, because it is a real exception to the whole lesson. If your retrieved chunks are steps 2, 5, and 3 of an installation procedure, presenting them in rank order gives the model a scrambled procedure and it will produce a scrambled answer. Sequential material must be assembled in sequence. This interacts directly with the chunking decisions in 06-02 and the metadata decisions in 06-03: if you do not store each chunk's position within its parent document, you cannot restore the order.

The other half of assembly is the token budget, and it is arithmetic you should do explicitly (02-03 on counting tokens, 04-06 on the context window):

context_window  =  system_instructions
                +  conversation_history          (12-11)
                +  Σ (chunk_text + chunk_header) for k chunks
                +  the user's question
                +  ROOM FOR THE ANSWER            ← the one people forget

The last line is the most commonly omitted term. The context window is shared between input and output. If you fill 99% of it with retrieved chunks, the model has no room to answer and will be truncated mid-sentence. Reserve the output budget first, then spend what remains.

L3 — Why the effect exists, and what else assembly must handle

Why position matters at all. A careful, hedged account: attention over a long sequence is a finite resource distributed across many positions, and both the architecture and the training data shape how it is distributed. Positional encoding schemes (04-02) give the model its sense of where a token sits, and their behaviour at long distances is not uniform. Training data has structure — documents open with summaries and close with conclusions — so a model trained on next-token prediction (01-01) over such data learns that early and late positions carry disproportionate information. Attention is also normalised across positions, so more competing content means less attention per chunk.

I am presenting that as a plausible, partial explanation rather than a settled mechanism. The empirical U-curve is the robust part; the causal story is not. For the exam, know the effect and the mitigation; do not build an argument on the mechanism.

Three further things assembly must get right, each of which is a silent failure if missed:

Render the metadata the answer depends on. 07-03 established that dates, versions, and source authority are invisible to the retriever unless stored as metadata. They are equally invisible to the model unless rendered into the text it reads. A chunk header is the cheapest fix in the entire pipeline:

[SOURCE: Expense Policy v4 · effective 2026-01-15 · status: current · doc_id: p-1182#c3]
Expense claims must be submitted within 30 days of the transaction date.

That header does three jobs at once: it lets the model prefer recent sources when asked to, it gives it a citation handle (07-11), and it makes the assembled prompt debuggable by a human (07-10).

Deduplicate at assembly time, not just at ingestion. 06-04 removes near-duplicates from the corpus, but retrieval can still return three chunks that overlap heavily — especially with chunk overlap enabled, which deliberately duplicates text across adjacent chunks. Three near-identical chunks in a context waste two slots and, worse, create the impression of corroboration where there is only repetition. Assembly should collapse them.

Handle the empty and weak cases explicitly. If reranking returns nothing above your relevance floor (07-07), the correct assembly is not "the top 3 anyway". It is a context that says no sufficient source was found, plus an instruction to decline (07-11). Assembling weak context and hoping is how a retrieval failure becomes a hallucination.

And the confusable distinctions this lesson has to keep straight, because they appear as distractors:

That last row is worth dwelling on. The module has used three different k-like numbers: the retrieval pool per leg (07-06), the reranker's candidate pool N (07-07), and the number of chunks assembled into the context. The canonical shape is deep, then deeper, then narrow: retrieve 50 per leg, fuse to ~80 unique, rerank all of them, assemble 3–5. A pipeline that retrieves 3 and assembles 3 has no reranking headroom and no fusion depth.

This is a constructed illustrative example. The token counts are computed from stated assumptions; the described model behaviours are the expected consequences of the U-curve, not measured outcomes of a specific run. No accuracy percentages are quoted because I have no sourced figures for them.

A user asks: "how many days do I have to submit an expense claim, and does the deadline change for international travel?"

This is a two-part question, which makes it the right example — two facts must both survive assembly.

Reranking (07-07) returned five candidates:

Both facts the user needs are in e1 and e2 — ranks 1 and 2. Everything else is noise of decreasing usefulness.

Order A — naive rank order, all five chunks

[SYSTEM]  Answer using only the provided context.
[CONTEXT] e1 (30 days)          ← position 1, strong
          e2 (60 days, intl)    ← position 2
          e3 (review time)      ← position 3, middle
          e4 (portal location)  ← position 4, middle
          e5 (revision history) ← position 5, end position — occupied by the WEAKEST chunk
[QUESTION] how many days …

Token accounting: 40 + 45 + 40 + 38 + 42 = 205 chunk tokens, plus headers, instructions, and the question.

The structural defect is visible without measuring anything: the strongest end position is occupied by the least useful chunk. e5 — a revision-history line — sits in one of the two positions the model attends to most reliably, while e2, which carries half the answer, sits at position 2 and is pushed toward the middle by everything after it. Three of five slots carry no information the user asked for.

The likely failure mode: the model answers "30 days" correctly and either omits or hedges the international-travel exception, because e2's position is weaker than e1's and it is competing with three irrelevant chunks for attention.

Order B — edge-loaded, all five chunks

[CONTEXT] e1 (30 days)          ← position 1, strong
          e3 (review time)      ← middle, appropriately
          e4 (portal location)  ← middle, appropriately
          e5 (revision history) ← middle, appropriately
          e2 (60 days, intl)    ← final position, strong
[QUESTION] how many days …

Same 205 tokens. Same five chunks. Both answer-bearing chunks now sit in the two strong positions, and the three low-value chunks occupy the middle, where their lower influence matches their lower value. This is the "best first, second-best last" pattern, and it costs exactly nothing to implement — it is a list reorder.

Order C — relevance floor at 0.5, edge-loaded

Apply a reranker score threshold of 0.5. e4 (0.44) and e5 (0.19) are dropped:

[CONTEXT] e1 (30 days)          ← position 1
          e3 (review time)      ← middle
          e2 (60 days, intl)    ← final position
[QUESTION] how many days …

Token accounting: 40 + 40 + 45 = 125 chunk tokens. That is a 39% reduction from Order A's 205, with both answer-bearing chunks in strong positions and only one weak chunk remaining. Fewer tokens, lower latency, lower cost, less dilution, and a better-structured context.

This is the lesson's central arithmetic. Order A and Order C use the same retrieval and the same reranking. Order C sends 39% fewer chunk tokens and gives the model a strictly better-arranged context. The difference is two lines of assembly code: sort with edge-loading, filter by score.

Order D — the "more is better" failure

Suppose someone reasons that the model has a large context window, so why not include the top 20 candidates?

[CONTEXT] e1 … e20            ← ~800+ chunk tokens

What went wrong, in order of severity:

1. e2 is now somewhere in the middle of twenty chunks — the deepest part of the U-curve. The international-travel exception is at maximum risk of being under-used.
2. Seventeen chunks carry no relevant information, and each one competes for attention.
3. Contradiction risk rises. With twenty chunks, the odds of including a superseded policy version stating a different number go up sharply — exactly the 07-03 recency failure. The model now has "30 days", "60 days", and possibly "90 days" from a 2019 document, with no basis for choosing.
4. Cost and latency rise roughly linearly in tokens (12-09, 12-10), for negative quality return.
5. Debugging gets harder. A 20-chunk context is not something a human will read when investigating a wrong answer, so the failure becomes opaque (07-10).

Order D is the most common context-assembly mistake in production RAG, and it is usually the result of setting k once during prototyping and never revisiting it — or of reasoning that a bigger context window is an invitation to fill it.

Reading the four orders together

Order C wins on every column. It is also the only one that required thinking about assembly as a stage with its own decisions rather than as a string concatenation.

The one rule that subsumes most of the table: every chunk in the context must justify its slot. A chunk earns inclusion by clearing a relevance floor you set from measurement, not by being in the top-k of an arbitrary k.

This lesson sits on the boundary between the retrieval objectives and the prompt-engineering objectives, which is unusual and makes it high-yield:

• 1.9 — Use prompt engineering principles to create prompts to achieve desired results [OFFICIAL]. Context assembly is prompt construction. The retrieved chunks are the largest part of the prompt in most RAG systems.
• 1.3 / 4.2 — Build LLM use cases such as RAG, chatbots, and summarizers [OFFICIAL]. Assembly is a named stage in NVIDIA's pipeline: after retrieve and decode, the LLM synthesises from what it was given [NVIDIA-DOC].
• 1.4 — Curate and embed content datasets for RAGs. Chunk size and overlap decisions made at curation time surface here as assembly problems.
• 4.4 — Identify system data, hardware, or software components required to meet user needs, and 4.1 / 1.1 — assist in deployment and evaluation of scalability, performance, and reliability [OFFICIAL]. Token count is a latency and cost variable.

Prompt engineering is a Tier-1 reported topic [FIELD], and context-window budgeting sits directly inside it. The exam's general-level calibration means: know that position affects use, know the U-curve shape, know that more context is not better, and know that the fix is ordering plus a relevance floor. Do not expect to be tested on attention mechanics.

Question phrasings you should recognise:

Distractor families:

• "Increase k to improve answer quality." The single most attractive wrong answer in this lesson. More context frequently makes things worse.
• "Use a model with a larger context window." Offered as the fix for lost-in-the-middle. It is not; it enlarges the middle.
• Lost-in-the-middle conflated with context overflow. Overflow is a hard truncation; lost-in-the-middle happens inside a valid window.
• Chunk size conflated with context window. A retrieval granularity decision (06-02) versus a model ceiling (04-06).
• "Reranking makes ordering irrelevant." Reranking produces a ranking; assembly decides where in the prompt that ranking lands. A perfect ranking pasted in naive order still puts rank 2 in a weak position.
• "Fine-tune the model to attend uniformly." Not a realistic intervention, and it does not address what was assembled (11-02).
• "Put the question first, always." Commonly the question goes last, so it occupies the strong final position — but this is worth testing on your own eval set rather than asserting.

Mistake 12 is the one that makes all the others harder to fix. The assembled prompt is the single most useful artefact when debugging a RAG failure, because it is the exact input the model saw. Logging the ordered list of chunk ids and their scores costs almost nothing and turns "the answer was wrong" into "the answer was wrong and chunk e2 was at position 8 of 15". That is the difference between a hypothesis and a fix, and it is the method 07-10 builds on.

Honestly: the effect is much better established than its explanation. Here is what can be said with different levels of confidence.

Well established (empirical). Across models and tasks, accuracy at using a fact from a long context follows a U-shape by position: high at the start, high at the end, lowest in the middle. The effect grows as contexts lengthen. It has been reproduced widely enough to be a design constraint.

Plausible contributing factors (partial explanation, not settled).

Positional encoding behaviour at long distances. Transformers have no inherent notion of order; position is injected (04-02), and different schemes behave differently at long range. Some extrapolate poorly beyond the lengths they were trained on.

Training-data structure. Human documents front-load and back-load importance — abstracts, introductions, conclusions, summaries. A model trained on next-token prediction over such text (01-01) learns that early and late positions predict more, and that learned prior persists at inference.

Attention normalisation. Attention weights across positions sum to a fixed total, so more competing content means less weight available per item. That explains dilution — why adding chunks hurts — more directly than it explains the U-shape.

Instruction-tuning position bias. Instructions typically appear at the very start or very end of training examples, so those positions get extra weight during instruction tuning.

What follows for you. The mitigation does not depend on which explanation is right:

1. Put the best chunk first and the second-best last.
2. Reduce k so there is less middle.
3. Repeat critical constraints in the instruction block at the start or end, rather than relying on a mid-context chunk to carry them.
4. Measure it on your own stack. Take a question your system answers correctly, move the answer-bearing chunk from position 1 to the middle of a longer context, and see whether the answer degrades. That is a ten-minute experiment and it tells you how exposed your particular model and context length are.

Two things not to do: do not quote a specific accuracy figure for the effect (it varies by model, task, and length), and do not assume a newer or larger-context model is immune. Test.

The fewest that clear a relevance floor — commonly 3 to 5, chosen by measurement rather than convention.

Set k from a curve, not a default. Run your eval set at several values and read the shape:

The characteristic shape is a rise then a plateau then a decline. Find your plateau and sit at its left edge, where you get the quality at the lowest token cost.

Prefer a score floor to a fixed k. A fixed k sends five chunks whether five are good or one is. A floor sends however many clear the bar — one for a narrow factual question, five for a broad comparative one. This is where reranker scores earn their keep, because they are far more thresholdable than cosine similarities (07-07, 07-02).

Let the question shape it. Single-fact lookups need one or two chunks. Comparative questions ("what changed between v3 and v4?") structurally need at least two. Multi-part questions like the worked example's need one per part. If your system serves both shapes, a floor handles the variation automatically and a fixed k cannot.

Count the tokens. Not "about five chunks" — the actual number, including headers and instructions (02-03). That number sets your prefill latency (12-10) and your per-query cost (12-09), and both are things someone will eventually ask you to reduce.

And do not fill a large window because it is there. A 128k-token window means you can send 300 chunks. Sending 300 chunks means the answer-bearing one is buried at the bottom of the U-curve, surrounded by 299 competitors, at maximum cost and latency. Large context windows are most valuable for genuinely long single documents, not for retrieval slop.

No, and the belief that it does is one of the more expensive misconceptions in applied RAG. Work through what a larger window does and does not change.

What it genuinely helps with. Long single documents that must be read whole — a contract, a full specification, a long transcript. Longer conversation histories without aggressive summarisation (12-11). More headroom before hard truncation. Occasionally, skipping retrieval entirely for a corpus small enough to fit in the window, which is a legitimate architecture and part of the 07-12 discussion.

What it does not help with:

The framing that holds up: a context window is a constraint you must respect, not a resource you must exhaust. The engineering goal is the smallest context that reliably contains the answer, because small contexts are cheaper, faster, better-attended, easier to debug, and less likely to contain contradictions.

There is a real architectural question hiding here, and it is worth naming rather than dodging: if a corpus fits in a modern context window, is retrieval even necessary? Sometimes not. Sending a whole 50-page manual and asking a question is a valid design for a small corpus, and it eliminates every retrieval failure mode in this module at the cost of tokens, latency, and lost-in-the-middle exposure. It does not scale, it gets expensive fast, and it degrades as the document grows — but it is a real option and pretending otherwise is the kind of RAG over-application 07-12 exists to correct.

• Context assembly is a stage with its own decisions, not a string concatenation. Count, order, metadata, instructions, and the empty case are all choices that change the answer.
• Lost-in-the-middle is a U-curve: information at the start and end of a long context is used more reliably than information in the middle. The effect is well reproduced; its causal explanation is not settled.
• Edge-load: best chunk first, second-best last. It costs one list reorder and it puts both answer-bearing chunks in strong positions.
• The worked example's headline result: applying a 0.5 relevance floor and edge-loading cut chunk tokens from 205 to 125 — a 39% reduction — while moving both answer-bearing chunks into strong positions. Same retrieval, same reranking, two lines of assembly code.
• More context is not better context. Additional marginal chunks dilute attention, raise contradiction risk, cost tokens, and slow prefill.
• A bigger context window does not fix lost-in-the-middle — it enlarges the middle. Capacity is a ceiling, not a target.
• Sequential material must be assembled in source order, not relevance order, or the model produces a scrambled procedure.
• Render metadata into chunk headers. A date stored in a field the model never sees cannot influence the answer.
• Prefer a relevance floor to a fixed k, so a narrow question gets a narrow context and a broad one gets more.
• Log the assembled context — ids, scores, and order. It is the single most useful artefact when an answer is wrong.

Every component now exists in isolation: parsing, chunking, deduplication, two retrievers, filtering, fusion, reranking, and assembly. What you do not yet have is the object they add up to — and that object has a property none of its parts have, which is that its end-to-end quality equals its single worst stage. A flawless reranker cannot repair a parser that dropped every table. Next: 07-09 assembles the complete pipeline stage by stage, names all nine stages as one thing you can reason about, places NVIDIA's own pipeline description and the NeMo Retriever and AI Blueprint for RAG framings against it, and makes the case that a RAG system is best understood as a chain whose weakest link you have to be able to find on demand.