Reranking with a cross-encoder in RAG: how and when to use it

Reranking is a two-stage retrieval architecture. Stage one is cheap and wide; stage two is expensive and narrow.

query
  │
  ├─► stage 1: RETRIEVE  (bi-encoder / BM25 / hybrid fusion — 07-06)
  │            cheap per corpus item, precomputed index
  │            returns N candidates, N ≈ 25–200
  │
  └─► stage 2: RERANK    (cross-encoder)
               one model forward pass PER CANDIDATE
               returns the same N candidates, reordered
                          │
                          ▼
               truncate to the top k (k ≈ 3–8) → context assembly (07-08)

The pattern has a name in information retrieval: retrieve-and-rerank, or a cascade. The general principle behind cascades is that you spend compute in proportion to how much a decision matters. Deciding that 999,950 of a million documents are irrelevant deserves almost no compute each; deciding which of the surviving 50 goes into a context window with room for five deserves a lot.

What the cross-encoder actually outputs. Not an embedding. A single scalar — typically a logit that a sigmoid turns into a 0–1 relevance probability, or an unbounded score depending on the model. There is no vector to store, no index to build, and no re-embedding project when you change models. That last point is a genuine and underrated advantage: swapping rerankers is free, while swapping embedding models means re-encoding the corpus (12-12).

What reranking is not. It is not fusion (07-06) — fusion merges two lists using ranks; reranking re-scores one list using a model. It is not a retriever — it cannot search. It is not an LLM answering the question, though an LLM can be used as a reranker (§9). And it is not a filter, though a score threshold applied to reranker output is a reasonable way to drop weak candidates, and is far more meaningful than thresholding cosine similarity (07-02).

In the NVIDIA stack framing, reranking is part of what a retrieval service does to achieve retrieval accuracy at scale — NeMo Retriever is NVIDIA's retrieval offering and the NVIDIA AI Blueprint for RAG is the reference workflow that assembles these stages [NVIDIA-DOC]. 07-09 covers the full assembled pipeline.

L1 — The intuition you can carry into an exam

Bi-encoder: two people summarise a question and a document separately, then someone compares the two summaries. Fast, because the document summary is written once and reused for every question. Lossy, because the document was summarised without knowing the question.

Cross-encoder: one person reads the question and the document side by side and says how well the document answers it. Accurate. And it has to be repeated for every question–document pair, because the judgement depends on both.

Three facts to attach:

• The cross-encoder outputs a score, not a vector. Nothing is precomputable.
• Cost is per candidate, so pool size is a direct latency multiplier.
• Reranking fixes ordering, not coverage. Recall comes from stage one.

L2 — The mechanism

A cross-encoder is typically a transformer encoder (BERT-family, 04-03) with a classification head, fed a single sequence that contains both texts with a separator:

input:  [CLS] why does my client cert handshake fail [SEP]
              the server MUST abort the handshake when the presented
              certificate chain cannot be validated ... [SEP]

           ▼  full self-attention across BOTH segments
        ┌──────────────────────────────┐
        │  transformer encoder layers  │   query tokens attend to passage tokens
        └──────────────┬───────────────┘   and vice versa, at every layer
                       ▼
                  [CLS] vector
                       ▼
                 linear head → 0.94   (relevance score)

The critical line in that diagram is "query tokens attend to passage tokens". In a bi-encoder those two texts never meet inside the model; the only interaction is a dot product at the very end, between two vectors each computed in ignorance of the other. In a cross-encoder they interact at every attention layer. That is why the cross-encoder can do things similarity fundamentally cannot:

Cross-encoders are trained on labelled (query, passage, relevance) data — often derived from large relevance-judgement collections — with a classification or ranking objective. Because they are scoring rather than embedding, they can be smaller than you might expect and still outperform a much larger bi-encoder on ranking quality, and this is one of the reasons reranking is such a good deal in practice.

L3 — The cost model, and how to control it

The arithmetic is the whole engineering problem, and it is worth writing as a formula rather than a feeling.

rerank latency ≈ ceil(N / batch_size) × per_batch_forward_time
rerank cost    ≈ N × (query_tokens + passage_tokens) processed

Three consequences:

Latency is roughly linear in N. Doubling the candidate pool roughly doubles reranking latency (modulo batching efficiency). This makes N the single most important tuning parameter in the stage, and the one to set from measurement rather than from a default.

Cost is linear in passage length too. Reranking 50 candidates of 200 tokens each is a quarter the work of reranking 50 candidates of 800 tokens. Chunk size (06-02) therefore shows up as a reranking cost, which is a connection people miss. Long chunks are expensive twice: once in the context window (04-06) and once in the reranker.

Cross-encoder inputs have a length limit. The model has a maximum sequence length covering query and passage together. A chunk longer than that budget is truncated — silently, in most implementations — so the reranker may score a passage on its first half only. If your chunks exceed the reranker's window, you are reranking on partial evidence and will not be told.

Levers for controlling the cost:

The last one is worth naming because it is a genuine production pattern: if stage one returns a candidate with an overwhelming score and a large gap to second place, reranking will probably not change the outcome, so skip it. This is an optimisation to apply after you have measured, not before.

One structural caution. The reranker is on the critical path of every request, and it is usually a GPU-backed model call. That makes it a latency floor, a capacity planning item (12-10), and an availability dependency. A RAG system with a hard sub-100-ms budget may not be able to afford a reranker at all — which is one of the honest counter-cases for RAG architecture generally (07-12).

Four readings the exam probes.

Bi-encoder and cross-encoder are not competing designs; they are the two stages of one cascade. A question that offers "replace the bi-encoder with a cross-encoder for retrieval" is describing something computationally infeasible at corpus scale. A question that offers "add a cross-encoder to reorder retrieved results" is describing the correct architecture.

Fusion and reranking do different jobs and compose. Fusion (07-06) is a recall intervention that merges lists. Reranking is a precision intervention that reorders one list. The canonical order is retrieve → fuse → rerank → truncate, and the worked example in 07-06 ended precisely at the point where reranking is needed.

LLM-as-reranker and LLM-as-a-judge are different things. A reranker orders retrieved passages at query time, in production, on the critical path. A judge scores generated answers, usually offline, as part of evaluation (09-10). Both use an LLM; they sit in different parts of the system and are confusable in exam options.

The "swap cost" row is the sleeper advantage. Because a reranker stores nothing, you can A/B two rerankers this afternoon on the same index. Trying two embedding models requires embedding the corpus twice. That asymmetry means reranking is usually the cheapest experiment available in a RAG pipeline, and it is a good reason to reach for it early when quality is disappointing.

This continues the constructed example from 07-06 with invented scores, so the mechanism is inspectable end to end. These are not measurements.

The query was:

"getting ERR_CERT_AUTHORITY_INVALID when the client connects — why?"

And RRF fusion produced this ranking, with two acknowledged defects:

The two defects, restated: d2 — a correct mechanistic explanation — sank to rank 5 because it appeared in only one leg's list; and d4 — a pinning reference with no bearing on the question — rose to rank 4 because it scraped the bottom of both lists. RRF's bias toward mutual presence produced both.

If the context budget takes the top 3, the model receives d1, d3, and d6: one good chunk and two generic ones, while d2 sits unused.

Step 1 — score every candidate with the cross-encoder

Each of the six is concatenated with the query and passed through the model. Constructed relevance scores (0–1 after sigmoid):

d1  0.96    names the exact error AND explains its cause
d2  0.89    explains the mechanism precisely, without the error string
d6  0.54    correct topic, no specific answer
d3  0.31    an index of error pages; does not answer anything itself
d5  0.22    mentions the error string in a changelog context
d4  0.11    pinning configuration — different subject

Step 2 — read the reordering

Reranked: d1 (0.96) > d2 (0.89) > d6 (0.54) > d3 (0.31) > d5 (0.22) > d4 (0.11).

Compare the three configurations side by side:

Four specific things the reranker did that no earlier stage could:

It promoted d2 from rank 5 to rank 2. Reading the query jointly with d2, the model can see that "the client aborts the connection if the server certificate cannot be validated against a configured trust anchor" is the answer to "why does this happen when the client connects" — even with zero lexical overlap and only one leg's vote. Neither BM25, nor cosine similarity, nor RRF had any mechanism for recognising this.

It demoted d4 from rank 4 to last. Joint encoding reveals that a pinning configuration reference shares vocabulary with the query and answers nothing in it. Both retrievers were fooled by vocabulary; the reranker was not.

It separated d1 from d5 decisively — 0.96 versus 0.22 — even though both contain the exact error string. This is the "mention versus explanation" distinction, and it is the single clearest demonstration of what joint encoding buys. BM25 gave d5 rank 2 for containing a rare term; the reranker recognised that containing a term and explaining it are different things.

The scores are interpretable enough to threshold. There is a visible gap between d2 at 0.89 and d6 at 0.54, and another between d6 and d3 at 0.31. A threshold around 0.5 keeps d1, d2, and d6 and discards the rest — a defensible cut. Compare this with cosine similarity, where unrelated passages score 0.2–0.5 (07-02) and no natural cut exists. Reranker scores are more thresholdable than similarity scores, because the model was trained on a relevance objective rather than a similarity one. This is not a promise of calibration — you still have to pick the threshold on your own labelled data — but the signal is genuinely more separable.

Step 3 — the resulting context, and what it costs

The top-3 context is now d1, d2, d6: the direct answer, the mechanistic explanation, and useful background. Compare to hybrid-only's d1, d3, d6. The same retrieval, the same corpus, the same query — one extra stage, and one of the three context slots went from a generic index page to the second-best passage in the corpus.

The cost, stated honestly: six extra model forward passes for this toy example; in a realistic setting, N = 50 forward passes on the critical path of every request. If your budget cannot absorb that, you do not get this improvement — which is the trade-off, not a hidden catch.

Step 4 — what reranking could not fix

Suppose the corpus also contained d7: "ERR_CERT_AUTHORITY_INVALID means the CA that signed the certificate is not in the client's trust store; import the CA certificate to resolve it." — a chunk that is better than d1, because it also gives the remedy.

If neither leg retrieved d7, the reranker never sees it, never scores it, and cannot promote it. The final answer is missing the remedy, and no amount of reranking quality changes that.

This is the recall/precision division made concrete. d7's absence is a stage-one failure, addressed by better chunking, a better embedding model, deeper retrieval, or hybrid coverage — the 07-06 and 06-02 interventions. Reranking is powerless against it, and diagnosing which failure you have is precisely the discipline 07-10 teaches.

The diagnostic that makes the whole table usable, repeated because it is the highest-leverage habit in this module: retrieve top-50 for a failing query and find out where the correct chunk actually is.

Half of all RAG debugging time is saved by asking this before anything else, which is why 07-10 exists as its own lesson.

Reranking appears explicitly in the course's RAG coverage — the pipeline description includes ingest, chunk, overlap, and rerank as named stages — and it serves:

• 1.3 / 4.2 — Build LLM use cases such as RAG, chatbots, and summarizers [OFFICIAL]. Retrieve-and-rerank is standard production architecture.
• 1.8 — Select and use models to create text embeddings [OFFICIAL]. Knowing that a cross-encoder does not produce embeddings is part of understanding what an embedding model is for.
• 1.4 — Curate and embed content datasets for RAGs. Chunk length is a reranking cost as well as a context cost.
• 4.1 / 1.1 — Assist in deployment and evaluation of model scalability, performance, and reliability [OFFICIAL]. A per-candidate model pass on the critical path is a scalability decision.
• 4.4 — Identify system data, hardware, or software components required to meet user needs. A reranker is an additional model to serve.

On the stack side, NeMo Retriever is NVIDIA's retrieval offering aimed at retrieval accuracy at scale, and the NVIDIA AI Blueprint for RAG is the reference workflow [NVIDIA-DOC]. Reranking models are commonly served as inference microservices, which connects to the NIM and Triton material in 12-13. Note the [FIELD] calibration that NVIDIA-branded options tend to be favoured when two answers are technically defensible — useful as a tie-breaker, not as a substitute for understanding the architecture.

Exam depth is general-level [FIELD]. Know what a cross-encoder is, know that it scores jointly and cannot be precomputed, know it runs as a second stage over a small pool, and know it improves precision rather than recall. Do not memorise architecture details or training objectives.

Question phrasings you should recognise:

Distractor families:

• "Use a cross-encoder to search the corpus." Computationally infeasible; the definitive wrong answer.
• "Reranking improves recall." It cannot. It reorders an existing list.
• Reranking offered as the fix for recency, authority, or permissions. It helps with authority indirectly at best; the others need metadata (07-03, 07-05).
• "A reranker produces embeddings you store in the vector database." It produces a score. Nothing is stored.
• Reranking conflated with RRF fusion. Fusion merges lists by rank with no model; reranking re-scores one list with a model.
• Reranking conflated with LLM-as-a-judge. One orders retrieved passages in production; the other evaluates generated answers, usually offline (09-10).
• "Reranking is free because it does not touch the index." It is free of storage cost and expensive in per-request compute.
• "Increase k instead of reranking." More context is not better context — see 07-08 on lost-in-the-middle.

Mistakes 1 and 2 together account for most disappointing reranker deployments. The pattern is: quality is bad, a reranker is added because it is the well-known fix, the pool is left at top-5 because that is the existing retrieval config, and nothing improves — because the pool was thin and the real problem was recall. The corrective habit is the top-50 diagnostic, and it takes ten minutes.

Mistake 10 is worth building in from the start. The reranker is a network call to a model server on every request. Give it a timeout and a fallback to the pre-rerank ordering. A degraded ranking is a far better failure mode than an error page, and this is the kind of resilience the deployment objectives care about (12-14 on monitoring).

Because nothing about it is precomputable, and retrieval requires precomputation to be tractable.

Compare the work at query time for a corpus of N documents:

The asymmetry is structural, not a matter of optimisation. A bi-encoder's passage vector is independent of the query, which is exactly what lets you compute it once and store it. A cross-encoder's score is a function of both texts jointly, so it cannot exist before the query arrives. The cross-encoder's accuracy and its unusability for retrieval come from the same property. That trade-off is the single most exam-relevant fact in this lesson, and it also explains why the two-stage cascade is not a workaround but the correct design: cheap-and-precomputable for the wide filter, expensive-and-joint for the narrow decision.

A related question people ask: why not use a cross-encoder to build a better index? You cannot, because there is no vector to index. Some research directions try to recover part of the cross-encoder's accuracy in a precomputable form — late-interaction models (ColBERT-style) store per-token vectors and do a cheap interaction at query time, landing between bi-encoder and cross-encoder on both cost and quality. Know that this middle ground exists; the exam's taxonomy is bi-encoder retrieval plus cross-encoder reranking.

Yes, and it is a real technique with a clear trade-off.

How it works. Present the query and the candidate passages to an instruction-following LLM and ask it to rank them, score each, or select the most relevant. Variants include scoring one passage at a time, comparing pairs, and ranking a whole list in one prompt.

The honest summary: a cross-encoder is the right default for a production reranking stage because it is cheap, fast, deterministic, and purpose-trained. An LLM reranker is worth considering when queries carry complex multi-part constraints that benefit from actual reasoning, when volume is low enough that the cost is acceptable, or when you are prototyping and do not want to deploy another model.

Two cautions specific to LLM reranking. List-wise prompts have positional bias: the model's judgement is influenced by where a passage sits in the prompt, which is the same attention phenomenon that 07-08 covers as lost-in-the-middle. Shuffling and averaging across permutations mitigates it and multiplies the cost. And output parsing is a failure surface: an LLM asked to return a ranked list of ids will occasionally return prose, a hallucinated id, or a truncated list, so the code path needs validation and a fallback (05-05 on structured output).

Do not confuse either with LLM-as-a-judge (09-10), which scores generated answers during evaluation rather than retrieved passages during serving.

By extending the same frozen-eval-set table this module has built since 07-01, and by measuring the right metric — which for reranking is not recall@10.

Use rank-sensitive metrics. Recall@10 asks whether the correct chunk is in the top 10. Reranking's job is to move it up within those 10, so recall@10 may not move at all while quality improves substantially. Metrics that see the improvement:

09-05 covers metric selection generally and 09-07 covers RAG-specific metrics; the point here is narrower: a precision intervention measured with a recall metric will look like it did nothing. That mismatch is a common reason teams wrongly conclude reranking was not worth it.

The measurement sequence:

The three N rows are the point. Reranking is not one decision but a pool-depth curve, and the curve has a knee: quality rises with N and then flattens while latency keeps climbing linearly. Find the knee on your own data and set N there. A default of 100 copied from a blog post is a latency bill you may be paying for nothing.

Two further checks:

Look at the queries that changed, not just the average. Confirm they moved for the reason you expect — a near-miss demoted, a paraphrase promoted, a mention-versus-explanation distinction made. If the wins are unexplainable, you may be measuring noise on a small eval set (09-09 on sample size).

Measure retrieval separately from generation. Reranking changes what enters the context; whether the answer improves also depends on assembly (07-08) and grounding (07-11). Attribute at the stage boundary, which is the whole method of 07-10.

• A cross-encoder reads query and passage together and outputs one relevance score. Joint encoding is why it beats a bi-encoder; not being precomputable is why it cannot retrieve.
• Reranking improves precision, never recall. A chunk absent from the candidate pool cannot be promoted.
• The worked example's headline result: the reranker separated d1 at 0.96 from d5 at 0.22, even though both contain the exact same rare error string. BM25 ranked d5 second for containing the term; the reranker recognised that mentioning a term and explaining it are different things.
• Same example, second result: the second-best chunk moved from rank 5 to rank 2, rescuing exactly the document RRF's mutual-presence bias had buried.
• Cost is per candidate, on the critical path. Latency is roughly linear in pool size N, so N must be set from a measured quality/latency curve, not from a default.
• Reranking is the cheapest experiment in a RAG pipeline because nothing is stored — unlike changing the embedding model, which means re-embedding the corpus (12-12).
• Measure it with rank-sensitive metrics — recall@3, MRR, nDCG@5. Recall@10 will often show no change while quality has improved substantially.
• Diagnose before intervening. Correct chunk at rank 4–50 means rerank; absent from the top-50 means fix stage one; at rank 1 with a wrong answer means the problem is generation.
• Give it a timeout and a fallback. Degrading to the fused ranking beats failing the request.

You now have a small, correctly ordered set of passages. The obvious next move — paste them into the prompt and ask the question — turns out to contain a decision nobody mentions: in what order? Models do not attend uniformly across a long context, and information placed in the middle is systematically under-used compared with the same information placed at the start or the end. Next: 07-08 covers context assembly and the lost-in-the-middle effect: why a carefully reranked list can be squandered by a careless paste, how many chunks is too many, and why "more context" and "better context" are not the same thing.