Hybrid search: combining keyword and vector retrieval in RAG

Hybrid search is a two-leg retrieval architecture with a fusion step:

                        user query
                            │
              ┌─────────────┴─────────────┐
              ▼                           ▼
   ┌────────────────────┐      ┌─────────────────────┐
   │ sparse leg (BM25)  │      │ dense leg (vectors) │
   │ inverted index     │      │ ANN index           │
   │ returns top-k_s    │      │ returns top-k_d     │
   └─────────┬──────────┘      └──────────┬──────────┘
             │      ranked list A          │  ranked list B
             └──────────────┬──────────────┘
                            ▼
                  ┌───────────────────┐
                  │  fusion (RRF)     │  ← one ranking
                  └─────────┬─────────┘
                            ▼
                  optional reranking (07-07)
                            ▼
                     context assembly (07-08)

Three design parameters, and it is worth being precise about them because they are commonly conflated:

What hybrid search is not. It is not a single retriever that does both things. It is not an embedding model with keyword awareness. And it is not the same as query expansion — rewriting the query with synonyms before a single-leg search — although the two are often confused because both aim at vocabulary coverage. Two legs and a fusion step is the architecture; anything with one retrieval index is not hybrid search, whatever it is doing to the query.

The other name you will see is "hybrid retrieval", and in some vector databases the feature is exposed as "sparse-dense search" or "sparse vectors alongside dense vectors". These are the same idea. Note that "sparse vectors" in that phrasing sometimes means a learned sparse representation — SPLADE-style models that produce term weights from a neural network rather than from BM25 arithmetic — which is a genuine third option and covered briefly in §9.

L1 — The intuition you can carry into an exam

Ask two specialists the same question and merge their answers, trusting a document more when both nominate it and when either nominates it highly. Merge by position in each list, not by score, because the two specialists grade on incompatible scales.

Three facts to carry:

• RRF is the default fusion method because ranks are comparable across retrievers and raw scores are not.
• Documents appearing in both lists get promoted, which is the property you actually wanted — agreement is evidence.
• Hybrid search costs two retrievals per query, so latency is roughly the slower leg plus fusion, and both legs can run in parallel.

L2 — The mechanism: RRF and its alternative

Reciprocal rank fusion. For each document d, sum a reciprocal-rank term over every result list it appears in:

RRF(d) = Σ over lists L :   1 / (k_rrf + rank_L(d))

where rank_L(d) is d's 1-based position in list L, and k_rrf is a smoothing constant conventionally set to 60. Documents absent from a list contribute nothing from that list.

Read the shape rather than the formula. With k_rrf = 60:

Two behaviours follow, and they are the whole reason k_rrf exists:

The curve is deliberately flat at the top. Rank 1 and rank 3 differ by about 3%. That means RRF does not treat "first" as vastly better than "third" within one list — which is correct, because neither retriever's top-3 ordering is trustworthy at that resolution. A smaller k_rrf sharpens the curve and lets a single leg's rank-1 dominate; a larger one flattens it further and weights agreement across legs more heavily.

Appearing in both lists usually beats topping one. A document at rank 3 in both lists scores 0.01587 + 0.01587 = 0.03174. A document at rank 1 in one list and absent from the other scores 0.01639. The two-list document wins by nearly 2×. That is the fusion doing its job: cross-retriever agreement is stronger evidence than one retriever's confidence.

Weighted score fusion, the alternative, normalises each leg's scores to a common range and takes a weighted sum:

score(d) = α × norm(bm25_score(d)) + (1 − α) × norm(cosine_score(d))

It is more expressive — you can tune α toward whichever leg your eval set says helps more — and it is more fragile, for reasons that matter:

The practical rule: start with RRF, move to weighted fusion only if a real eval set shows a real gain, and re-verify after any corpus change. RRF's parameterlessness is a feature, not a limitation.

L3 — Where fusion sits relative to filtering and reranking, and why the order is fixed

The pipeline order is not arbitrary and each position has a reason:

The step-1 note is the one that gets missed in real systems and it is a security defect, not a quality one. If the dense leg is permission-filtered and the sparse leg is not, the sparse leg leaks. Both legs are retrieval, so both legs carry the predicate. This is one of the underrated arguments for running hybrid search inside a single system that already does both — Elasticsearch/OpenSearch with a vector field, or a vector database with BM25 support — because there is then one filter expression rather than two that must agree.

A second L3 subtlety: fusion depth interacts with reranking. RRF over top-10 from each leg gives a fused pool of at most 20 unique documents, which is a thin pool for a reranker to work with. Retrieving top-50 or top-100 per leg and fusing gives the reranker a genuine choice. The cost is that reranking is per-candidate, so pool depth is a direct latency multiplier. The canonical shape is deep retrieve → fuse → rerank → shallow context, with each stage narrowing.

Third: the two legs need not return the same document granularity. If your sparse index is over whole documents and your dense index is over chunks, fusion has to reconcile them — either by rolling chunk hits up to their parent document or by expanding document hits into their chunks. Mismatched granularity across legs is a real source of silently poor fusion, and the fix is to make both legs index the same units.

Four readings that exam items probe.

Hybrid is a recall intervention; reranking is a precision intervention. Hybrid gets the right chunk into the candidate pool. Reranking gets it to the top of the pool. They compose, and neither substitutes for the other — which is exactly why this lesson and 07-07 are taught as one paired session.

The three "no" rows are the exam's favourite trap. Recency, authority, and permissions are unaffected by hybrid search. A scenario describing stale documents or a leak, with "enable hybrid search" among the options, is testing whether you know the difference between a coverage problem and a metadata problem.

Query expansion is a different intervention with a similar goal. Rewriting "why is my thing slow" into "why is my request slow OR high latency OR timeout" widens a single leg's lexical reach. It is cheaper than a second index and weaker than one, because it depends on guessing the corpus's vocabulary. Multi-turn query rewriting — resolving "what about the second one?" into a self-contained question — is a related but distinct technique covered in 12-11.

Hybrid search can make things worse. This is the honest caveat. If one leg is badly broken — a mismatched encoder (07-02), or an analyser mismatch in the sparse index (07-01) — fusing it with a working leg injects noise into a ranking that was fine. Fusion assumes both legs are individually competent. Measure each leg alone before fusing them, which is the discipline 07-01 set up.

This is a constructed illustrative example. The ranks, BM25 scores, and cosine similarities are invented so the arithmetic is checkable. They are not measurements.

A user of an internal support assistant asks:

"getting ERR_CERT_AUTHORITY_INVALID when the client connects — why?"

The query mixes both signal types, which is why it is the right example: ERR_CERT_AUTHORITY_INVALID is a rare exact identifier (sparse territory) and "when the client connects — why?" is natural language about a concept (dense territory).

Six candidate chunks exist in the corpus:

Step 1 — the sparse leg (BM25, top-5)

ERR_CERT_AUTHORITY_INVALID is a maximally rare term, so both chunks containing it dominate. Constructed BM25 scores:

rank 1  d1   9.84    contains the exact error string, and explains it
rank 2  d5   8.91    contains the exact error string, but it is a changelog line
rank 3  d3   2.15    "client connection errors"
rank 4  d6   1.62    "connections fail"
rank 5  d4   0.94    "certificate"

Note d2 is absent. It is arguably the second-best chunk in the corpus — it explains the mechanism correctly — and it shares almost no terms with the query: no ERR_CERT_AUTHORITY_INVALID, no getting, no why. This is the vocabulary-mismatch failure from 07-01, live.

Note also d5 at rank 2. A changelog mention of the error string is nearly useless as an answer, and BM25 ranks it second because IDF cannot distinguish "explains the error" from "mentions the error". This is the precision problem that reranking exists for.

Step 2 — the dense leg (cosine, top-5)

Constructed similarities:

rank 1  d2   0.83    "cannot be validated against a configured trust anchor"
rank 2  d1   0.79    the correct explanatory chunk
rank 3  d6   0.74    "TLS handshake failure causes"
rank 4  d3   0.68    "common client connection errors"
rank 5  d4   0.55    "certificate pinning reference"

The dense leg recovered d2 — the chunk sparse retrieval could not see — and it correctly demoted d5 out of the top-5, because a changelog line about a release is semantically distant from a question about why a connection fails. It also put d2 above d1, which is arguably wrong: d1 names the exact error. Dense retrieval's identifier weakness (07-02) in miniature.

Neither leg's top-1 is unambiguously correct. Sparse says d1 (right, for a fragile reason). Dense says d2 (a good chunk, but not the one that names the error). This is the situation fusion is for.

Step 3 — fuse with RRF, k_rrf = 60

For each document, sum 1/(60 + rank) over the lists it appears in:

d1:  sparse rank 1 → 1/61 = 0.016393
     dense  rank 2 → 1/62 = 0.016129
                     RRF  = 0.032522     ← highest

d2:  sparse absent → 0
     dense  rank 1 → 1/61 = 0.016393
                     RRF  = 0.016393

d3:  sparse rank 3 → 1/63 = 0.015873
     dense  rank 4 → 1/64 = 0.015625
                     RRF  = 0.031498     ← second

d4:  sparse rank 5 → 1/65 = 0.015385
     dense  rank 5 → 1/65 = 0.015385
                     RRF  = 0.030770     ← third

d5:  sparse rank 2 → 1/62 = 0.016129
     dense  absent → 0
                     RRF  = 0.016129

d6:  sparse rank 4 → 1/64 = 0.015625
     dense  rank 3 → 1/63 = 0.015873
                     RRF  = 0.031498     ← tied second

Fused ranking: d1 (0.032522) > d3 ≈ d6 (0.031498) > d4 (0.030770) > d2 (0.016393) > d5 (0.016129).

Step 4 — read the result honestly, including what went wrong

The good news first:

• d1 is now unambiguously first. It was rank 1 in one list and rank 2 in the other, and dual appearance promoted it above everything. The system got the right answer to the top for the right reason: two independent retrievers agreed.
• d5 collapsed to last. The changelog line was rank 2 in sparse and absent from dense, and fusion correctly discounted a document only one leg liked.
• d2 entered the pool at all. Sparse-only retrieval would never have surfaced it.

Now the bad news, which is the more instructive half:

• d2 — the second-best chunk in the corpus — fused to rank 4, below d3, d6, and d4, all of which are generic and less useful. Why? Because d2 appeared in only one list, and RRF's structure means one appearance rarely beats two mediocre appearances. d3 at ranks 3 and 4 scored 0.031498; d2 at rank 1 in one list scored 0.016393.
• d4 — a pinning configuration reference, largely irrelevant — fused to rank 3, purely because it scraped rank 5 in both lists.

This is RRF's real and inherent bias: it rewards mutual presence over single-leg conviction. Usually that is what you want, because agreement is evidence. Sometimes it buries a genuinely excellent single-leg hit under a pile of mediocre mutual ones.

The correction is not to abandon RRF. It is:

1. Fuse a deeper pool. With top-5 per leg the pool is tiny and mediocre documents crowd it. At top-50 per leg, d3/d4/d6-class documents sit at ranks 20–40 where their contributions are visibly smaller, while d2 still holds rank 1 in the dense list.
2. Rerank the fused pool. A cross-encoder reading the query jointly with d2 will recognise that it answers the question, and with d4 that it does not. Reranking is the stage that fixes exactly the defect this example exposes — which is why this lesson and 07-07 are a paired session and why the module's design note says the two corrections are close to meaningless apart.
3. Do not tune k_rrf to patch it. Lowering k_rrf sharpens the top of the curve and would promote d2, and it would also make the whole ranking hostage to each leg's rank-1 pick. That is a worse trade.

Step 5 — what the comparison table shows

Hybrid search fixed the top of the ranking and left the rest of the pool imperfectly ordered. That is precisely the division of labour: hybrid search is responsible for the right chunk being in the pool and near the top; reranking is responsible for the pool being correctly ordered.

The diagnostic that makes this table usable: distinguish "not retrieved" from "retrieved but ranked low." Those are different failures with different fixes, and telling them apart requires looking at the retrieved candidates rather than the final answer. Retrieve top-50 for a failing query and search the list for the correct chunk:

• Absent from the top-50 → a recall problem → hybrid search, chunking (06-02), or the embedding model (03-03).
• Present but at rank 23 → a precision/ordering problem → reranking (07-07).
• Present at rank 1 and the answer is still wrong → a generation problem → 07-10, 07-11.

That three-way split is the most valuable half-hour in RAG debugging and 07-10 formalises it.

Hybrid search is named explicitly in the course's retrieval coverage, alongside dense-versus-sparse retrieval, ANN indexing, the recall/latency trade-off, metadata filtering, and when a vector database is unnecessary. It serves:

• 1.3 / 4.2 — Build LLM use cases such as RAG, chatbots, and summarizers [OFFICIAL]. Hybrid retrieval is standard practice in a production RAG build.
• 1.4 — Curate and embed content datasets for RAGs. Whether a corpus needs a sparse leg is a corpus-shape judgement made during curation.
• 1.6 / 4.3 — Familiarity with the capabilities of Python natural language packages (spaCy, NumPy, vector databases, etc.) [OFFICIAL]. Hybrid support is one of the capabilities that distinguishes vector databases from ANN libraries.
• 4.4 — Identify system data, hardware, or software components required to meet user needs. Two indexes plus a fusion step is a component decision.

On the NVIDIA stack side, NeMo Retriever is the product framing for retrieval accuracy at scale and the NVIDIA AI Blueprint for RAG is the reference workflow [NVIDIA-DOC]; 07-09 covers both when the full pipeline is assembled. It is also worth remembering the [FIELD] calibration that NVIDIA-branded answers tend to be favoured when two options are technically defensible — which is about the stack map (12-13 covers Triton and NIM; 13-01 the trust principles) rather than about retrieval algorithms, but it is a useful tie-breaker.

Exam depth is general-level [FIELD]. Know that hybrid search means two legs plus fusion, know RRF by name and know why rank-based fusion avoids the score-calibration problem, and know the coverage/metadata distinction. Do not memorise k_rrf derivations.

Question phrasings you should recognise:

Distractor families:

• "Hybrid search means using two embedding models." Two dense models is ensembling, not hybrid search. Hybrid means sparse plus dense.
• "Hybrid search means query expansion." One index with a rewritten query is not two legs and a fusion step.
• "Normalise BM25 and cosine scores and average them." Offered as the standard method. It is a method, it is the fragile one, and its fragility is the reason RRF exists.
• Hybrid offered as the fix for recency, authority, or permissions. The highest-value distractor family in this lesson.
• Hybrid offered as the fix for a ranking-order problem. That is reranking.
• "Hybrid search always improves retrieval." It can dilute a good leg with a broken one.
• "Hybrid search doubles latency." The legs parallelise; cost is closer to the slower leg plus fusion overhead than to the sum.
• "Hybrid search requires a single unified index." It does not, although one system supporting both is operationally simpler and makes the shared filter easier.

Mistake 10 is the methodological one and it recurs through this whole module. The four-row table — sparse alone, dense alone, hybrid, hybrid plus rerank, each with recall@5 and recall@10 on the same frozen questions — is the artefact that makes every later decision defensible. Build it once and it pays for the rest of the project.

Because the two retrievers' scores are not on a common scale and cannot be put on one without introducing a calibration that drifts.

Look at what each leg actually emits. BM25 returns an unbounded positive sum of per-term contributions whose magnitude depends on the query's term rarity, the number of query terms, document lengths, and the collection's IDF distribution. A score of 9.84 means nothing on its own — for a one-rare-term query it might be exceptional, and for a five-term query, mediocre. Worse, it shifts as the corpus grows, because IDF is a function of collection statistics.

Cosine similarity returns a value in a compressed positive band, typically 0.2–0.9 for text embeddings, where unrelated text already scores 0.2–0.5 (07-02). The distribution is model-specific and corpus-specific and is not calibrated to anything.

Now try to combine them. You need a mapping from each leg's scale to a shared one, and every available option is flawed:

Rank sidesteps all of it. "Third in the sparse list" is a statement whose meaning does not depend on scale, corpus size, query length, or model version. It is directly comparable to "third in the dense list". That comparability is bought at a real price — RRF discards magnitude information, so it cannot tell that sparse rank 1 scored 9.84 while sparse rank 2 scored 8.91 rather than 0.94 — and the trade is generally judged worth it, because a robust method with no tuning parameters beats a sharper method whose parameters silently decay.

When is score fusion the better choice? When you have a substantial labelled eval set, a stable corpus, and a measured gain you can re-verify on a schedule. That is a real situation and it is not the default one.

Not necessarily, and the options differ mainly in operational cost.

The last row is worth taking seriously for the same reason 07-04 argues against premature vector databases: at small corpus scale, hybrid search is fifteen lines of code, not an architecture. You compute BM25 scores, you compute cosine similarities, you sort each, you sum reciprocal ranks. There is no reason for a team with 8,000 chunks to defer hybrid search until they have deployed two search systems.

The learned-sparse row deserves a calibration note. SPLADE-style retrieval is real and increasingly used, and it blurs the sparse/dense boundary this module has drawn cleanly: it produces a sparse, interpretable, term-weighted vector from a neural network, so it gets some of dense retrieval's expansion behaviour while remaining searchable by an inverted index. For the exam, the taxonomy that matters is sparse (lexical, BM25) versus dense (embedding, cosine), with hybrid as their combination. Know that learned sparse representations exist; do not expect to be tested on them.

By building the four-row table this module has been pointing at since 07-01, on a frozen labelled eval set, changing one thing at a time.

Setup. Take 20–40 real questions — from users, tickets, or colleagues, never invented, because invented questions are written in the corpus's vocabulary and quietly favour whichever leg you already prefer. For each, label the specific chunk id (or ids) that genuinely answer it. 01-08 and 03-04 cover the discipline.

Measure, in this order:

Read it with three specific questions:

1. Did hybrid beat the better single leg on recall@10? If not, the legs are not as complementary as assumed on your corpus, or one leg is broken. Investigate rather than shipping.
2. Did recall@5 improve more than recall@10? recall@10 measures whether the chunk is in the pool; recall@5 measures whether fusion pushed it up. A gain concentrated in recall@10 means hybrid added coverage; a gain in recall@5 means it also improved ordering.
3. Which specific questions changed? The aggregate hides the mechanism. Look at the queries that flipped from fail to pass and confirm they flipped for the reason you expect — an exact identifier now found, or a paraphrase now matched. If they flipped for reasons you cannot explain, you do not yet understand your retriever.

Measure retrieval, not answers, at this stage. End-to-end answer quality moves for many reasons and cannot attribute a change to retrieval. Separating retrieval quality from generation quality is the core discipline of 07-10 and the metric decomposition in 09-07.

Then leave the table in the repository. Six weeks later someone will propose removing the sparse leg to simplify the stack, and this table is the only thing that will answer them.

• Hybrid search is two retrieval legs plus a fusion step. Sparse covers exact identifiers and unseen vocabulary; dense covers synonyms, paraphrase, and cross-lingual matching.
• RRF is the default because it fuses ranks, not scores. BM25 scores are unbounded and corpus-dependent; cosine similarities are compressed and uncalibrated. Rank is scale-free.
• The worked example's headline result: d1 appeared at rank 1 in one list and rank 2 in the other, fusing to 0.0325 — roughly double the 0.0164 of a document that topped one list and was absent from the other. Mutual presence is the evidence RRF rewards.
• The same example's honest failure: the second-best chunk fused to rank 4, buried under mediocre documents that appeared in both lists. RRF's bias toward mutual presence is real, and reranking is what corrects it.
• Hybrid search is a recall intervention; reranking is a precision intervention. They compose and neither substitutes for the other.
• Hybrid search does nothing for recency, authority, or permissions. BM25 is exactly as blind to all three as the embedding model. This is the exam's favourite distractor family.
• Apply the permission filter in every leg. One filtered leg and one unfiltered leg is an unfiltered system (07-05).
• Fuse deep, rerank, then narrow. A top-5-per-leg pool is too thin for either fusion or reranking to work well.
• Hybrid can hurt if one leg is broken. Measure each leg alone before fusing, and keep the four-row table.

Fusion got the right chunk into the pool and usually near the top, and the worked example showed exactly where it stops: the pool's ordering is only as good as two retrievers that each score a passage without ever reading it alongside the query. Next: 07-07 introduces the cross-encoder — a model that encodes query and passage jointly and therefore can tell that a changelog line merely mentions an error code while another passage explains it. It is the most reliable single quality improvement available in a RAG pipeline, it is priced per candidate rather than per query, and it is the reason this lesson and the next are one paired session rather than two.