Dense retrieval with embeddings: how vector search finds meaning

Dense retrieval is retrieval by vector similarity in a learned embedding space. Three things define it, and each contrasts directly with the sparse method from 07-01:

The representation is dense and fixed-length. Where BM25 gives you a 50,000-dimension vector with 90 non-zero entries, an embedding model gives you a 384-, 768-, 1024-, or 3072-dimension vector in which essentially every entry is non-zero and meaningful. Dimensionality no longer scales with vocabulary; it is a property of the model. 03-01 establishes what those learned dense vectors are and 03-02 distinguishes token embeddings from the sentence and document embeddings retrieval actually needs.

The representation is learned, not counted. BM25 computes its weights from the corpus arithmetically. An embedding model produces its vectors by running text through a trained neural network — typically a transformer encoder, often BERT-family (04-03), fine-tuned specifically so that semantically related text lands near related text. That training is what buys you synonym and paraphrase handling, and it is also what introduces every dependency and failure mode in this lesson.

Similarity replaces overlap. There is no notion of a "matched term". There is a geometric distance, and a passage either sits near the query in that space or it does not. M0.1 gives you the dot product and cosine similarity mechanics; 03-05 shows the vector-arithmetic intuition that made the idea famous.

The practical shape of a dense retriever:

INDEX TIME (once per corpus version)
  for each chunk:
      vector = passage_encoder(chunk_text)      # 768 floats, say
      store (chunk_id, vector, metadata)

QUERY TIME (every request)
  q = query_encoder(user_question)              # same 768 floats
  candidates = nearest_neighbours(q, index, k)  # cosine similarity
  return top-k chunks

Two facts about that pseudocode carry most of the lesson's weight. First, the corpus is encoded once and the query is encoded per request — which is why dense retrieval is cheap at query time and expensive to change. Second, passage_encoder and query_encoder must be compatible. In most modern models they are literally the same weights used twice. In some — the dual-encoder or "bi-encoder" designs descending from Dense Passage Retrieval — they are two separately trained towers, and using tower A on the query with tower B's stored vectors produces garbage that looks exactly like valid output.

L1 — The intuition you can carry into an exam

Every piece of text becomes a point in a high-dimensional space. The embedding model has been trained so that text about the same thing lands in the same neighbourhood, whatever words it used. Retrieval is "find the points nearest my question's point."

That is genuinely all of it. If you can hold "text becomes a point, related text lands nearby, retrieval is nearest-neighbour search," you can answer most exam questions on this topic. The three follow-on facts to attach:

• Cosine similarity is the standard distance measure, ranging from −1 (opposite) through 0 (unrelated) to 1 (identical direction). It measures angle, not magnitude, which is why it is preferred over raw Euclidean distance for text.
• Every vector in the index must come from the same model version. A vector from model A and a vector from model B live in unrelated spaces and cannot be compared, even at identical dimensionality.
• Changing the embedding model means re-embedding the entire corpus. There is no migration path (12-12).

L2 — The mechanism: bi-encoders and the vector space

The dominant architecture for retrieval embeddings is the bi-encoder, also called a dual encoder. Query and passage are encoded independently:

   "why is my request refused upstream"          "the gateway returned 503 when
              │                                   no worker was available"
              ▼                                            ▼
     ┌─────────────────┐                          ┌─────────────────┐
     │  query encoder  │                          │ passage encoder │
     └────────┬────────┘                          └────────┬────────┘
              ▼                                            ▼
        q = [0.021, −0.118, …]                     p = [0.019, −0.104, …]
              └──────────────── cos(q, p) = 0.84 ──────────┘

The independence is the entire reason dense retrieval is practical. Because the passage vector does not depend on the query, you can compute all of them offline and store them. At query time you run the encoder once and then do arithmetic. Contrast this with a cross-encoder, which takes query and passage together as one input and outputs a relevance score — far more accurate, and completely unusable for first-stage retrieval because it would require one forward pass per candidate passage. That trade-off is the subject of 07-07, and understanding it here is what makes reranking make sense there.

How does a transformer that outputs one vector per token produce one vector per passage? By pooling. The common strategies:

You do not choose this at will — you must use the pooling the model was trained with. Mean-pooling a model trained for CLS pooling produces vectors that are not wrong-looking, merely worse, and the degradation is invisible without measurement. This is the second silent-failure family in the lesson.

Most retrieval embedding models are then L2-normalised, meaning every vector is scaled to unit length. Once vectors are unit-length, cosine similarity and dot product are the same operation, which is why vector databases let you pick either and why the choice often does not matter. It matters enormously if your vectors are not normalised: dot product then rewards long vectors, and "long" in embedding space frequently correlates with nothing you care about.

L3 — Why the model must be trained on retrieval, and what asymmetric encoding means

A crucial and frequently missed point: a general-purpose language model's hidden states are not good retrieval embeddings. Taking raw BERT and mean-pooling it gives famously mediocre retrieval quality. Retrieval embedding models are trained with a contrastive objective: given a query, pull the correct passage's vector closer and push unrelated passages' vectors away. The training data is (query, relevant passage, irrelevant passages) triples, and the quality of the negatives — especially hard negatives, passages that look relevant but are not — largely determines the model's quality.

This training regime creates an asymmetry that surprises people. A query and a passage are different kinds of text: a query is short, often ungrammatical, and interrogative; a passage is long, declarative, and complete. A model trained to map them into one shared space must handle both shapes. Three designs exist:

The prefix case deserves emphasis because it is the modern trap. Several widely used embedding models require an instruction prefix and will happily embed text without one. Nothing errors. Vectors come out. Similarities are computed. The numbers look like the numbers you expected. Retrieval quality is meaningfully worse and you will not know unless you measured, which is why 03-04 and 01-08 come before this lesson in the course rather than after it.

The general statement of the danger, and the design note this lesson exists to deliver: mismatched query and passage encoders return plausible nonsense with high scores and no error. Every layer of the stack behaves correctly. The vector database returns the nearest neighbours it was asked for. The similarity scores are real cosine similarities. The LLM synthesises a fluent answer from the passages it was given. The only thing wrong is that the passages have nothing to do with the question, and the only way to detect it is to look at what was retrieved — which is precisely the discipline of 07-10.

Three readings that exam items probe:

Dense and sparse are complements, not competitors. Their failure sets barely overlap. That fact is the entire argument for hybrid search in 07-06, and it is why mature systems keep both permanently rather than migrating from one to the other.

Dense retrieval and reranking are different layers, not alternatives. Bi-encoders retrieve; cross-encoders reorder. A distractor that offers "use a cross-encoder to search the corpus" is describing something computationally infeasible at corpus scale.

"Semantic search", "vector search", "embedding search", and "dense retrieval" are the same thing in exam language. Recognise all four phrasings as pointing at this lesson.

This is a constructed illustrative example. The cosine similarities below are invented so the reasoning is inspectable; they are not measurements from any model or benchmark.

A support knowledge base has been chunked (06-02) and embedded. Four chunks, with their text shown in shortened form:

The user asks: "why are my requests being refused upstream?"

Step 1 — what BM25 does with this. The query's discriminating terms are requests, refused, upstream. Suppose requests and refused are common in this corpus (low IDF) and upstream appears only in c1. But c1 says "upstream" — good — while c2, c3, and c4 each match two query terms. Depending on the exact IDF values and lengths, a plausible BM25 ranking is c4 or c2 first, with c1 mid-pack. The unquestionably correct chunk, c1, is the one that describes the actual mechanism, and it wins only on the single term upstream. Notice also that c4 — an expense-policy chunk with nothing to do with the question — ranks highly purely on lexical coincidence. This is the vocabulary-mismatch failure of 07-01 in miniature.

Step 2 — encode the query. One forward pass through the embedding model, producing a unit-length 768-dimension vector. Cost: one model call, typically single-digit milliseconds on GPU for a short query.

Step 3 — cosine similarity against the four stored passage vectors. Constructed values:

cos(q, c1) = 0.81      "no upstream worker … gateway responds 503"
cos(q, c2) = 0.66      "requests are refused when the certificate expired"
cos(q, c3) = 0.48      "dashboard shows refused requests per minute"
cos(q, c4) = 0.29      "expense requests are refused after 90 days"

Step 4 — read the ranking. c1 > c2 > c3 > c4. The chunk sharing zero content words with the query ranks first, because "requests being refused upstream" and "no upstream worker available, gateway responds 503" occupy the same region of the learned space. The expense-policy chunk, which lexically matched two query terms, is correctly pushed to last because its meaning is unrelated.

Step 5 — read what the numbers do not tell you. Three cautions, all of them exam-relevant:

• 0.29 is not "no match". Every passage has a similarity. Cosine similarity over normalised text embeddings is typically compressed into a narrow positive band — unrelated text often scores 0.2–0.5 rather than 0.0. You cannot set a universal relevance threshold. A threshold must be calibrated per model and per corpus, and a value that works for one embedding model is meaningless for another.
• The gap between 0.81 and 0.66 is not a probability. It is not "23% more relevant". Cosine similarities are ordinal signals for ranking, not calibrated confidences. Treating them as probabilities is a common and consequential error.
• c2 at 0.66 will very likely be handed to the LLM in a top-3 retrieval, and it says requests are refused because a certificate expired. If the model is not instructed to ground its answer and cite (07-11), it may well produce a fluent answer about certificate expiry. Retrieval succeeded; the context still contained a plausible wrong answer.

Step 6 — now break it deliberately. Suppose the corpus was embedded with the passage prefix "passage: " as the model requires, but the query is embedded with no prefix. Constructed post-break values:

cos(q, c1) = 0.58
cos(q, c2) = 0.61
cos(q, c3) = 0.55
cos(q, c4) = 0.54

Every score is still in a normal-looking range. The ranking has changed — c2 now leads — and the spread has collapsed, so the retriever is close to returning an arbitrary order. No error was raised. No log line appeared. The vector database was not at fault. The system now confidently answers questions about worker pools with information about certificates. This is the single most important operational fact in this lesson, and the reason it is emphasised twice.

The decision procedure, stated as a sequence rather than a preference:

1. Measure BM25 first (07-01) on a frozen labelled eval set (01-08).
2. Add dense retrieval and measure the same recall@k on the same set. Choosing the model itself is 03-03.
3. If dense does not beat sparse on your corpus, stop and diagnose. The usual culprits, in order of likelihood: a broken prefix or pooling convention; chunks too long, so a chunk's vector is an average of five topics (06-02); boilerplate dominating chunks (06-04); parsing that silently mangled the text (06-01); an eval set written in the corpus's own vocabulary and therefore not testing the thing dense retrieval is for.
4. Then combine both and measure again (07-06), then add reranking and measure again (07-07). One change at a time.

Dense retrieval sits on the shortest possible path from the official objectives to a question:

• 1.8 — Select and use models to create text embeddings. This is the objective, near verbatim. Selecting an embedding model is the dense-retrieval decision.
• 1.4 — Curate and embed content datasets for RAGs. "Embed content datasets" is the index-time half of this lesson.
• 1.3 / 4.2 — Build LLM use cases such as RAG, chatbots, and summarizers. The retrieval stage is where a RAG use case is won or lost.
• 1.6 / 4.3 — Familiarity with the capabilities of Python natural language packages (spaCy, NumPy, vector databases, etc.). Vector databases are named in the objective text, and they exist to serve dense retrieval.

The course index places NVIDIA's pipeline description at the centre of its RAG coverage: query → embedding model → vector match against indexed knowledge base → retrieve and decode → LLM synthesises and cites sources [NVIDIA-DOC]. Learn that ordering as an ordering. Component-order questions are explicitly a drill emphasis, and the second element of the sequence is the subject of this lesson. On the NVIDIA stack side, NeMo Retriever is the product framing for retrieval accuracy at scale, and the NVIDIA AI Blueprint for RAG is the reference workflow [NVIDIA-DOC]; 07-09 handles both properly.

Exam depth is calibrated to general-level knowledge — candidate reports agree that deep attention math and config-file detail did not appear [FIELD]. For this lesson that means: know what an embedding model does, know cosine similarity is the standard measure, know the re-embedding constraint, know dense-versus-sparse trade-offs. Do not memorise contrastive-loss formulations.

Question phrasings you should recognise:

Distractor families:

• "Dense retrieval requires no model." It requires a trained encoder — that is the definition. This distractor is sometimes dressed up as "unlike sparse retrieval, dense retrieval needs no training data."
• Dense retrieval conflated with the vector database. The encoder produces vectors; the vector database stores and searches them. Two components (07-04).
• Dense retrieval conflated with the LLM. The embedding model is not the generator. They are different models, usually different sizes, and often from different families.
• "Higher cosine similarity means the answer is correct." It means the passage is nearby in the space. Generation can still fail (07-10), and a nearby passage can still be wrong (see step 5 above).
• Euclidean distance offered as the standard text-similarity measure. Cosine is the convention, because direction rather than magnitude carries the semantics.
• "Embeddings can be compared across models of the same dimensionality." They cannot. Two 768-dimension spaces from two models are unrelated coordinate systems.
• "Dense retrieval solves hallucination." Grounding reduces it; it does not eliminate it (09-12, 07-11).

One further calibration point, and it must be carried as uncertainty rather than as fact: candidate reports converge on a heuristic that in scenario questions, when one option proposes a RAG solution, it is usually the keyed answer [FIELD]. This is field calibration, not official NVIDIA guidance. It is a tie-breaker for genuinely ambiguous items and nothing more; 07-12 covers the cases where the heuristic would lead you wrong.

Mistake 1 is the one this lesson is built around. It is worse than an outage because an outage is visible. A retrieval system with mismatched encoders passes every health check, serves every request, produces every metric, and answers every question wrongly.

Because nothing in the stack has any way to know. Walk the failure path:

The encoder cannot know. Its job is to map text to a vector of the declared dimensionality. Given any text, with or without a required prefix, it does that successfully. There is no "wrong input" for a function whose domain is all strings.

The vector database cannot know. It receives a query vector of the correct dimensionality and returns the nearest stored vectors by the configured metric. It performed correctly. A dimensionality mismatch would error — 768 versus 1024 raises an exception — which is exactly why the dangerous cases are the ones where dimensionality happens to agree.

The similarity score cannot tell you. Because cosine similarity over text embeddings occupies a compressed positive band, a broken system's scores look like a working system's scores. There is no natural zero to fall below. A 0.61 does not announce itself as meaningless.

The LLM cannot tell you. It receives passages and a question and does what it was trained to do: produce fluent text conditioned on that input. If the passages are irrelevant, a well-prompted model may decline (07-11), but its default behaviour is to synthesise something plausible. Fluency is not evidence of grounding, and 09-12 covers why.

Your metrics may not tell you either, if you only measure end-to-end answer quality. This is the strongest argument for the retrieval/generation decomposition in 07-10 and 09-07: measure whether the right chunk was retrieved, separately from whether the answer was good. A retrieval-recall metric drops immediately when encoders mismatch. An end-to-end LLM-judge score may not, because the judge sees a fluent answer.

The three defences, in order of value:

1. A self-retrieval assertion. Embed a known chunk's exact text as a query. It must return that chunk at rank 1 with similarity near 1.0. This single test catches nearly every mismatch class, costs one API call, and belongs in CI.
2. One place that owns the embedding convention. Model name, version, prefix strategy, pooling, normalisation — one module, used by both the indexer and the query path. Never two call sites that "should" agree.
3. Retrieval recall in the eval suite, not just answer quality (01-08, 03-04).

No, and the honest framing is that it never did. It replaces keyword search's weaknesses, and it introduces its own.

Dense retrieval solves vocabulary mismatch, paraphrase, and cross-lingual retrieval — problems BM25 cannot touch. It fails on exact identifiers, on newly coined vocabulary, on negation, on recency, and on authority, and the first two of those failures are structural consequences of subword tokenisation and fixed training data rather than tuning problems.

The economics reinforce the same conclusion. A sparse index is nearly free to maintain and cheap to update. A dense index costs an encoder pass per chunk at index time, and — the constraint everyone underestimates — a full re-embedding of the entire corpus every time the embedding model changes. For a large corpus that is a real project with a real bill, and it is why 03-03 treats embedding-model selection as a decision to make deliberately once rather than casually and often.

Which is why the mature answer is: keep both legs, fuse their results, and rerank the fused list. 07-06 and 07-07 are taught as a paired session precisely because either one alone leaves the other's failure mode in place.

The cost profile is asymmetric in a way that shapes architecture decisions, so it is worth naming the components even without quoting figures — the following are cost structures, not measured numbers.

Index time, paid once per corpus version:

• One encoder forward pass per chunk. Batched on GPU this is throughput-bound and embarrassingly parallel.
• Storage: dimensionality × 4 bytes per chunk for float32, plus the ANN index's own overhead. A 768-dimension float32 vector is 3,072 bytes before index structures; quantising to int8 or using a smaller-dimension model reduces this proportionally, at some recall cost.
• ANN index construction (07-04), which for graph indexes like HNSW is the expensive part.

Query time, paid per request:

• One encoder forward pass on a short query — the query is short, so this is latency-bound rather than throughput-bound, and it is usually the dominant term in retrieval latency.
• One ANN search, sublinear in corpus size by design.
• Optional reranking (07-07), which is a per-candidate cost and typically dwarfs both of the above.

On model change, paid in full:

• Re-encode every chunk. Rebuild every index. Run the eval set again to confirm the new model is actually better on your corpus rather than on a public benchmark. 12-12 treats this as the migration problem it is.

That last line is the one to remember for the exam, because it is the constraint most likely to appear in a scenario question. Switching embedding models is not a configuration change. It is a re-indexing project.

• Dense retrieval turns text into points and retrieval into nearest-neighbour search. Cosine similarity is the standard measure.
• It is the second stage of NVIDIA's stated RAG pipeline: query → embedding model → vector match against the indexed knowledge base → retrieve and decode → synthesise and cite [NVIDIA-DOC]. Know the ordering.
• Bi-encoders make it practical, because passage vectors are computed once offline and only the query is encoded per request.
• The worked example's headline result: the chunk sharing zero words with the query ranked first (0.81), and the lexically-matching but semantically-unrelated chunk ranked last (0.29). That inversion is the whole value proposition.
• Cosine similarities are ordinal, not calibrated. Unrelated text commonly scores in the 0.2–0.5 band. No universal threshold exists; calibrate per model and per corpus.
• Mismatched query and passage encoders produce plausible nonsense with high scores and no error. Defend with a self-retrieval assertion, one owner for the embedding convention, and retrieval recall in the eval suite.
• Vectors from different models are not comparable, even at equal dimensionality. Changing the embedding model is a full re-indexing project (12-12).
• Dense retrieval does not replace sparse retrieval. It fails on exact identifiers, unseen vocabulary, negation, recency, and authority — which is what the rest of this module addresses.

You now have two retrievers whose failures barely overlap, and a clear picture of what dense retrieval buys. What you do not yet have is a precise account of what a vector cannot see — and there are three specific blind spots that no amount of better embedding-model selection fixes, because they are properties of what similarity means rather than of how well it is computed. Next: 07-03 names them — negation, recency, and source authority — and shows why each one produces a confidently wrong answer rather than an empty result. Those three limits are what make hybrid search in 07-06 and reranking in 07-07 corrections rather than optimisations.