How to Test Retrieval Quality by Hand (Pre-Metric)

Manual retrieval testing is a small, fixed, hand-labelled set of query-to-passage expectations, run against your actual index, inspected result by result. It has four parts:

1. A query set. Twenty to fifty queries that resemble what real users will type — including the awkward ones, the acronym-only ones, and the ones your system probably cannot answer.
2. A relevance judgement per query. For each query, which chunk (or chunks) in your corpus genuinely answers it. You write this down before running anything, so you are not grading on what came back.
3. A retrieval run. Each query embedded and searched, top-k results captured — k = 5 is a sensible default because five is the number a person can actually read.
4. An inspection pass. For every query, did the right passage appear, at what position, and if not, what came instead and why.

The output is not a score. The output is a failure list with named causes, plus a rough hit rate you can compare across candidate models. The failure list is what you act on; the hit rate is what you use to choose.

Two things this is not. It is not a substitute for formal evaluation — twenty queries cannot support a claim like "model A is 3% better", and 09-09 explains exactly why that sample size cannot detect small differences. And it is not end-to-end RAG evaluation, because you are testing only the retrieval stage: no generation, no answer quality, no faithfulness. Isolating the stage is the point; 07-10 makes the case that asking "did retrieval fail or did generation fail?" first saves roughly half of all RAG debugging time, and you cannot ask it unless you can test retrieval alone.

L1 — The procedure

1. Write 20 queries.                      (30 minutes)
2. For each, find and record the passage that should answer it.   (60 minutes)
3. Embed and index the corpus with candidate model A.
4. Run all 20 queries, capture top-5 results with scores.
5. Score each query: hit at 1 / hit in top 5 / miss.
6. Read every miss and every near-miss. Name the cause.
7. Repeat 3–6 for candidate model B.
8. Compare hit counts and — more importantly — compare failure lists.

Steps 2 and 6 are the ones people skip, and they are the ones that carry the value. Step 2 skipped means you are grading the output against your impression of it, which is not a test. Step 6 skipped means you have a number and no diagnosis.

L2 — Building the query set so it is not useless

A query set assembled carelessly will make every model look fine, because the easy queries are easy for everyone. Deliberately include all of these categories:

That last-but-one row matters more than it looks. A retrieval system with no unanswerable queries in its test set has never been checked for the failure mode where it confidently returns the nearest thing regardless — and "nearest thing" is always defined, because cosine similarity always returns a number.

L3 — Reading the near-misses, which is where the diagnosis lives

A miss where the correct passage was ranked 6th is a completely different problem from a miss where it was ranked 4,000th, and the top-5 print-out hides that. Capture the correct passage's rank and score even when it is outside the top-k. Then classify:

Every row of that table is a different action. That is the argument for reading failures rather than only counting them, and it is why this lesson exists before the metrics lesson rather than after it.

The two baselines you must run

Two comparisons cost almost nothing and prevent the most common self-deceptions:

• A keyword baseline. Run the same twenty queries through simple keyword search or BM25 (07-01). If the embedding model does not beat it on your query set, you have learned something important and cheap. This is not a rhetorical exercise — sparse retrieval is genuinely strong on many corpora, and a team that never ran it cannot claim the vector store helped.
• A random baseline, conceptually. With 60,000 chunks and k = 5, random retrieval hits essentially never. Knowing that means a hit rate of 0.4 is not "bad" in the abstract — it is enormously better than chance and possibly good enough, depending on what happens downstream.

The relationship is sequential rather than competitive. The manual set you build here becomes the seed of the formal set: 01-08 had you build 20 hand-written triples for the LLM as a whole, 09-01 scales an eval set to a hundred items, and 10-04 puts it in CI so a build fails when quality drops. The same twenty queries you write this afternoon are the first twenty rows of that artifact. Nothing here is throwaway.

Constructed scenario. An internal support knowledge base, 60,000 chunks, two candidate embedding models. All numbers below are invented for the illustration; the arithmetic on them is exact. Cosine scores in particular are model-dependent and must not be read as typical.

Step 1 — the tally sheet

Twenty queries, top-5 retrieved, correct passage's rank recorded even when outside the top 5. Model A first.

Step 2 — the arithmetic

Query 20 is unanswerable, so it is scored separately, leaving 19 answerable queries.

Hit@1 count = queries 1, 2, 5, 10           = 4
Hit@1 rate  = 4 / 19 = 0.211  → 21%

Hit@5 count = 1,2,3,4,5,6,8,10,17,19        = 10
Hit@5 rate  = 10 / 19 = 0.526  → 53%

Correct passage in top 20 (recoverable by reranking):
  add 7 (11), 9 (7), 11 (18), 18 (9)        = 14
Top-20 rate = 14 / 19 = 0.737  → 74%

Never recoverable in top 20: 12 (340), 13 (2904), 14 (1177), 15 (61), 16 (88) = 5

And the unanswerable query, scored on its own terms:

Query 20 top-1 cosine score: 0.71
Median top-1 score across the 19 answerable queries: 0.78
Gap: 0.07  → NOT separable by a fixed threshold

Step 3 — read it

The headline numbers say 53% hit@5. Taken alone that is a shrug. The breakdown says something actionable and much more specific:

• Paraphrase queries: 5/5 in top 5, 3 at rank 1. The embedding model is doing its core job. Nothing to fix here.
• All four exact-code and acronym queries failed catastrophically — ranks 340, 2904, 1177, and 18. This is not a model-quality problem and a better embedding model will not fix it; it is the structural limit from 03-01. The fix is a keyword or exact-match path, i.e. hybrid search (07-06).
• Both negation queries failed at ranks 61 and 88 — again structural (07-03), again not a model-selection issue.
• Four failures sat at ranks 7–18. Those are ordering failures, not recall failures: the right passage was in the neighbourhood and got out-ranked. That is precisely the case a cross-encoder reranker fixes (07-07), and it is worth 4 more hits — pushing the effective ceiling from 53% to 74% without touching the embedding model at all.
• The unanswerable query scored 0.71 against a median of 0.78. A 0.07 gap means no threshold can separate "we have an answer" from "we do not" on this evidence. That is a safety finding, and it is more consequential than the hit rate: it says the system will confidently return an irrelevant passage rather than abstain (07-11).

Now compare model B on the identical twenty queries.

Model B:
  Hit@1  = 5 / 19 = 0.263  → 26%
  Hit@5  = 11 / 19 = 0.579 → 58%
  Top-20 = 14 / 19 = 0.737 → 74%
  Exact-code and acronym queries: still 0/4
  Negation queries: still 0/2
  Unanswerable gap: 0.05 (worse)

Model B is one hit better at k = 5 — eleven versus ten. That difference is not evidence of anything. One query out of nineteen is within the noise a twenty-query set can produce; 09-09 shows the arithmetic on why a sample this small cannot resolve differences this small. What is evidence is that both models fail the same four categories identically, which tells you the binding constraint is architectural, not the model choice. Spending another week bake-off-ing embedding models would be time taken from the hybrid-search work that would actually move the number.

That inference — the failure categories matter more than the hit count — is the whole reason to do this by hand.

Step 4 — the keyword baseline, which changes the plan

Same twenty queries, BM25 keyword search:

BM25:
  Hit@5 = 8 / 19 = 0.421 → 42%
  Exact-code queries:      2/2 at rank 1   ← dense scored 0/2
  Acronym query:           1/1 at rank 1   ← dense scored 0/1
  Paraphrase queries:      1/5             ← dense scored 5/5

The two methods fail in opposite directions, exactly as 03-01 predicted: dense wins paraphrase and loses exact strings; sparse wins exact strings and loses paraphrase. Union of the two top-5 lists in this constructed run covers 14 of 19. That is the argument for hybrid search, arrived at from your own data in one afternoon rather than taken on authority, and it is the strongest possible answer to "should we add BM25?".

While you are inspecting text by hand, there are two text-level measurements worth knowing, both computable with a pencil, both explicitly reported as exam items, and both frequently confused with each other. They measure genuinely different things.

Lexical diversity measures vocabulary variety — how many different words a text uses relative to how many words it contains. The simplest form is the type–token ratio (TTR): the number of distinct word types divided by the total number of tokens. High lexical diversity means a wide vocabulary with little repetition; low means a narrow, repetitive vocabulary.

Syntactic complexity measures sentence structure — how elaborately clauses are built and nested. It is estimated with quantities like mean sentence length in words, clauses per sentence, subordinate-clause count, and the depth of a dependency parse tree. High syntactic complexity means long sentences with embedded subordinate structure; low means short, flat sentences.

The two are independent. A text can be lexically rich and syntactically simple ("Otters swim. Herons wade. Kingfishers dive.") or lexically poor and syntactically complex ("The thing that the thing that we mentioned referred to was the thing we meant."). Confusing them is the reported error, and the fix is to remember which noun each name modifies: lexical → words; syntactic → structure.

Worked arithmetic on two constructed texts

Text 1 — "The otter swims. The heron wades. The kingfisher dives."

Tokens (words, lowercased, punctuation dropped):
  the, otter, swims, the, heron, wades, the, kingfisher, dives   = 9 tokens
Distinct types:
  the, otter, swims, heron, wades, kingfisher, dives             = 7 types
TTR = 7 / 9 = 0.778

Sentences = 3
Mean sentence length = 9 / 3 = 3.0 words
Clauses = 3  →  clauses per sentence = 1.0

Text 2 — "The animal that the observer who arrived early had noticed was the animal that the report, which was filed late, had described."

Tokens: the, animal, that, the, observer, who, arrived, early, had, noticed,
        was, the, animal, that, the, report, which, was, filed, late,
        had, described                                            = 22 tokens
Distinct types: the, animal, that, observer, who, arrived, early, had,
        noticed, was, report, which, filed, late, described       = 15 types
TTR = 15 / 22 = 0.682

Sentences = 1
Mean sentence length = 22 / 1 = 22.0 words
Clauses ≈ 5 (main + 4 subordinate/relative) → clauses per sentence = 5.0

Side by side:

The metrics move in opposite directions on the same pair of texts, which is the cleanest possible demonstration that they are not the same measurement. Text 1 is lexically more diverse and syntactically trivial; Text 2 is syntactically far more complex and lexically more repetitive — the word the alone appears five times.

One caveat to carry: raw TTR is length-sensitive. Every text eventually reuses common function words, so a 10,000-word document will have a lower TTR than a 100-word excerpt from it even with identical vocabulary richness. Comparing TTR across texts of different lengths is therefore invalid, which is exactly why length-corrected variants (root TTR, moving-average TTR, MTLD) exist. If you need to compare, compare on equal-length samples.

Why these two metrics belong in a retrieval lesson

Practically, they characterise the text you are about to embed and search, and both extremes cause retrieval trouble:

• Very low lexical diversity across your corpus means chunks look alike in vocabulary, so their embeddings crowd together and retrieval cannot discriminate. This is exactly what boilerplate does (06-04).
• Very high syntactic complexity means long sentences with nested clauses, which makes naive sentence-boundary chunking produce awkward fragments and pushes chunk token counts toward the model's limit (03-03).
• A large gap in these metrics between your queries and your corpus — short, simple, low-diversity queries against long, complex, high-diversity passages — is a concrete description of the asymmetric retrieval situation from 03-03, and a reason to prefer an asymmetrically trained model.
• Lexical diversity is also a cheap generated-text diagnostic: output that is degenerating into repetition shows a collapsing TTR, which relates to the decoding-parameter effects in 04-05.

For the exam, the requirement is identification depth: know that lexical diversity is about vocabulary variety measured by type–token ratio, that syntactic complexity is about sentence and clause structure measured by things like sentence length and parse depth, and that they are distinct metrics measuring distinct properties. You will not be asked to compute MTLD.

Three objectives converge here. 1.8 covers selecting and using embedding models, and by-hand testing is how a selection is justified. 1.4, "curate and embed content datasets for RAGs", covers the corpus side, and manual inspection is where you find out that your corpus is full of boilerplate. 1.6 brings in the Python NLP packages — spaCy for tokenization and parsing, NumPy for the vector arithmetic — that make both this test and the text metrics above computable.

The exam's calibration is also directly relevant. Candidate reports describe questions as general-level, favouring "know at a high level what each thing is and when to use it". That maps onto this lesson as: know that retrieval is evaluated separately from generation, know roughly what recall@k means, know that a keyword baseline is the honest comparison, and know that lexical diversity and syntactic complexity are two different things.

Question phrasings to expect

• Stage isolation. "A RAG system produces a wrong answer. What should you check first?" Whether the correct passage was retrieved at all — separating retrieval failure from generation failure (07-10).
• Baseline discipline. "How would you determine whether adding a vector database improved search quality?" Compare it against a keyword/BM25 baseline on the same query set.
• Metric identification. "Which metric expresses how often the relevant document appears in the top k results?" Recall@k. Precision, MRR, and nDCG are the neighbouring options; know what each emphasises (09-07, 09-05).
• Eval-set construction. "What should an evaluation set for retrieval contain?" Queries paired with the passages judged relevant to them — written before results are seen.
• Lexical diversity vs syntactic complexity. "Which metric measures vocabulary variety in a text?" Lexical diversity, typically type–token ratio. "Which measures sentence structure elaboration?" Syntactic complexity. Expect these to appear as a matched pair of distractors for each other — this is a specifically reported exam item.
• Type–token ratio. "What does a type–token ratio of 0.4 indicate?" That distinct words are 40% of total words — moderate repetition. A follow-up trap: whether TTR is comparable across texts of different length (it is not).
• Human evaluation. "Why inspect retrieval results manually rather than only tracking a metric?" Because the metric aggregates causes away, and different failure categories need different fixes.
• Sample size. "Model B scored one hit better on a 20-query test set. Is it better?" No — the sample cannot support the claim (09-09).

Distractor families

Twenty is the right number for the job this lesson is doing, and it is right for a specific reason: twenty is small enough that you will genuinely read all of the failures, and reading the failures is the deliverable. Fifty is better if the categories in §2 need more coverage; a hundred stops being a by-hand exercise.

What twenty cannot do is support a fine-grained comparison. With nineteen answerable queries, one hit is 5.3 percentage points, so any difference smaller than a couple of hits is indistinguishable from noise. That is a real limit, not a caveat to wave away — 09-09 works through the standard-error arithmetic that makes it precise. The practical rule: use twenty queries to detect category failures and gross differences, and scale the set (09-01) before you make claims about percentages.

The other thing twenty queries do is make the categories visible. Four consecutive failures on exact-identifier queries is an unmistakable pattern in a set of twenty, and it would be an unremarkable 4% dip in a set of five hundred summarised into a single number. Small sets are better at diagnosis and worse at measurement, which is exactly why this lesson comes before the metrics module rather than after it.

There is no portable answer, and treating one as portable is a common and expensive mistake. Score ranges depend on the model, its training objective, and its normalisation, so one model's "clearly relevant" band may begin around 0.6 while another's begins around 0.85, and a threshold copied from a tutorial written about a different model is a guess.

What you can do is derive a threshold for your own model from the by-hand run, which is one of its most useful outputs. Take the top-1 score of every query you judged answerable, and the top-1 score of every unanswerable query. If the two distributions separate cleanly, the gap between them is your candidate threshold. If they overlap — as they did in §4's constructed example, 0.71 against a median of 0.78 — then no threshold exists that will work, and the honest response is to say so and handle abstention some other way: a reranker score instead of a retrieval score, an explicit grounding check, or letting the model say it does not know (07-11).

Also treat any threshold as invalidated by a model change. If you upgrade the embedding model, the threshold must be re-derived, because the whole score distribution has moved. This is one more reason 12-12 treats an embedding-model change as a migration.

Yes, always, and this is one of the highest-leverage habits in the whole course. A RAG system has stages, and a bad answer can originate in any of them: parsing dropped a table, chunking split a sentence, the embedding model missed the paraphrase, the index returned the wrong neighbours, the context assembly buried the passage in the middle, or the generator ignored what it was given.

If you only look at the final answer, all six causes present identically as "the answer was wrong". If you check retrieval first — was the correct passage in the top-k at all? — you immediately partition the space in two. Passage absent means the problem is upstream, in parsing, chunking, embedding, or indexing. Passage present but the answer is still wrong means the problem is downstream, in context assembly or generation. 07-10 argues this single question saves roughly half of all RAG debugging time, and the by-hand test in this lesson is the instrument that answers it.

There is a related discipline worth adopting now: when you change something, re-run this set. It is a twenty-query regression test. 05-04 puts prompt versions under the same treatment, and 10-04 turns the whole thing into a CI gate that fails a build. Everything in that chain starts with the twenty queries you write today.

• Twenty queries, judged before you look at results, top-5 inspected one at a time. That is the whole procedure, and it decides between candidate embedding models better than any leaderboard because it runs on your corpus.
• Record the correct passage's rank even when it is outside the top-k. Rank 6 means add a reranker; rank 4,000 means something structural is wrong. The top-5 print-out alone cannot tell them apart.
• Read the failures; do not just count them. Exact-identifier misses, negation misses, ordering misses, and dilution misses each require a different fix, and a single hit-rate number erases the distinction.
• Include the hard categories deliberately — exact terms, acronyms, negation, multi-hop, very short queries, very long queries, and unanswerable ones. A query set of easy paraphrases makes every model look adequate.
• Run a keyword baseline on the same queries. Dense and sparse retrieval fail in opposite directions; measuring both on your data is how the hybrid-search decision gets made on evidence.
• A one-hit difference on twenty queries is noise. Use the set for category diagnosis and gross comparison; scale it before making percentage claims.
• Derive any score threshold from your own answerable-versus-unanswerable score gap — and if those distributions overlap, accept that no threshold works and handle abstention another way.
• Lexical diversity is vocabulary variety (type–token ratio); syntactic complexity is sentence structure (sentence length, clauses per sentence, parse depth). They are independent, they can move in opposite directions on the same texts, and they are a reported exam confusable.
• Raw TTR falls as texts get longer, so it is only comparable across equal-length samples.
• Test retrieval separately from generation. "Was the correct passage in the top-k?" partitions every RAG failure into upstream and downstream halves in one question.
• Keep the query set. It is the seed of the hundred-item eval set in 09-01 and of the CI gate in 10-04.

You can now assess an embedding model against your own corpus and name the failure that is hurting you. One thing remains, and it is the piece of embedding folklore most likely to give you wrong instincts while you are doing exactly that diagnostic work: the claim that king − man + woman ≈ queen, and the belief that embedding spaces are neatly arranged into meaningful, composable directions you can reason with arithmetically.

That result is real, it is narrower than it sounds, and taking it at face value produces bad debugging intuitions — you start expecting the geometry to be tidier and more semantic than it is, and then you are surprised when not overdue sits right next to overdue. Next: 03-05 closes the module by taking the analogy result apart: what word2vec's vector arithmetic actually demonstrates, the conditions and exclusions that make the famous examples work, why it does not generalise to sentence embeddings, and what the honest version of "directions in embedding space" is. After that, 04-01 opens the transformer itself and shows where those contextual vectors come from — and why context length costs quadratically.