How to Choose an Embedding Model: Selection Criteria

Selecting an embedding model is committing to one function that will convert both your corpus and every future query into vectors, and committing to it for as long as the index lives. The commitment has four dimensions that constrain each other:

1. Maximum sequence length — the longest input, measured in the model's own tokens, that the model will process. Beyond it, input is silently truncated.
2. Output dimensionality — how many numbers each vector has. This multiplies directly into storage, index memory, and per-query compute.
3. Training objective and intended use — whether the model was trained for symmetric similarity or asymmetric search, and whether it expects prefixes or instructions.
4. Domain and language coverage — whether the model's training data resembles your corpus enough for its geometry to be meaningful on your text.

Underneath those sit the practical constraints: where the model can run (your GPU, a managed endpoint, an on-prem microservice), what it costs per million tokens embedded or per hour hosted, whether the licence permits your use, and whether the version is pinned so it cannot change under you.

What selection is not: picking the highest-ranked model on a public leaderboard. A benchmark ranking is a measurement on someone else's data, and the gap between models near the top of a leaderboard is frequently smaller than the gap between a leaderboard corpus and yours. Leaderboards are useful for building a shortlist of three or four candidates. Your own corpus decides among them, which is why 03-04 exists.

L1 — The intuition: fit the model to the text you actually have

Every one of the four criteria is a fit question, and each has a characteristic failure that produces no error message:

• Sequence length too short → the ends of your chunks are invisible to search.
• Dimension too large → your index does not fit in memory, or queries get slow, or both.
• Objective mismatched → short queries retrieve badly against long passages.
• Domain mismatched → your specialist vocabulary lands in undifferentiated regions of the space.

Notice the pattern: all four fail quietly. Nothing throws. This is why selection has to be deliberate rather than discovered later.

L2 — Criterion by criterion

Maximum sequence length

An embedding model has a hard input ceiling in tokens. Feed it more and the standard behaviour is truncation — the extra tokens are dropped and you get a vector for the beginning of your text with no warning. A chunk whose second half is silently discarded is a chunk whose second half cannot be retrieved, ever, and nothing in your logs will say so.

The check is arithmetic, and you have the tools from 02-03:

1. Decide your chunking strategy first (or at least a candidate).
2. Measure the token length distribution of your chunks with the model's own tokenizer — not characters, not words, and not another model's tokenizer, because vocabularies differ.
3. Look at the tail, not the mean. If your 95th-percentile chunk is 400 tokens and the model's limit is 512, you are fine. If the tail runs to 900, a meaningful slice of your corpus is being cut in half.
4. Leave headroom for anything the pipeline prepends — a required query prefix, an instruction string, a title concatenated onto the chunk. Those consume the same budget.

A common misreading: a longer maximum length is not automatically better. A model that accepts 8,192 tokens still compresses whatever you give it into one fixed-width vector, so feeding it 8,000 tokens produces the diluted centroid 03-02 warned about. Long-context embedding models are useful because they permit larger chunks when your documents genuinely need them, not because large chunks are good.

Output dimensionality against index memory

Dimensionality is the criterion with the cleanest arithmetic, and the one most often skipped until an index will not fit. Two rules:

• Storage scales linearly in dimension. A vector of d float32 values takes 4·d bytes. Doubling d doubles the raw vector storage for the whole corpus.
• Query compute scales linearly in dimension too. A dot product over d dimensions costs d multiply-adds; ANN indexes reduce how many comparisons you do, not the cost of each one.

Higher dimension often buys some quality, with diminishing returns, and it always costs memory and latency. It also interacts with the index: HNSW graphs carry per-vector link overhead on top of the vectors themselves, so real index memory exceeds the raw vector arithmetic — often substantially. §4 does the numbers.

Two mitigations worth knowing by name. Quantization of the vectors stores each component in fewer bits (float16 halves it; int8 quarters it) at some recall cost — the same accuracy-versus-footprint trade-off 12-01 and 12-02 make for model weights. And some newer models are trained so their dimensions are ordered by importance, letting you truncate a vector to a shorter prefix and keep most of the quality; where a model documents that property you can treat dimension as a tunable rather than a fixed cost. Do not assume it — a model that was not trained that way loses badly under truncation.

Symmetric vs asymmetric training objective

This is the criterion most often missed and it has the largest quality effect for retrieval.

• A symmetric model was trained on pairs where both sides are the same kind of text: two paraphrases, two duplicate questions, a sentence and its translation. It expects like-for-like comparison.
• An asymmetric model was trained on pairs where the two sides differ in kind and length: a short question against the long passage that answers it. It expects search.

If your users type five-word questions and your corpus is 200-token paragraphs, that is asymmetric, and an asymmetric-trained retrieval model is what you want. If you are deduplicating documents or clustering tickets, both sides are the same kind of text and a symmetric model fits. Getting it backwards produces a system that works — just worse than the one you could have had, with nothing indicating why.

Bound up with this is the prefix or instruction convention. Many retrieval models are trained with a marker distinguishing the two roles, such as a query prefix and a passage prefix, or a short instruction sentence prepended to queries. If the model card specifies one, using it is mandatory, applying it to the wrong side is a bug, and forgetting it entirely is a silent quality loss. Read the model card; this information is not inferable from the model's name.

Domain and language coverage

An embedding space is only as good as the co-occurrence statistics behind it (03-01). If your corpus is full of terms that were rare or absent in the model's training data — drug names, part numbers, internal project codenames, ICD codes, a specialist legal register — the model has no well-formed geometry for them. Domain terms get shredded into odd subword sequences by the tokenizer and land in poorly differentiated regions.

Checks, cheapest first:

1. Tokenize a sample of your domain terms. If a term you care about becomes eight subword fragments, the model has never seen it as a unit. That is a warning sign, not a verdict, but it is free to check.
2. Language coverage. A monolingual English model on a multilingual corpus is a straightforward mistake; if you need cross-lingual retrieval — query in one language, passages in another — you need a model explicitly trained for it, because that is a specific training property and not a bonus.
3. Domain-specialised models exist for biomedical, legal, code, and financial text and can beat a bigger general model on their own turf. They may also be worse on ordinary prose, so if your corpus is mixed, test both.
4. Nearest-neighbour spot checks on your own text are the real evidence. Take twenty domain terms, embed them, look at each one's nearest neighbours, and ask whether a domain expert would accept the grouping. That is a 03-04 exercise and it is the check that actually decides.

L3 — Constraints that decide it in practice

Beyond the four, these often make the decision before quality does:

On the NVIDIA stack specifically: embedding models are commonly served as NIM microservices — pre-optimised inference containers with stable APIs — and NeMo Retriever is NVIDIA's offering aimed at retrieval accuracy at scale, with the AI Blueprint for RAG as the reference workflow that wires the pieces together. 12-13 covers the deployment mechanics and 07-09 the pipeline. For selection purposes, the relevant point is that hosting choice and model choice are separable decisions: the same four criteria apply regardless of whose runtime serves the model.

Where a leaderboard fits

Real arithmetic, with every input stated as an assumption. The corpus figures are a constructed scenario; the memory arithmetic from them is exact.

Scenario. An internal support knowledge base:

Documents:                     12,000
Average document length:       ~1,400 tokens
Chunk size target:             ~350 tokens with ~50 tokens overlap
Chunks per document:          1,400 / 300 effective ≈ 4.7 → call it 5
Total chunks:                 12,000 × 5 = 60,000
Total tokens to embed once:   12,000 × 1,400 ≈ 16,800,000 (16.8 M)

Step 1 — raw vector storage at two dimensions

float32 = 4 bytes per component.

d = 384:
  bytes per vector = 384 × 4 = 1,536 B
  60,000 vectors   = 60,000 × 1,536 = 92,160,000 B
                   = 92.16 MB (or 87.9 MiB)

d = 1,536:
  bytes per vector = 1,536 × 4 = 6,144 B
  60,000 vectors   = 60,000 × 6,144 = 368,640,000 B
                   = 368.64 MB (or 351.6 MiB)

A 4× dimension increase is a 4× storage increase, exactly. At 60,000 chunks both fit comfortably in memory on any ordinary machine — which is the honest conclusion for a corpus this size, and the reason 07-04 argues that small corpora do not need a vector database at all.

Step 2 — the same arithmetic at 20 million chunks

d = 384:   20,000,000 × 1,536 B = 30,720,000,000 B ≈ 30.7 GB
d = 1,536: 20,000,000 × 6,144 B = 122,880,000,000 B ≈ 122.9 GB

Now the choice is architectural. 30.7 GB is a large single machine; 122.9 GB means sharding, or moving vectors to disk-backed storage, or quantizing. The dimension you picked in week one determined the topology of your infrastructure in year two.

Step 3 — index overhead is not optional

Raw vectors are a floor, not the total. An HNSW graph stores neighbour links per vector; a rough sizing rule is links_per_vector × 4 bytes for int32 ids, where links_per_vector is on the order of 2·M for the parameter M used at build time. Taking M = 16 as a common default, so ~32 links:

Link overhead per vector ≈ 32 × 4 B = 128 B
At 20 M vectors           ≈ 2.56 GB

d = 384:   30.7 GB + 2.6 GB ≈ 33.3 GB
d = 1,536: 122.9 GB + 2.6 GB ≈ 125.5 GB

Note the asymmetry: graph overhead is independent of d, so at low dimension it is a visible fraction of the total and at high dimension it disappears into the noise. This is a rough sizing rule, not a vendor specification — real overhead depends on the index implementation, the M and efConstruction settings, and whether payload metadata is stored alongside. Check your index's own documentation before you size a machine on it. The point that survives the imprecision: the vectors are not the only thing in memory.

Step 4 — quantization as the lever

d = 1,536, 20 M vectors:
  float32 (4 B/component): 122.9 GB
  float16 (2 B/component):  61.4 GB
  int8    (1 B/component):  30.7 GB

int8 gets a 1,536-dimensional index down to the footprint of a float32 384-dimensional one. It costs some recall, and how much is an empirical question for your data — which is again a 03-04 measurement and not something to accept on faith.

Step 5 — the ingest bill, and the cost of changing your mind

Tokens to embed once: 16.8 M

At an assumed $0.02 per 1 M tokens:  16.8 × 0.02 = $0.34
At an assumed $0.10 per 1 M tokens:  16.8 × 0.10 = $1.68
At an assumed $0.13 per 1 M tokens:  16.8 × 0.13 = $2.18

Those per-token prices are illustrative placeholders, not quoted vendor pricing — real prices change and vary by provider, and you must look up the current figure for the model you are considering. The structural lesson is what matters: for a 16.8 M-token corpus the one-off ingest cost is trivial, and it stays trivial through several re-embeddings. Scale the corpus to 10 billion tokens and the same arithmetic gives 200 to 1,300 per full pass, at which point "we will just re-embed if we change our minds" stops being free. The cost that bites at scale is not usually the money anyway — it is the wall-clock time and the operational choreography of rebuilding a live index, which is 12-12's subject.

Step 6 — the truncation check, which costs nothing and prevents the worst failure

Model max sequence length:            512 tokens
Required query prefix:                  ~4 tokens
Chunk title prepended at ingest:       ~12 tokens
Effective budget for chunk body:  512 − 12 = 500 tokens

Measured chunk token lengths (constructed distribution):
  median          310
  90th percentile 430
  95th percentile 505   ← at the limit
  99th percentile 690   ← 190 tokens silently dropped
  max             880   ← 380 tokens silently dropped

Roughly 1% of chunks lose material, and the loss is concentrated in the longest chunks, which are disproportionately likely to be the dense reference material people search for. Two fixes: cap chunk size below the effective budget during chunking, or choose a model with a longer limit. Either is fine; discovering the problem six months later from a user complaint is not. Measure the distribution with the model's own tokenizer before you index anything.

The selection procedure, in order

1. Sample your corpus and your queries. Twenty to fifty real queries and the passages that should answer them. If you do not have real queries, write the ones you expect and mark them as assumptions.
2. Fix a candidate chunking strategy and measure the chunk token-length distribution — median, 95th, 99th, max.
3. Filter the field by hard constraints first: hosting and privacy, licence, max sequence length against your 99th percentile, and the memory budget implied by dimension at your chunk count. This usually leaves few candidates.
4. Filter by objective: asymmetric for search, symmetric for like-for-like. Note each candidate's required prefixes and pooling.
5. Shortlist 2–4 using a public benchmark for discovery only.
6. Test on your own data with the by-hand procedure in 03-04: run your queries, look at the top-5, count how often the right passage is there, and read the near-misses.
7. Check cost and latency for both ingest and query at your real volume.
8. Pin the model id and version in configuration, and record the pooling and prefix conventions next to it.
9. Write down the four-sentence defence of your choice — model, why the length fits, why the dimension fits, why the objective fits. That is the week-2 deliverable of this course, and it is also what your team will ask for.

Objective 1.8 is unusually explicit: "Select and use models to create text embeddings." Of all thirty-one official sub-objectives, this is one of the few whose verb is select — the exam is telling you that choosing is a tested skill, not just using. Objective 1.4, "Curate and embed content datasets for RAGs", adds the corpus side, and objective 1.6 brings vector databases in, where dimensionality and index memory live.

This also matches the exam's calibration. Candidate reports describe the questions as general-level: know at a high level what each thing is and when to use it. That description fits selection criteria almost perfectly — the exam wants to know whether you can reason about which model suits a described situation, not whether you can recite a model's architecture.

Question phrasings to expect

• Scenario selection. "A team is building RAG over 500-token chunks with an embedding model limited to 256 tokens. What is the consequence?" Silent truncation; half of each chunk is unretrievable. This is the most likely single question in this area.
• Dimensionality trade-off. "What is the effect of choosing a higher-dimensional embedding model?" Higher storage and per-query compute, potentially better quality with diminishing returns. Distractors will claim it increases the maximum input length, or that it improves quality without cost.
• Symmetric vs asymmetric. "Users submit short keyword-like questions against long documentation pages. Which kind of embedding model?" One trained for asymmetric retrieval.
• Changing models. "A team wants to switch embedding models on an existing index. What must they do?" Re-embed the entire corpus and rebuild the index. Distractors: "embed only new documents", "convert the existing vectors", "just change the API endpoint". All wrong, and the second is impossible.
• Domain fit. "A general-purpose model performs poorly on clinical notes. What is the most likely reason and remedy?" Domain vocabulary was underrepresented; evaluate a domain-adapted model, or fine-tune, or add hybrid keyword search.
• Versioning discipline. "Why pin the embedding model version?" Because query and corpus vectors must come from the same space; an upgraded model invalidates the index.
• Language. "The corpus is in five languages. Which model?" One explicitly trained multilingually.
• Leaderboards. "Is the top-ranked model on a public benchmark necessarily the best choice?" No — the benchmark is not your corpus, and constraints such as max length, licence, hosting, and cost are not in the ranking.

Distractor families

No. Dimensionality is weakly and non-monotonically related to retrieval quality, and it is strongly and exactly related to cost. More dimensions give the model more room to encode distinctions, and up to a point that helps — but the returns diminish, and a well-trained smaller-dimensional model routinely beats a poorly-fitted larger one on the same corpus. Meanwhile the costs are precise: storage is 4·d bytes per vector, per-query comparison cost is proportional to d, and both scale by your chunk count.

Where dimension is not the binding constraint — a small corpus, generous memory — pick on fit and ignore it. Where you are indexing tens of millions of chunks, dimension is an architectural decision and belongs in the arithmetic before the bake-off. And note the two independent levers if you find yourself wanting quality and a small footprint: vector quantization, and models explicitly trained to tolerate dimension truncation.

It gets truncated, quietly. The model processes the first N tokens up to its limit and discards the rest; you receive a normal-looking vector with no indication that anything was dropped. The consequence is precise and permanent: content past the cut-off never influences the vector, so it can never be retrieved through that vector, no matter how well it matches a future query.

Three responses, in order of preference. Chunk under the limit — the standard answer, and it is what you should be doing anyway for the dilution reasons in 03-02. Choose a model with a longer limit if your documents contain genuinely indivisible units that exceed it. Aggregate multiple chunk vectors into a document-level representation only when you specifically need document-level retrieval, and be aware that averaging chunk vectors reintroduces dilution.

One trap: the limit is in the model's own tokens, and tokenizers differ. A 400-token chunk under one model's tokenizer may be 430 under another's, especially for text heavy in code, numbers, or non-English script. Re-measure when you change candidates.

Usually not first, and the ordering of alternatives matters more than the answer.

Try these before fine-tuning: pick a model whose training objective matches your query shape; apply the model's prefix convention correctly; fix your chunking; add hybrid keyword retrieval to catch the exact-term cases embeddings structurally miss (07-06); add a cross-encoder reranker over the top-k, which frequently recovers more quality than a better first-stage model would (07-07). Each is cheaper than fine-tuning and each addresses a distinct failure mode.

Fine-tuning an embedding model becomes reasonable when you have a genuinely idiosyncratic domain vocabulary and enough labelled query–passage pairs to train on — and the pair requirement is the real gate, because contrastive training needs positives and preferably hard negatives, which means labelling work. It also creates a maintenance obligation: your model is now a versioned artifact you must re-train and re-validate, and every change means re-embedding the corpus. The evaluation discipline for deciding whether it helped is Module 9's subject, particularly 09-05 and 09-07. Do not fine-tune to fix a problem you have not first measured.

On evidence, and knowing the price. The price is a full re-embedding of the corpus and an index rebuild, so the trigger should be more than a new model appearing on a leaderboard.

Legitimate triggers: your eval set shows the current model missing a class of query you care about, and a candidate does better on that same eval set; your corpus has changed character enough that domain fit no longer holds; your chunk-length distribution has drifted past the current model's limit; cost or latency has become a binding constraint that a different dimension would relieve; or the provider is deprecating the version you pinned.

The mechanics, when you do change: embed into a new index rather than mutating the live one, evaluate both on the same eval set, then swap atomically. Never let two models' vectors coexist in one index — that is not a degraded system, it is two incompatible spaces sharing a table, and it will return confident nonsense. 12-12 treats this as the index-lifecycle problem it is.

• Four criteria, in order: max sequence length, dimensionality, training objective, domain coverage. All four fail silently, which is why they must be checked deliberately rather than discovered.
• Measure your chunk token lengths with the target model's own tokenizer, look at the 95th and 99th percentiles, and leave room for prefixes and prepended titles. Text past the limit is truncated with no error and becomes permanently unretrievable.
• Dimensionality is exact arithmetic: 4·d bytes per float32 vector, times your chunk count, plus index overhead. At 20 M chunks, d = 384 is ~30.7 GB and d = 1,536 is ~122.9 GB. That decision shapes your infrastructure.
• A longer max length is not a licence to use longer chunks. The vector width is fixed regardless of input length, so long spans still dilute.
• Match the objective to your query shape. Short query against long passage is asymmetric; like-for-like is symmetric. Getting it backwards costs quality with no error.
• Apply the model's prefix and pooling conventions exactly as documented, on both the ingest and the query path, using the same code.
• Check domain fit cheaply: tokenize your specialist terms, and inspect nearest neighbours on your own text.
• Leaderboards shortlist; your own data decides. A benchmark measures someone else's corpus and knows nothing about your licence, hosting, latency, or length constraints.
• Pin the version. An unpinned hosted model can change beneath a live index and put your query vectors in a different space than your corpus vectors.
• Changing the model means re-embedding everything. Build a new index, evaluate both, swap atomically, and never mix two models' vectors.
• Try objective fit, prefixes, chunking, hybrid search, and reranking before fine-tuning. Fine-tuning needs labelled pairs and creates a permanent maintenance obligation.

Every criterion in this lesson ends at the same place: test it on your own data. Two candidate models both fit your length budget and your memory budget — now which one actually retrieves your passages for your queries? You cannot answer that from a model card, and you do not yet have any of the retrieval metrics that would answer it formally. Recall@k, MRR, and nDCG are four modules away in 09-07, and reaching for them now would mean computing a number before you can read one.

Next: 03-04 does it the pre-metric way, on purpose — twenty hand-written queries, the top-5 results printed out, a count of how often the right passage appeared, and a careful look at the near-misses, which are where the diagnosis actually lives. It is crude, it is fast, and it will tell you more about your embedding choice in an afternoon than a leaderboard will. After that, 03-05 closes the module by taking apart the one piece of embedding folklore most likely to give you bad debugging instincts: king − man + woman.