Keeping a Vector Index Fresh: Re-Embedding and Migration

A vector index (built with an ANN structure like HNSW or IVF — see 07-04 for the indexing mechanics themselves) is only useful if the vectors inside it accurately represent the documents a retriever is meant to find. That representation decays in two distinct ways, and conflating them is a common source of confused incident response.

The first kind of staleness is corpus drift: the underlying documents change — new pages get published, existing ones get edited, old ones get deleted or superseded — but the index isn't updated to match. A support knowledge base that adds ten new articles a week but only re-indexes monthly is serving stale results for up to a month at a time; a user asking about a feature documented last Tuesday gets nothing, because Tuesday's article was never embedded. This kind of staleness is incremental and additive: you can fix it by embedding just the new or changed documents and either adding them to the index or overwriting the stale entries, without touching anything else.

The second kind of staleness is model drift, and it is categorically different: it's not that the corpus changed, it's that the function used to turn text into vectors changed. If a team upgrades from one embedding model to a newer one — because the new model has better retrieval accuracy, or the old one was deprecated, or a benchmark showed it separates near-duplicate content better — every vector already sitting in the index was produced by the old model's geometry. The new model's queries will be embedded into a different vector space, and comparing a new-model query vector against old-model document vectors via cosine similarity or dot product produces numbers that mean nothing: the two spaces are not aligned, not even approximately, because different models learn different geometric arrangements of meaning even when trained on similar objectives. There is no incremental fix for this. The entire corpus has to be re-embedded with the new model before that model can be used for a single production query, which is why model choice is treated here as a migration decision rather than a runtime configuration switch.

L1 — The intuition: a card catalog filed in two different systems

Imagine a library's card catalog where every card lists a book's shelf location using a coordinate system: aisle number, shelf number. Corpus drift is simple — a new book arrives, you write a new card with its coordinates, and file it. An edited book (new edition) gets its old card pulled and a new one filed. That's ordinary catalog maintenance, and you do a little of it constantly.

Now imagine the library decides to renumber the entire aisle-and-shelf system — a better scheme that groups books more usefully. Every card in the old catalog gives coordinates in the old system. If you hand someone a new-system coordinate and ask them to find the matching old-system card, there's no way to do it — the numbers aren't offset by some fixed amount, they're not even the same kind of number in the new arrangement. You cannot patch this a card at a time. You have to walk every shelf, note every book's location under the new system, and rebuild the entire catalog before the new coordinate system means anything for a single lookup. That is exactly the situation when you switch embedding models: the "coordinate system" (the vector space) changed entirely, and nothing in the old index can be reinterpreted under it.

L2 — The mechanism: incremental updates vs full re-embedding

Incremental freshness (corpus drift) is maintained by a pipeline that: (1) detects new, changed, or deleted source documents — typically via a content hash, a last-modified timestamp, or a change-data-capture feed from whatever system owns the source content; (2) for new or changed documents, chunks and embeds only those documents with the current embedding model; (3) upserts the resulting vectors into the index (adding new IDs, replacing vectors at existing IDs for edited documents); and (4) removes vectors whose source document was deleted. None of this touches the vectors for documents that didn't change, which is the entire point — the cost of freshness scales with how much of the corpus actually changed, not with the corpus's total size. 02-04-style productionizing-RAG material covers this pipeline at the application level; this lesson is specifically about why the embedding-model case breaks that incremental assumption.

Full re-embedding (model drift) has no equivalent shortcut, because step (2) above — "embed only what changed" — has no meaning when what changed is the embedding function itself, not the documents. Every single chunk in the corpus, changed or not, has to be run through the new model and its vector recomputed. This is a batch job whose cost scales with the entire corpus size, not with a delta, and it has real operational shape:

1. Stand up the new index alongside the old one. Never re-embed in place into the same index the old model is still serving from — a partially-migrated index, with some vectors from the old model and some from the new one intermixed, is actively worse than either pure state, because similarity comparisons across the mixed set are meaningless in both directions.
2. Re-embed the entire corpus with the new model into the new index. This is the expensive step — an embedding API or model call per chunk, for every chunk that exists, which for a large corpus can be a genuinely significant compute and cost line item, comparable in shape to 12-09's cost-per-token accounting but on the embedding side rather than the generation side.
3. Validate the new index before cutting over. Run the existing eval set (see 01-08) against the new index and compare retrieval quality metrics to the old index's baseline before anyone routes production traffic to it. A model upgrade that looks better on a benchmark can still retrieve worse on your specific corpus and query patterns — the only way to know is to actually measure it against your own eval set, not trust the vendor's reported numbers.
4. Cut over atomically. Switch the retrieval service's target index from old to new in one deployment step, not gradually, because there is no valid intermediate state where queries embedded with the new model should be compared against a partially-migrated index.
5. Keep the old index available for rollback until the new one has been observed under real production traffic for long enough to trust it, then retire it.

L3 — Why this isn't just "re-run the embedding script"

The naive mental model treats re-embedding as a script you re-run — swap the model name in a config, rerun the ingestion pipeline, done. Two things make it more than that in practice. First, embedding an entire corpus at production scale is not instantaneous — for a corpus of hundreds of thousands or millions of chunks, re-embedding can take hours to days depending on the embedding provider's throughput limits and your own compute budget, and that duration is exactly the window during which the old index has to keep serving live traffic correctly while the new one builds in the background. Second, embedding models frequently differ not just in their vector space but in their expected input conventions — maximum sequence length, whether the model expects a task-specific prefix (some embedding models are trained with different encodings for "this text is a query" vs "this text is a document," and mixing those up silently degrades retrieval quality even within one consistent model), and dimensionality. 03-03 covers choosing an embedding model and the dimension-vs-memory trade-off directly; the version-pinning discipline that lesson sets up is exactly what makes it possible to know, later, that a re-embedding migration is needed at all — without version pinning, "did the embedding model change" is not even a question you can answer with confidence.

These three failure modes are easy to conflate because all three present as "retrieval got worse," but they have different causes and different fixes.

The practical diagnostic question when retrieval quality drops: did specific documents stop being found (corpus drift), or did retrieval quality degrade across the board right after a model or configuration change (model drift or corruption)? Those two symptoms point in opposite directions for where to look first.

Consider a documentation corpus with 400,000 chunks, currently indexed with an embedding model that produces 768-dimensional vectors, being migrated to a newer model that produces 1,024-dimensional vectors. This example is constructed to illustrate the arithmetic of a migration's scope — it is not a benchmark of any specific vendor's pricing or throughput.

Step 1 — the re-embedding workload. All 400,000 chunks need to pass through the new model once. If the embedding API processes chunks at a batch rate of 2,000 chunks per minute (an illustrative, assumed rate for this example), the full corpus takes 400,000 ÷ 2,000 = 200 minutes, or roughly 3.3 hours of pure embedding throughput — before accounting for any rate limiting, retries, or orchestration overhead that would realistically stretch this in production.

Step 2 — the storage delta. The new model's vectors are larger — 1,024 dimensions instead of 768, roughly 33% more floats per vector. At 4 bytes per float (standard float32 storage), the old index stores 400,000 × 768 × 4 bytes ≈ 1.23 GB of raw vector data; the new index stores 400,000 × 1,024 × 4 bytes ≈ 1.64 GB — an increase of about 410 MB, before any ANN graph overhead the index structure itself adds on top of the raw vectors. During migration, both indexes exist simultaneously (Section 2's step 1), so peak storage during the cutover window is the sum of both — roughly 2.87 GB — not just the new index's footprint.

Step 3 — the validation pass. Before cutover, the eval set from 01-08 is run against both the old and new index, retrieving the top-k chunks for each of the eval questions and comparing which index surfaces the correct source document more often. Suppose the eval set has 100 questions; running each against both indexes is 200 retrieval calls — cheap and fast relative to the re-embedding step itself, but the step that actually justifies the migration, since a "better" model on paper that retrieves worse on this specific corpus should not be shipped regardless of how the migration's other numbers look.

Step 4 — reading the result. If the new index retrieves the correct source document in the top-3 results for 91 of the 100 eval questions, versus 84 of 100 for the old index, that's the evidence the migration was worth the 3.3 hours of embedding compute and the storage delta — a concrete, measured improvement rather than a decision made on a benchmark leaderboard alone. If the new index instead scores 79 of 100, the honest conclusion is that this specific model upgrade underperforms on this specific corpus despite whatever general benchmark motivated trying it, and the migration should not proceed to cutover.

The decision rule: ask whether the documents changed or the embedding function changed. Document changes are incremental and cheap. Any change to the embedding model — upgrade, provider switch, even a stated dimension or preprocessing change within the same model family — invalidates every existing vector and requires a full migration, because there is no partial-compatibility guarantee between two different vector spaces.

Objective 1.4 ("curate and embed content datasets for RAGs") and 1.8 ("select and use models to create text embeddings") both put embedding-model choice and corpus maintenance squarely in scope, and 03-03's treatment of choosing an embedding model sets up exactly the version-pinning discipline this lesson depends on. Expect scenario questions where a RAG system's retrieval quality has degraded and the question asks you to identify the cause — a distractor set will typically include "the vector database needs more storage," "the retrieval algorithm needs tuning," and the correct answer, "the embedding model was changed without re-embedding the corpus." The exam's calibration notes (per COURSE-INDEX) that scenario questions reward recognizing the right category of fix over deep mechanism, so the load-bearing fact to hold cold is simply: changing the embedding model invalidates the whole index, corpus changes alone do not. A second likely phrasing tests whether you understand that "re-indexing" and "re-embedding" are not synonyms — re-indexing can mean rebuilding an ANN structure over unchanged vectors, while re-embedding specifically means recomputing the vectors themselves.

Every vector already in the index was computed by the old model and sits in a different vector space than anything the new model will produce; queries embedded with the new model then get compared against those old vectors, and the resulting similarity scores are essentially meaningless. Retrieval quality collapses — not gradually, but immediately and across the board — because the mismatch affects every single query, not just edge cases. The only fix is re-embedding the entire corpus with the new model before it serves any production query.

Re-embedding means recomputing the actual vectors — running text back through an embedding model to produce new numbers, which is required whenever the embedding model itself changes. Re-indexing means rebuilding the search structure (the ANN graph or index, such as HNSW or IVF, covered in 07-04) over vectors that may not have changed at all — useful for restoring search performance or recall after many incremental insertions and deletions have degraded the structure. A corpus that has only had documents added or edited needs incremental re-embedding of just those documents; a corpus whose embedding model changed needs full re-embedding of everything, which then also requires a fresh index to hold the new vectors.

As often as the underlying corpus actually changes in ways that matter to users — there's no universal cadence, because a corpus that publishes new content hourly needs near-real-time incremental updates, while one that changes a few times a month is fine with a daily or weekly batch refresh. The right approach is event-driven or scheduled incremental upserts (Section 5) tied to actual content-change signals, keeping the cadence matched to how stale a result would have to get before it actually hurts a user, rather than picking an arbitrary refresh interval unrelated to how the corpus behaves.

• Corpus drift: staleness caused by source documents changing (added, edited, deleted) without the index being updated to match.
• Model drift (embedding): incompatibility introduced when the embedding model itself changes, invalidating every previously computed vector.
• Re-embedding: recomputing vectors for text using a (typically new) embedding model.
• Re-indexing: rebuilding an index's search structure (e.g., an ANN graph) without necessarily recomputing the underlying vectors.
• Incremental upsert: adding, replacing, or removing individual vectors in response to specific document changes, rather than rebuilding the whole index.
• Atomic cutover: switching production traffic from an old index to a new one in a single step, avoiding any intermediate mixed state.

Corpus drift and model drift are different problems with different fixes: new or edited documents need incremental re-embedding of just the changed content, while a change to the embedding model itself invalidates every vector in the index and requires re-embedding the entire corpus from scratch into a freshly built index. There is no safe way to mix vectors from two different embedding models in one index, no way to validate a model upgrade without testing it against your own eval set before cutover, and no way to shortcut the arithmetic of re-embedding a corpus that scales with its total size rather than with what changed. Treat any embedding-model swap as a migration project — new index, full re-embed, validation against a real eval set, atomic cutover, and a rollback path — not as a line in a config file.

Next: 12-13 moves from the index layer to the serving layer itself — deploying models with NVIDIA NIM and Triton Inference Server, two frequently confused products that solve genuinely different problems in getting a model from a trained checkpoint to a request that a real application can call.