Cross-Validation Explained: k-Fold, Stratified, and When Not to Use It

Cross-validation is a way of estimating how well a modelling procedure generalises, by reusing the same data as both training and evaluation material across several rotations. Standard k-fold, with k = 5, on 100 items:

Fold 1: train on items  21-100 (80 items), test on items   1-20
Fold 2: train on items 1-20 + 41-100,     test on items  21-40
Fold 3: train on items 1-40 + 61-100,     test on items  41-60
Fold 4: train on items 1-60 + 81-100,     test on items  61-80
Fold 5: train on items  1-80,             test on items  81-100

report: mean of the five test scores, and their standard deviation

Two properties follow directly from the construction and are worth stating as the metric's identity:

1. Every item is used for testing exactly once and for training exactly k−1 times. No item is wasted, which is why cross-validation is the standard technique on small datasets where a 20% holdout would be too small to measure anything.
2. You get k scores, so you get a spread. A mean of 0.82 with a standard deviation of 0.02 is a very different finding from a mean of 0.82 with a standard deviation of 0.11, and a single split cannot distinguish them.

The important conceptual caveat: cross-validation evaluates a procedure, not a model. Each fold trains a different model, so at the end you have k models and no single artefact. What you have estimated is "if I train this pipeline on data like this, how well does it do on unseen data like this?" To ship, you retrain on all the data using the procedure you validated. That distinction generates several exam questions.

The variants you must be able to name and place:

Stratified k-fold and grouped k-fold are the two that prevent the most damage, and both are frequently omitted.

L1 — Intuition: rotate the holdout

You have 100 labelled items and want to know how good your model is. Hold out 20 and you have a 20-item test set — which, per 09-01, gives you a ±22-point interval and cannot be split by slice. Hold out 50 and your training set halves. Cross-validation escapes the dilemma by rotating: every item eventually serves as test data, so your effective evaluation size is the whole 100, while every training run still sees 80.

The cost is k training runs instead of one. Whether that is cheap or catastrophic depends entirely on what "training" means — which is exactly why cross-validation is routine in classical ML and largely impractical for LLM fine-tuning.

L2 — Mechanism: the procedure and where the leaks are

1. Shuffle the data — unless it is time-ordered or grouped (see below)
2. Partition into k folds
   - stratified: partition within each class so folds share the class distribution
   - grouped:    partition by group id, never splitting a group
3. For i in 1..k:
     a. train_set = all folds except i
     b. test_set  = fold i
     c. FIT every preprocessing step on train_set only
        (scaler, imputer, vectorizer, feature selector, encoder, resampler)
     d. TRANSFORM test_set with the already-fitted preprocessors
     e. train the model on train_set, score on test_set → score_i
4. Report mean(score_i) and std(score_i)
5. To ship: refit the whole pipeline on all the data

Step 3c is the step that is skipped, and skipping it is the single most common cross-validation defect. If you fit a scaler, a TF-IDF vectorizer, an imputer, a feature selector, or an oversampler on the entire dataset before splitting, then information from the test fold has leaked into the training process, and every fold score is optimistically biased. Nothing about the resulting number looks wrong; it is just too high. The defence is mechanical: put every preprocessing step inside a pipeline object and cross-validate the pipeline, not the model. 01-07 covers the leakage discipline in general, and 08-02 treats leakage as a data-quality defect.

Choosing k. The trade-off is bias against variance and cost.

k = 5 or k = 10 are the conventional choices, and the reasoning is that they hold bias low without the extreme variance and cost of LOOCV. LOOCV's variance problem is worth understanding: each fold's score on a single item is 0 or 1 for a classification metric, so the individual scores are maximally noisy, and the k models are nearly identical to each other (they differ by one training item), which means their errors are highly correlated and averaging them does not reduce variance as much as independence would suggest.

L3 — Depth: the four situations where cross-validation is wrong

1. Time-ordered data. Random k-fold trains on the future to predict the past. If your target has any temporal structure — demand, churn, prices, traffic, anything with a trend or a seasonal cycle — random folds let the model see later data and then test it on earlier data, which is impossible at deployment time and inflates the score. Use forward-chaining splits instead:

Fold 1: train on Jan-Mar,  test on Apr
Fold 2: train on Jan-Apr,  test on May
Fold 3: train on Jan-May,  test on Jun

Note this violates cross-validation's "every item tested once" property — the earliest data is never tested — and that is the correct trade.

2. Grouped or clustered data. If your dataset has multiple rows per entity — several utterances per speaker, several notes per patient, several chunks per source document — random folds put some of an entity's rows in training and others in test. The model then recognises the entity rather than learning the task, and the score reflects memorisation. Use GroupKFold with the entity id as the group. This is the leakage mode that most often survives a code review, because the split looks random and correct.

For a RAG corpus this bites in a specific way: if a document has been chunked into ten overlapping pieces (06-02), those ten chunks are near-duplicates, and random splitting puts nine in training and one in test. The relevant group id is the source document, not the chunk.

3. Cost-prohibitive training. Cross-validating a full LLM fine-tune means k fine-tuning runs. At the GPU cost from 11-04, k = 5 is a five-fold budget increase for a variance estimate. This is why cross-validation is the norm in classical ML and the exception in LLM work — a point covered in the next section.

4. When the thing you are evaluating does not train at all. A prompt is not trained. A zero-shot model is not trained. A RAG pipeline over a fixed corpus is not trained. In all three cases there is no train/test split to rotate, so classical k-fold does not apply — although a resampling idea still does, and that is where bootstrap resampling of the evaluation set enters, which 09-09 develops.

One more depth point about interpretation. The standard deviation across folds is not the standard error of the mean, and it is a mistake to treat it as one. The k fold scores are not independent — the training sets overlap heavily — so the usual std / sqrt(k) understates the true uncertainty. Use the fold spread as a qualitative signal of instability ("this model's performance depends a lot on which data it saw"), and use a proper interval on the pooled predictions for quantitative claims (09-09).

The three are not competitors so much as tools for three different questions. "Will this modelling approach generalise?" → cross-validation. "Is this specific trained artefact good enough?" → a held-out test split. "Did today's change to the system make it better than yesterday's?" → a frozen evaluation set. Confusing the third with the first is the mistake that produces "we cross-validated our prompt", which is not a coherent activity as usually described.

Two further distinctions the exam probes:

• Validation split vs test split. The validation split tunes hyperparameters and is looked at many times; the test split is looked at once, at the end, to report. Cross-validation replaces the validation split, not the test split. If you tune with cross-validation and then report the best cross-validated score as your final performance, that number is optimistically biased by the selection itself — which is precisely what nested cross-validation exists to fix, and which connects to the multiple-comparisons problem in 09-09.
• Cross-validation vs benchmark contamination. Cross-validation controls leakage within your dataset. It cannot detect that your dataset's items were in a foundation model's pretraining corpus. That is a different leak, at a different layer, and it is 10-01's subject.

Constructed example. A classifier is 5-fold cross-validated on 100 labelled items; each fold has 20 test items.

Step 1 — the mean.

mean = (0.85 + 0.75 + 0.90 + 0.80 + 0.80) / 5 = 4.10 / 5 = 0.820

Note that because all folds are the same size, the mean of the fold accuracies (0.820) equals the pooled accuracy (82/100 = 0.820). With unequal fold sizes these two differ, and the pooled figure is usually what you want, so either keep folds equal or weight the mean by fold size.

Step 2 — the standard deviation across folds.

deviations from 0.820:
  0.85 - 0.82 =  0.03  → 0.0009
  0.75 - 0.82 = -0.07  → 0.0049
  0.90 - 0.82 =  0.08  → 0.0064
  0.80 - 0.82 = -0.02  → 0.0004
  0.80 - 0.82 = -0.02  → 0.0004

sum of squared deviations = 0.0130

population std  = sqrt(0.0130 / 5) = sqrt(0.00260) = 0.0510
sample std      = sqrt(0.0130 / 4) = sqrt(0.00325) = 0.0570

Report: accuracy 0.820 ± 0.057 (sample standard deviation across 5 folds). State which standard deviation you used; the sample version (dividing by k−1) is the conventional choice.

Step 3 — read the spread. Fold scores range from 0.75 to 0.90 — a 15-point spread across folds on the same data with the same procedure. That range is the honest headline. If someone had run a single 80/20 split and happened to land on fold 3's partition, they would have reported 0.90 and believed it. Landing on fold 2, they would have reported 0.75. The single-split number is a sample from this distribution, and cross-validation is what shows you the distribution exists.

Step 4 — compare two models properly. Suppose model B is cross-validated on the same folds:

Model B's mean is 3 points higher. Is that real? Look at the paired differences, not the two means: [0.00, +0.05, 0.00, +0.05, +0.05]. Every difference is zero or positive, and the mean difference is +0.030 with a sample standard deviation of:

mean difference = 0.15 / 5 = 0.030
deviations: -0.03, +0.02, -0.03, +0.02, +0.02
squared:     0.0009, 0.0004, 0.0009, 0.0004, 0.0004  → sum 0.0030
sample std  = sqrt(0.0030 / 4) = sqrt(0.00075) = 0.0274

The difference (0.030) is a bit above one standard deviation of the differences (0.027), and B never loses to A on any fold. That is suggestive but not strong evidence — with five folds you have very little statistical power, and the folds are not independent. The key methodological point is the pairing: comparing B's mean to A's mean throws away the fact that they were measured on identical folds, and the paired view has far less noise because fold difficulty cancels out. Always use the same folds (same seed) when comparing procedures. The formal machinery for turning this into a claim is 09-09's subject.

Step 5 — see the effect of stratification. Suppose the 100 items are 90 negative and 10 positive, and plain (unstratified) 5-fold happens to distribute the positives as: fold 1 → 4 positives, fold 2 → 0, fold 3 → 3, fold 4 → 1, fold 5 → 2.

Fold 2 contains zero positive items. Any metric involving the positive class is undefined there: recall's denominator is zero, and precision's may be too. F1 cannot be computed. Implementations either error, or silently return 0, or return NaN and drop the fold — and each of those choices produces a different, misleading average. Meanwhile fold 1's four positives mean a single misclassification swings its recall by 25 points.

Stratified 5-fold puts exactly 2 positives and 18 negatives in each fold. Every fold can compute every metric, and fold-to-fold variance drops sharply because the folds are now comparable. For any classification task, stratified is the default, and it becomes mandatory as soon as the minority class is small — which, as 09-05 shows, is exactly when accuracy is misleading and you need per-class metrics that stratification makes computable.

The compressed rule: cross-validate when training is cheap and data is scarce; use a frozen evaluation set when training is expensive or absent. LLM application work is overwhelmingly the second case, which is why the entire rest of this module is built on frozen evaluation sets rather than folds.

Cross-validation is named verbatim in an official objective, which makes it one of the few topics in this module with unambiguous textual support. Objective 1.5 reads: "Familiarity with fundamentals of machine learning (e.g. feature engineering, model comparison, cross validation)." The Experimentation domain's suggested-reading list also names cross-validation directly. That combination — an objective naming it plus a suggested reading about it — puts it firmly in scope.

The objective-numbering defect, stated where the objectives are cited. The official study guide prints the Experimentation domain's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5 — data-mining awareness, comparing models using statistical performance metrics, conducting data analysis under supervision, creating charts, identifying relationships and trends. Read literally, 22% of the exam has objectives describing data analysis and visualization rather than model evaluation and RLHF, contradicting the section's own scope statement: "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)." The suggested-reading list agrees with the scope statement, and candidate reports independently confirm evaluation and RLHF content on the exam. Teach and answer from the derived scope. For this lesson, the objectives that legitimately apply are 1.5 (which names cross-validation outright), the duplicated pair 2.3 / 3.3 (conduct data analysis under supervision), and 2.5 / 3.5 (identify factors that could affect the results of research — leakage and an unlucky split are exactly such factors).

Question phrasings to expect:

• "What is the primary advantage of k-fold cross-validation over a single train/test split?" → Every item is used for both training and evaluation, and you obtain a variance estimate as well as a mean; the result depends less on one lucky or unlucky partition.
• "Which cross-validation variant preserves the class distribution in each fold?" → Stratified k-fold.
• "When should you not use random k-fold cross-validation?" → With time-ordered data (use forward-chaining), with grouped data (use GroupKFold), or when training cost makes k runs infeasible.
• "A scaler is fitted on the whole dataset before cross-validation. What is the consequence?" → Data leakage; the cross-validated scores are optimistically biased.
• "What does leave-one-out cross-validation do?" → Sets k = n, so each item is its own test fold; nearly unbiased but high-variance and expensive.
• "Why is nested cross-validation used?" → To obtain an unbiased performance estimate when hyperparameters are also being tuned, since reporting the best inner-loop score is biased by the selection.
• "After cross-validation, which model do you deploy?" → None of the k fold models directly; refit the validated procedure on all the data.
• "Why is cross-validation rarely used for LLM fine-tuning?" → Each fold requires a full training run, so the compute cost multiplies by k.
• "What does a large standard deviation across folds indicate?" → The procedure's performance is unstable with respect to which data it trains on, often a sign of a small dataset or high model variance.

Distractor families. (1) Cross-validation described as preventing overfitting — it detects overfitting by giving an honest generalisation estimate; it does not prevent it. Regularisation, early stopping and more data prevent it (01-01). (2) Random k-fold offered for time-series data — a favourite, because it sounds like best practice. (3) Stratification described as balancing the classes — stratification preserves the existing ratio in each fold; it does not resample to equalise classes. (4) "Cross-validation removes the need for a test set" — it replaces the validation split, not the final untouched test split. (5) Higher k presented as strictly better — higher k lowers bias but raises variance and cost, and LOOCV is notably high-variance. (6) Cross-validation offered as a fix for benchmark contamination — wrong layer entirely (10-01).

Plain k-fold partitions the data at random, so each fold's class distribution is whatever chance delivers. Stratified k-fold partitions within each class, so every fold reproduces the overall class ratio as closely as integer arithmetic allows. On a balanced dataset the two behave almost identically; on an imbalanced one they diverge sharply.

The §4 example makes it concrete: with 10 positives among 100 items, plain 5-fold gave folds containing 4, 0, 3, 1 and 2 positives. The fold with zero positives cannot compute recall, precision or F1 for the positive class at all — the denominator is zero — and depending on the library that yields an error, a silent zero, or a NaN that quietly changes the average. Stratified 5-fold gives every fold exactly 2 positives, so every metric is defined everywhere and fold-to-fold variance falls because the folds are comparable in difficulty.

One clarification that is a common exam distractor: stratification preserves the class ratio; it does not fix the imbalance. Every fold in the stratified example is still 90% negative. If you want to change the training distribution, that is resampling (oversampling the minority or undersampling the majority) — a different intervention, which must be applied inside each fold's training set only, never before the split, or you have leaked test items into training via duplicated minority examples.

Three reasons, in descending order of how often they bind.

Cost. Cross-validation requires k training runs. A full LLM fine-tune costs GPU hours in quantity (11-04), so k = 5 is a five-fold budget multiplier for a variance estimate you can approximate more cheaply. Even LoRA, which is dramatically cheaper (11-05), makes k = 5 a real decision rather than a free default.

Most LLM work involves no training at all. Prompt engineering, RAG configuration, decoding-parameter choices, model selection between hosted APIs — none of these fit parameters on your data, so there is no train/test rotation to perform. The right instrument is a frozen evaluation set scored repeatedly (09-01), and the right way to get an uncertainty estimate is bootstrap resampling of that set rather than k-fold (09-09).

The contamination layer is different. Cross-validation controls leakage between your own train and test partitions. It says nothing about whether a foundation model already saw your evaluation items during pretraining, which is a leak at a layer you do not control and cannot fix by re-partitioning (10-01).

Where cross-validation does remain valuable in LLM-adjacent work: training a classifier head or a reranker on a modest labelled set; fitting a small model on embeddings you generated; validating a classical ML component inside a larger LLM system; and tuning a LoRA adapter's hyperparameters when the dataset is small enough that a single validation split would be too noisy to choose between configurations. In all four cases the training step is cheap and the data is scarce — which is cross-validation's home territory, exactly as it always was.

No — it detects it. This distinction is worth being pedantic about because the misconception is widespread and the exam tests it. Cross-validation is a measurement procedure: it gives you an honest estimate of generalisation performance, so if your model has memorised its training data you will see it as a gap between training and fold scores. Nothing about rotating folds changes the model's tendency to memorise.

What actually reduces overfitting: more or more diverse data; regularisation (L1/L2, dropout, weight decay); early stopping on a validation signal; reducing model capacity; and data augmentation. Cross-validation's contribution is indirect but real — it makes the early-stopping and capacity decisions measurable, and it stops you from being fooled by a single lucky split. In the same way, cross-validation does not prevent leakage; it exposes leakage only if the leakage is not itself inside the cross-validation loop, which is why a preprocessing step fitted before the split is so dangerous: it defeats the very instrument that would otherwise have caught it.

Start at k = 5 and move to k = 10 if training is cheap and the dataset is small. The reasoning behind that default is a three-way trade.

Bias falls as k rises, because each fold trains on a larger fraction of the data (1 − 1/k), so the models being evaluated are closer to the model you would actually ship. k = 2 trains on half the data and therefore reports pessimistically.

Variance rises as k rises past a point, culminating in LOOCV, where each test fold is a single item and the fold scores are maximally noisy while the k training sets are almost identical, so their errors are strongly correlated and averaging buys less than you would hope.

Cost is linear in k. Five folds is five training runs.

Two practical constraints override the default. First, k cannot exceed the minority-class count if you want every stratified fold to contain at least one minority item — with 8 positives, k = 10 is impossible to stratify sensibly. Second, if fold scores come out with a wide spread, prefer repeated stratified k-fold (say 5 folds × 5 repeats with different seeds) over simply raising k: repeats reduce the influence of one unlucky partitioning without shrinking the test folds toward single items. And whichever you choose, fix it in the metric contract before you run, so k cannot become a parameter you tune until the number looks good.

1. k-fold rotates the holdout, so every item tests exactly once and trains k−1 times.
2. The spread is half the result. In the worked example, fold accuracies ranged 0.75–0.90 around a mean of 0.820 ± 0.057 — a single split could have reported either extreme.
3. Stratified k-fold is the default for classification. With 10 positives in 100 items, plain folds produced one fold with zero positives and therefore undefined positive-class metrics.
4. Stratification preserves the class ratio; it does not fix imbalance. Resampling does that, and only inside the training fold.
5. Fit every preprocessing step inside the fold. A scaler or vectorizer fitted before the split leaks the test fold into training and inflates every score.
6. Never use random folds on time-ordered data (use forward chaining) or on grouped data (use GroupKFold on the entity id).
7. k = 5 or 10 is the conventional choice. Higher k lowers bias but raises variance and cost; LOOCV is the extreme case of both.
8. Compare models on identical folds and analyse paired differences — pairing cancels fold difficulty and cuts noise substantially.
9. Fold standard deviation is not a standard error. The folds share training data, so std / sqrt(k) understates uncertainty.
10. Cross-validation validates a procedure, not an artefact. Refit on all the data to ship.
11. It detects overfitting; it does not prevent it. Regularisation, more data and early stopping prevent it.
12. It is rarely right for LLM application work — training is expensive or absent, so a frozen evaluation set plus bootstrap uncertainty is the correct instrument.
13. Objective 1.5 names cross-validation verbatim, which makes it one of the few topics in this module with direct textual support in the official objectives.

Cross-validation gave you a mean and a spread, and immediately raised a question it cannot answer: when model B beats model A by three points, is that a real difference or is it the same noise that made fold 2 score 0.75 and fold 3 score 0.90? Every comparison in this module has quietly assumed you can tell. You cannot, without arithmetic — and the arithmetic is unforgiving, especially once you have tried a dozen prompt variants and are about to report the best one.

Next: 09-09 covers sample size and statistical significance in LLM evaluation: the standard error of a proportion, how wide your intervals really are at a hundred items, paired testing, and what happens to your false-positive rate when you compare twelve prompt variants against one baseline.