ML Learning Paradigms, Feature Engineering, and Cross-Validation Explained

Identity statement: machine learning builds a model that learns patterns from data rather than being explicitly programmed with rules, and then generalizes those patterns to new, unseen inputs it was never shown during training. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states this framing directly: models "learn patterns from data rather than being explicitly programmed with rules."

That definition has a load-bearing second half that is easy to skim past: generalizes to new, unseen inputs. A system that only reproduces training examples verbatim has not learned anything useful — it has memorized a lookup table. Every technique in this module, from cross-validation to the bias-variance tradeoff in the next lesson, exists to answer one question honestly: does this model's apparent skill transfer to data it has never seen, or does it only look skillful on the data it was trained on?

When it matters: any time a scenario describes a training setup and asks you to classify it, or asks you to predict what kind of data or signal a described technique needs. The classification step is almost always: what is the source of the learning signal — a label, structure in unlabeled data, a self-generated label, or a reward?

Why "paradigm" is about the signal, not the algorithm

A common wrong instinct is to sort ML by architecture — "it's a neural network, so it's deep learning" — but the architecture and the paradigm are independent choices. A neural network can be trained in a supervised way (image classification), an unsupervised way (an autoencoder learning to compress images with no labels), a self-supervised way (predicting a masked word), or a reinforcement way (a policy network trained on reward signal). The paradigm answers "where does the training signal come from," and that answer is what a scenario question is actually testing, even when the scenario describes a specific architecture as a distractor.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names exactly four paradigms, differentiated by the shape of the data and the goal:

L1 — Intuition

Think of the four paradigms as four different answers to "who tells the model it is right or wrong." Supervised learning has an external, human-provided answer key. Unsupervised learning has no answer key at all — it is only asked to notice patterns. Self-supervised learning manufactures its own answer key out of the raw data, without a human ever writing one down. Reinforcement learning has neither a fixed label nor a manufactured one; instead it has a reward that arrives after an action, sometimes delayed, and the model has to figure out which of its past actions the reward is actually crediting.

L2 — Mechanism

Supervised learning needs a dataset of (input, correct output) pairs collected in advance. The model's job is to learn the function that maps one to the other well enough that it also works on inputs it has never seen. Every classic classification and regression problem — will this transaction be fraudulent, what price will this house sell for — fits here, and the cost of this paradigm is that someone has to produce the labels, which does not scale for free.

Unsupervised learning removes the labeling requirement entirely and asks a different question: what structure is already present in this data, with no notion of "correct" imposed from outside? Clustering groups similar items without ever being told what the groups should mean; dimensionality reduction finds a lower-dimensional representation that still captures most of what matters in the original data. There is no accuracy metric in the supervised sense, because there is no ground-truth label to compare against.

Self-supervised learning is the paradigm that made large-scale pretraining possible, and it is worth understanding precisely because it looks unsupervised (no human wrote any labels) but behaves like supervised learning internally (there is a genuine correct-answer target at every training step). The trick is that the label is derived from the data itself, automatically, with no human annotator. A language model predicting the next token is self-supervised: the "label" for a given context is simply whatever token actually came next in the real text, which the training pipeline already has for free from any document. CLIP's contrastive pretraining is the multimodal instance of exactly this pattern — the "label" for an image is the caption that already existed next to it on the page it was scraped from, so 400 million image-caption pairs required zero human labeling effort beyond the captions the web already contained.

Reinforcement learning replaces the fixed label with a reward signal that an agent receives from an environment after taking an action, and the model's job is to learn a policy that maximizes the reward it accumulates over time. RLHF — reinforcement learning from human feedback — is the multimodal- and LLM-relevant instance: human preference judgments (which of two model outputs is better) get turned into a reward signal, and the model is trained to produce outputs that score higher against that learned reward.

L3 — Why this exact framing is the exam-relevant depth

The exam-relevant trap here is treating CLIP pretraining and RLHF as though they were separate, novel training paradigms unique to generative and multimodal AI, when they are in fact textbook instances of self-supervised and reinforcement learning respectively, applied to a new kind of data. A scenario that describes "a model pretrained on image-caption pairs scraped from the web, with no human-written classification labels" is describing self-supervised learning, full stop, even though the word "self-supervised" never appears in the scenario. Recognizing the paradigm underneath an unfamiliar-sounding technique is the actual skill being tested — the four-row table above is the complete answer key, and everything downstream in this course is one of its four rows wearing a more specific name.

Identity statement: a feature is a measurable input variable a model consumes; feature engineering is the process of transforming raw data into features a model can actually use, and it happens before training, not during it. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) frames this explicitly for the multimodal case: "In multimodal settings each modality needs its own preprocessing... before features can be combined."

For a tabular dataset, feature engineering might mean computing a ratio between two raw columns, or bucketing a continuous age value into ranges. For a multimodal system, the same idea applies per modality, and each modality's raw form is different enough that the preprocessing step looks completely different across the three:

• Text needs tokenization — breaking a string into the discrete units (words, subwords) a model's vocabulary understands, since a neural network has no native way to consume raw characters as meaningfully as it consumes numeric feature vectors.
• Images need pixel normalization and patching — rescaling raw pixel values onto a consistent numeric range, and often dividing the image into fixed-size patches, so different images become directly comparable, fixed-shape numeric inputs.
• Audio needs conversion into spectrograms or waveforms — a raw audio recording is a long, high-frequency numeric signal, and a spectrogram converts it into a time-versus-frequency representation that is a far more learnable input shape for a network.

The unifying idea across all three: whatever a model eventually fuses across modalities (a topic this module returns to directly in M1-11), it can only fuse features that already exist in some numeric, model-consumable form. Feature engineering is the step that gets each modality to that form independently, before any combination happens.

Before you can trust any performance number a model reports, the data supporting that number has to be divided correctly. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names three splits with three distinct jobs:

The test set's "used once" rule is stricter than it first sounds. If you look at test-set performance, decide a model is not good enough, tweak a hyperparameter, and check the test set again, you have quietly turned the test set into a second validation set — every subsequent decision is now informed by test-set feedback, and the final number you report is no longer an honest estimate of performance on genuinely unseen data. This is the same discipline objective 1.10's "judge on held-out data only" framing rests on, and it recurs as a named exam trap in Domain 3's experiment-design material.

Identity statement: k-fold cross-validation rotates the validation fold across k splits of the training data and averages the results, producing a more reliable performance estimate than a single fixed train/validation split — especially valuable when the available data is limited. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) describes exactly this mechanism.

Why one split is not enough

A single train/validation split has a hidden fragility: the reported validation performance depends partly on which rows happened to land in the validation fold. Draw an easy validation subset by chance, and the model looks better than it really is; draw a hard one, and it looks worse. With a small dataset, this luck-of-the-draw effect can be large enough to change which of two models looks better, purely as an artifact of the split, with no real difference in underlying quality.

The mechanism, step by step

1. Divide the full training dataset into k roughly equal-sized folds (a common choice is k = 5 or k = 10).
2. Train the model k separate times. On each run, one fold is held out as the validation set and the remaining k − 1 folds are used for training.
3. Record the validation metric from each of the k runs.
4. Average the k scores. That average is the cross-validated performance estimate.

Every data point ends up in the validation fold exactly once across the k runs, and in a training fold for the other k − 1 runs — so the final average reflects performance across the entire dataset's worth of validation examples, not just whichever subset one arbitrary split happened to draw.

5-fold cross-validation, 500 examples, 100 per fold:

Run 1: train on folds 2,3,4,5 (400 examples) -> validate on fold 1 (100) -> score 0.82
Run 2: train on folds 1,3,4,5 (400 examples) -> validate on fold 2 (100) -> score 0.79
Run 3: train on folds 1,2,4,5 (400 examples) -> validate on fold 3 (100) -> score 0.85
Run 4: train on folds 1,2,3,5 (400 examples) -> validate on fold 4 (100) -> score 0.81
Run 5: train on folds 1,2,3,4 (400 examples) -> validate on fold 5 (100) -> score 0.77

cross-validated estimate = (0.82 + 0.79 + 0.85 + 0.81 + 0.77) / 5 = 0.808

This is a constructed scenario with illustrative scores, not measurements from any real dataset — but the arithmetic pattern is exactly the mechanism: five independent estimates, one per fold, averaged into a single, more trustworthy number. Note the spread between the best run (0.85) and the worst (0.77): an eight-point range across folds of the same dataset and the same model is a direct, visible measurement of how much a single lucky-or-unlucky split could have distorted a one-shot validation score. Reporting 0.808 with that spread known is a materially more honest claim than reporting whichever single fold happened to be drawn first.

⭐ THE EARNED INSIGHT Cross-validation's real value is not a better point estimate — it is turning an invisible risk (this score might just be which rows landed in validation) into a visible, measured quantity (the spread across the k folds). A model whose fold scores cluster tightly is one you can trust; a model whose fold scores swing widely is telling you something about either the model's stability or the dataset's size that a single split would have hidden entirely.

Why cross-validation matters more, not less, with limited data

Cross-validation is most valuable exactly when a fixed train/validation split would be least trustworthy: small datasets. With abundant data, a single held-out validation set is already large enough that its score is stable and unlikely to swing much from an unlucky draw. With a genuinely small dataset — the kind multimodal fine-tuning projects often actually have, since labeled image-caption or audio-transcript pairs are expensive to produce — a single split can carve out a validation set too small to be reliable on its own, and cross-validation's k-way rotation squeezes a more reliable signal out of the same limited pool of examples by using every row for both training and validation, just never at the same time.

Plain k-fold cross-validation assumes that splitting the data into k roughly equal-sized chunks, without paying attention to what is in each chunk, is good enough. That assumption breaks on imbalanced data, and recognizing when it breaks is worth a dedicated pass before the worked examples.

Suppose a dataset for detecting a rare defect in manufactured parts is 95% "no defect" and 5% "defect" — a realistic imbalance for this kind of problem. Plain k-fold shuffles rows into k folds essentially at random, which means pure chance decides how many of the rare "defect" examples land in each fold. With only 5% of the data carrying the minority label, an unlucky shuffle can produce a fold with almost no defect examples at all, or — worse — a training set for one run that has too few defect examples to learn the pattern from.

Stratified k-fold fixes this by preserving the original class proportions inside every fold: if the full dataset is 95/5, each of the k folds is also built to be as close to 95/5 as the fold size allows, rather than left to chance. This does not change the cross-validation mechanism — it is still k rotations, still an average at the end — it changes only how the folds are constructed, and the practical effect is that every fold's validation score is measuring performance on a representative slice of the class distribution, not a slice that happened to be starved of the class you care about most.

Dataset: 1,000 examples, 950 "no defect" / 50 "defect" (95% / 5%)

Plain 5-fold (unlucky shuffle):
  Fold 3 draws only 3 of the 50 defect examples by chance
  -> that fold's validation score is nearly meaningless for defect detection

Stratified 5-fold:
  Every fold gets ~190 "no defect" and ~10 "defect" examples
  -> every fold's validation score reflects the true class balance

This is a constructed scenario illustrating the mechanism, not a measurement from a real manufacturing dataset. The takeaway generalizes past this one example: any time the target label is imbalanced — which is common in defect detection, fraud detection, and rare-disease classification alike — stratified k-fold is the version of cross-validation to reach for, and plain k-fold's fold-to-fold variance becomes a symptom of the imbalance rather than of genuine model instability.

A team is building three separate systems. For each, identify the learning paradigm from the signal source alone.

System A. A dataset of 50,000 X-ray images, each hand-labeled by a radiologist as "normal" or "abnormal." The model is trained to predict the label for a new X-ray.

Signal source: human-provided labels, known in advance, one per example.
-> SUPERVISED LEARNING.
The radiologist's label is the answer key; the model's job is to reproduce
correct answers on X-rays it has never seen.

System B. A dataset of 10 million image-caption pairs scraped from the web, with the caption used as its own label — no radiologist, no separate human annotator, just the caption that already existed next to the image.

Signal source: a label the data supplies for itself, automatically,
with no separate human-labeling step.
-> SELF-SUPERVISED LEARNING.
This is the CLIP pattern: the caption already existing next to the image
IS the supervision, at effectively zero additional labeling cost.

System C. A conversational assistant is shown pairs of its own candidate responses and a human rater's preference between them ("response 1 is better than response 2"), and is trained to produce responses that score higher against a model of that preference.

Signal source: a reward derived from human preference judgments,
not a fixed label attached to a single input in advance.
-> REINFORCEMENT LEARNING (specifically RLHF).
The "correct" response is never written down anywhere; only a relative
preference signal exists, and the model learns a policy that improves
against it over many rounds.

The pattern to generalize: read the scenario for where the training signal comes from, not for what the model architecture is or what kind of output it produces. All three systems above could plausibly use the same underlying transformer architecture — the paradigm classification depends entirely on the signal, never the network shape.

A team has trained two candidate architectures for a multimodal sentiment classifier (text plus a short audio clip) and cross-validated each with 5-fold CV on the same 2,000-example dataset.

Model A fold scores: [0.803, 0.811, 0.798, 0.809, 0.795]
  mean = 0.8032   spread (max - min) = 0.016

Model B fold scores: [0.760, 0.910, 0.700, 0.870, 0.780]
  mean = 0.8040   spread (max - min) = 0.210

Both models report almost the same mean — 0.8032 versus 0.8040, a gap of eight-tenths of a point, which on its own looks like Model B is marginally ahead. Reading only the mean would end the comparison there. Reading the spread changes the conclusion: Model A's five folds sit within 1.6 points of each other, meaning its performance is stable across different subsets of the data. Model B's five folds range across 21 points — the same model looks anywhere from mediocre (0.700) to excellent (0.910) depending purely on which fold it happened to be validated against.

Constructed scenario, illustrative numbers. The generalizable lesson: a near-identical mean does not mean two models are equally trustworthy. Model B's wide spread is itself evidence — it says the model's behavior is highly sensitive to exactly which examples it sees, which is a warning sign about robustness that would be invisible from a single train/validation split reporting one lucky (or unlucky) number. Between two models with similar averages, the one with the tighter fold-to-fold spread is very often the safer choice to ship, because its cross-validated estimate is a more faithful preview of how it will behave on the next batch of genuinely new data.

Each row names a specific symptom against a specific, checkable fix.

Core Machine Learning and AI Knowledge is Domain 1 of the NCA-GENM blueprint at 20% weight, second only to Experimentation, [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) — and the domain's own framing states that this weight reflects the fact that "everything else builds on it." The four learning paradigms are the most foundational item in that domain, because later lessons on multimodal loss functions, transfer learning, and RLHF-adjacent alignment all assume you can already place a described technique into one of the four rows without hesitation.

The question tends to arrive in a small number of recognizable shapes.

1. Paradigm classification. A scenario describes a dataset and a training goal, and asks which paradigm it uses. The keyed answer is read directly off the signal source — labeled, unlabeled, self-generated label, or reward.
2. "Which paradigm does CLIP/RLHF use" items. These test whether you recognize a named multimodal technique as an instance of an existing paradigm rather than treating it as exotic and unclassifiable.
3. Cross-validation mechanism recall. "What does k-fold cross-validation do?" with the keyed answer naming the fold-rotation-and-average mechanism, against distractors describing a single split, or a technique that changes what the model learns rather than how its performance is estimated.
4. Split-discipline items. A scenario describes reusing the test set during tuning, or tuning hyperparameters on the training set directly, and asks what is wrong.

What the distractors typically look like

The reliable distractor families: offering "unsupervised" for a self-supervised setup because no human labeler is visible in the scenario, and describing cross-validation as simply "using more data" rather than the specific rotate-and-average mechanism. A third family conflates the validation set's repeated-use role with the test set's single-use role. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) calls out exactly this conflation directly, as tuning on the test set.

Unsupervised learning has no target at all — the model is only asked to find structure (clusters, a lower-dimensional representation) in unlabeled data, with no notion of a "correct" output anywhere in the process. Self-supervised learning does have a genuine target at every training step, but that target is derived automatically from the data itself rather than supplied by a human annotator — a language model's next-token target is the token that actually appears next in real text, and CLIP's target is the caption that already existed alongside the image. The practical tell is whether there is a right-answer-to-compare-against during training: unsupervised learning has none, self-supervised learning has one it manufactured for free.

A single train/validation split's reliability depends on the validation fold being large and representative enough that its score would not change much under a different random draw. With a large dataset, an ordinary validation split is already big enough to be stable. With a small dataset — common in multimodal fine-tuning, where labeled image-caption or audio-transcript pairs are expensive to collect — a single validation fold can be too small to trust on its own, and a single unlucky or lucky draw can swing the reported score substantially. K-fold cross-validation squeezes a more reliable estimate out of the same limited pool by rotating every example through the validation role exactly once, so the final averaged score reflects the whole dataset's worth of validation performance rather than one arbitrary slice of it.

• Machine learning is classified into four paradigms — supervised, unsupervised, self-supervised, reinforcement — by the source of the training signal, never by the model's architecture.
• CLIP pretraining is self-supervised (the caption is a free, self-supplied label) and RLHF is reinforcement learning (the reward is derived from human preference); neither is a new, separate paradigm.
• Feature engineering must happen per modality — tokenization for text, pixel normalization/patching for images, spectrograms for audio — before any fusion can occur.
• The test set is used exactly once; the validation set is reused during tuning; reusing the test set during tuning quietly inflates the final reported performance.
• K-fold cross-validation rotates the validation fold across k splits and averages the results, producing a more reliable estimate than a single split — and it matters most exactly when data is limited.
• The spread across fold scores, not just their average, is itself useful information about a model's stability.

This module now moves from "what kind of learning is happening" to "how do you tell if the learning went well." Next: M1-02 covers overfitting, underfitting, and the bias-variance tradeoff — the discipline of judging a model on held-out performance rather than mistaking a high training score for a genuine success signal, which is the exact failure mode cross-validation's held-out folds are built to catch.