Encoder Foundation Models and Masked Language Modeling

Identity statement: masked language modeling (MLM) is a pretraining objective that randomly hides a fraction of the tokens in an input sequence and trains the model to predict the original identity of each hidden token, using the surrounding context on both sides of the hidden position.

When it matters: whenever a scenario describes pretraining an encoder-based model — BERT and its family are the canonical example — or asks you to identify which training objective produced a model exhibiting bidirectional, non-generative behavior rather than left-to-right text generation.

[GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "Objective 4.7 folds in training encoder-based foundation LLMs with masked language modeling (predict masked tokens) — the pretraining objective behind BERT-style models — alongside understanding quantization/distillation/pruning as the levers that shrink these models for deployment." Mechanically, MLM takes an input sequence, replaces some fraction of its tokens with a special mask placeholder (or, in some variants, with another token entirely, to make the task slightly harder and prevent the model from simply learning to recognize the mask marker itself), and trains the model to output the correct original token at each masked position. Because the model can see tokens on both sides of a masked position — the words before it and the words after it in the same input — it learns to build a representation of each token that incorporates context from the entire surrounding sequence, not merely from what came before it.

This bidirectional visibility is the property that makes MLM specifically an encoder-appropriate objective. M1-03 established the architecture-family pairing this module now builds on directly: encoder-only architectures are naturally bidirectional, because there is no causal mask restricting what a given position can attend to, and MLM is the training objective that exploits exactly that bidirectionality — every prediction the model makes during MLM pretraining is allowed to use the full surrounding context, both preceding and following tokens, because nothing about the encoder's attention mechanism prevents it. A decoder-only architecture's causal mask, by contrast, restricts every position to attending only to what came before it, which is precisely what makes autoregressive generation possible — and precisely why MLM does not fit a decoder-only model at all. You cannot train a model to "predict a masked token using context on both sides" if the architecture itself structurally forbids looking at what comes after the current position; the training objective and the architecture's attention pattern have to agree, and MLM agrees with the encoder's bidirectional pattern specifically.

Why this matters for what an encoder foundation model is actually good at

The consequence of this pretraining choice shows up directly in what the resulting model is naturally suited for downstream. A model pretrained with MLM has learned to build a rich, context-aware representation of every token in an input, informed by the entire sequence surrounding it — which is exactly the property useful for classification, extraction, and embedding tasks, where the goal is understanding an entire given input rather than generating new text one token at a time. It is not, by itself, naturally suited to open-ended text generation, because nothing about its pretraining ever asked it to produce a coherent continuation of a sequence it had not yet seen the end of — that is the decoder-only family's causal language modeling objective's job, not MLM's. M1-03's architecture-to-objective pairing and this lesson's MLM mechanics are two views of the same underlying fact: the training objective a foundation model was pretrained with is not an incidental detail, it determines what kind of foundation model you end up with and what it is naturally good at once training finishes.

An encoder foundation model pretrained with MLM at real scale is exactly the kind of asset this entire module's toolkit exists to compress. Pretraining is expensive and typically happens once; deployment happens repeatedly, across many downstream tasks and many different hardware and latency budgets, and a model sized for a research lab's pretraining cluster is very often not sized for the constrained environment a specific production deployment actually has available. This is the structural reason objective 4.7 pairs "how encoder foundation models are trained" with "the levers that shrink them" in a single objective rather than as two unrelated topics — the pretraining story explains why the model exists in its original, typically large, form, and the shrinking story explains what happens to it on the way to actually running somewhere with a real memory and latency budget.

Every lever this module has covered so far applies to an encoder foundation model exactly as it would to any other trained model, because none of the levers care what pretraining objective produced the weights they are operating on — they care only about the weights, the architecture's shape, and the runtime behavior at inference. That universality is worth stating plainly before walking through each lever specifically: quantization does not need to know whether a model was pretrained with MLM or causal language modeling to reduce its numeric precision; pruning does not need to know either, to remove low-contribution weights; and distillation's teacher-student mechanism works whether the teacher is an encoder or a decoder, because what transfers is the teacher's learned output distribution, not anything specific to one architecture family.

M4-01 covers PTQ, QAT, and GPTQ in full — the mechanics do not change for an encoder model, and this lesson does not re-derive them. What is worth stating specifically here is how the choice among the three plays out for a pretrained encoder like BERT rather than for a decoder-only generative model.

A BERT-class encoder deployed for a downstream classification or extraction task is very often serving requests at a fixed, moderate precision target — INT8 is a common production choice for encoder-style models used as classifiers, rerankers, or embedding generators, because the accuracy bar for many of these tasks tolerates INT8's typical accuracy cost comfortably. PTQ's calibration-only, no-retraining path is frequently sufficient here: calibrate on a representative sample of the downstream task's actual inputs, derive scale factors, and deploy — no training budget required, exactly M4-01's PTQ niche. Where a team pushes to INT4 for a more aggressive memory target and the accuracy bar is tighter, M4-01's QAT-versus-GPTQ tradeoff applies without modification: QAT if a training budget and labeled data exist and maximum accuracy at low precision matters most, GPTQ if neither exists but the accuracy bar still needs to be met. Nothing about the model being an MLM-pretrained encoder changes which of the three techniques fits which constraint — the decision rule from M4-01's worked example transfers directly.

This is the lever with the most direct historical connection to encoder foundation models specifically, because M4-02's canonical worked example — DistilBERT — is a distilled encoder foundation model. DistilBERT's teacher is BERT, an encoder pretrained with exactly the MLM objective this lesson opened with, and its student is a smaller encoder in the same family, trained to reproduce the teacher's behavior using the teacher's soft outputs as a richer training signal than hard labels alone. M4-02's trio — roughly 40% smaller, about 60% faster, about 97% of the teacher's performance retained — is therefore not merely an example distillation happens to use; it is the specific, measured result of distilling exactly the kind of model this lesson is about.

The practical implication for a team deploying an encoder foundation model is that distillation is very often the first lever reached for, precisely because a validated, well-documented recipe already exists for this exact model family. A team facing a BERT-class model too slow or too large for a deployment target does not need to invent a distillation approach from first principles — DistilBERT's own recipe (halve the transformer layer count, train the resulting student against the teacher's soft outputs) is a proven starting point, with M4-02's trio setting a realistic expectation for what a comparable result might look like on a similarly-sized encoder, though M4-02 was explicit that the exact figures for a different teacher, task, or student depth are their own measurement rather than a guaranteed repeat.

M4-03 established the structured-versus-unstructured distinction and the specific 2:4 sparsity pattern that maps to Tensor Core acceleration, none of which depends on what the model was pretrained to do. An encoder foundation model's weight matrices — the attention projections and feed-forward layers inside each transformer layer — are pruned by exactly the same criteria M4-03 described: magnitude-based ranking as the simplest baseline, with the same structured-versus-unstructured tradeoff governing whether the resulting sparsity converts into a real speedup on the deployment hardware.

What is specific to an encoder deployment scenario is which combination tends to get reached for. A BERT-class model serving as a reranker or a classifier at high query volume is a latency-sensitive, throughput-sensitive deployment exactly like the ones M4-03's section on pairing 2:4 sparsity with INT8 described — and that documented TensorRT path (structured 2:4 sparsity plus INT8 quantization, compounding rather than substituting) applies to an encoder foundation model with no modification needed, because nothing about that combination's mechanism cared whether the underlying model was pretrained with MLM or with a causal objective. Fine-tuning after pruning, M4-03's standard recovery step, applies here too — an encoder pruned and then fine-tuned on its downstream classification or extraction task recovers accuracy the same way any pruned model does, by letting the surviving weights adjust to the removed ones' absence.

Constructed scenario, illustrating how this module's levers stack for one concrete deployment rather than reporting a measured result. A team has a BERT-base-class encoder, pretrained with MLM, fine-tuned as a cross-encoder reranker for a search pipeline, at 110 million parameters, FP32, and needs it to serve reranking requests at high query volume with a tight latency budget.

Starting point: 110,000,000 parameters, FP32 (4 bytes each)
Weight memory: 110e6 x 4 bytes = 440 MB

Step 1 — distillation, following DistilBERT's recipe. The team has the original BERT teacher and a training budget, and applies M4-02's recipe: halve the transformer layer count, train the resulting student against the teacher's soft outputs on the reranking task.

Applying M4-02's trio as an expectation, not a guarantee:
  ~40% smaller: 110e6 x (1 - 0.40) = 66,000,000 parameters
  Weight memory at FP32: 66e6 x 4 bytes = 264 MB

Step 2 — structured 2:4 sparsity, applied to the distilled student's weight matrices. Following M4-03's pattern, the team prunes to a 2:4 structured pattern, keeping exactly two of every four consecutive weight values.

Nonzero weights after 2:4 sparsity: 66e6 x 0.50 = 33,000,000
(exact count depends on group alignment; treated as ~50% here for illustration)

Step 3 — INT8 quantization, paired with the 2:4 sparsity from step 2, following M4-03's documented TensorRT combination.

33e6 nonzero weights at INT8 (1 byte, plus small position overhead
for the sparse encoding, ~1.25 bytes effective per nonzero entry):
  33e6 x 1.25 bytes ≈ 41.25 MB

Step 4 — fine-tune the pruned, quantized student on the reranking task, recovering accuracy per M4-03's standard recovery step, and re-run the evaluation set, per the discipline objective 4.2 demands throughout this module.

Final footprint: ~41.25 MB, versus 440 MB at the starting point
Overall reduction: roughly 90.6% smaller than the original FP32 BERT-base checkpoint

Read the four steps as a sequence of independent, compounding decisions rather than one big undifferentiated "make it smaller" action. Distillation changed the architecture — a genuinely smaller network, following M4-02's recipe. Structured 2:4 sparsity then changed which of that smaller network's weights survive, in a pattern that maps to hardware acceleration per M4-03. INT8 quantization then changed how many bits represent each surviving weight, per M4-01's mechanics applied inside the documented TensorRT pairing M4-03 named. None of these three steps substituted for either of the others — each addressed a different axis of the model's footprint, exactly as this module's earlier lessons established individually, and the roughly 90% combined reduction is the visible result of stacking three independent levers rather than any single one of them doing all the work.

Not every deployment scenario justifies stacking all three levers, and the choice of which to skip is itself a decision this module's material equips you to make. Constructed scenario. A different team has the same BERT-class reranker, but their constraint is different: no training budget exists at all, and the deployment target is general-purpose CPU inference with no confirmed sparse-matmul or Tensor Core support.

Walking the same three levers against this different constraint: distillation is ruled out immediately, because M4-02 established that distillation requires training a new student model from the start — no training budget means no distillation, full stop, regardless of whether a teacher model exists. Structured 2:4 sparsity is a poor fit too, not because it is impossible to apply, but because M4-03 was explicit that 2:4 sparsity's entire value proposition is Tensor Core acceleration — without confirmed sparse hardware support, the pattern's guaranteed speedup simply does not materialize, leaving only a memory-storage benefit that unstructured pruning could deliver just as well with fewer constraints on what gets removed. What remains that actually fits this constraint is PTQ from M4-01: no training budget required, no hardware-specific sparse support assumed, a calibration pass against representative reranking queries, and a deployable INT8 (or, if the accuracy bar allows, even lower-precision) model at the end of it.

The pattern to extract, and the one this module's closing lesson is built to leave you with: this module's levers are not a checklist to apply exhaustively regardless of constraint. Each one has a niche defined by what resources exist (a training budget, a teacher model, confirmed hardware support) and what is being optimized (parameter count and architecture, which weights survive, or how many bits each weight uses). Reading a scenario correctly means identifying which of those resources and targets are actually in play, not defaulting to "apply every lever this module covered."

Two axes cut across this whole table and are worth holding as the module's final, single-sentence summary. First, every lever here is a bargain — memory, latency, or both, traded for some form of cost (accuracy, in most cases; a real information-access cost, for streaming attention; a portability cost, for TensorRT's architecture-specific engines) — which is exactly the framing objective 4.2 insists on for the whole domain. Second, none of these levers substitute for each other, because each operates on a genuinely different property of the model or its runtime: what the weights are, how many bits represent them, which of them survive, how much history is ever computed against, and how the whole graph executes on one specific piece of hardware. A scenario question in this domain is, almost without exception, testing whether you can locate which of these six properties is actually the one under constraint.

Model Optimization is Domain 4 of the NCP-GENL blueprint, at 17% — the single largest domain on the exam, ahead of Fine-Tuning and Evaluation and every other domain in the ten-domain structure. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) frames objective 4.7 as the domain's own synthesis point: encoder foundation model training via MLM, deliberately paired with the shrinking levers covered everywhere else in this module, rather than left as an isolated architecture fact. That pairing is the reason this lesson closes the module rather than opening it — every lever it references (M4-01 through M4-05) needed to already exist before this lesson could meaningfully apply them to a concrete scenario.

Expect the question to arrive in these shapes:

• A pretraining-objective identification item. "Which training objective predicts randomly hidden tokens using bidirectional context?" with masked language modeling as the keyed answer, distinguished from causal language modeling's left-to-right, unidirectional setup.
• An architecture-objective matching item. A scenario pairs an architecture family with a training objective, and asks whether the pairing is coherent — MLM with an encoder is coherent; MLM with a decoder-only causal architecture is not, because the objective requires bidirectional context the architecture structurally forbids.
• A synthesis scenario naming multiple constraints at once. A description states a pretrained encoder foundation model, a specific deployment target, and a resource constraint (a training budget present or absent, confirmed sparse-hardware support or not), and asks which combination of this module's levers fits — testing the same constraint-matching skill section 7's worked example builds.
• A "why is this here" conceptual item. A question asks why encoder pretraining and shrinking levers are grouped under one objective, testing whether the pretraining-to-deployment pipeline is understood as one continuous story rather than two disconnected topics.

What the distractors typically look like

Expect MLM described as compatible with a decoder-only architecture, or causal language modeling described as compatible with bidirectional context — both scramble the exact architecture-objective pairing M1-03 and this lesson both establish. Expect a scenario offering "just distill it" or "just quantize it" as a complete answer to a multi-constraint deployment question, when the actually-correct answer requires matching each stated constraint to a specific lever, as section 7 demonstrates. And expect an option treating this module's six levers as interchangeable synonyms for "make it smaller and faster," when the entire tested skill across this domain is knowing which lever addresses which specific, named constraint.

Masked language modeling hides a fraction of an input's tokens and trains the model to predict each hidden token's identity using context from both directions — the tokens before and after the masked position — which requires an architecture with no causal restriction on what a position can attend to, making it the encoder-appropriate objective. Causal language modeling (M2-03 covers this in full for the decoder side) instead trains a model to predict the next token in a sequence using only the tokens that came before it, one position at a time, left to right — the objective that makes autoregressive text generation possible, and the one a decoder-only architecture's causal attention mask is specifically built to support. The two are not interchangeable and do not fit each other's architecture family: MLM needs bidirectional visibility a decoder's causal mask forbids, and CLM's left-to-right prediction task would be trivially solvable (and useless as a training signal) for a model that could already see the future tokens an encoder's bidirectional attention exposes.

Yes, and section 6's worked example is exactly this combination applied to one concrete BERT-class reranking scenario: distillation first (changing the architecture to a genuinely smaller network), structured 2:4 sparsity second (changing which of that smaller network's weights survive, in a hardware-exploitable pattern), and INT8 quantization third (changing how many bits represent each surviving weight). The three levers compound because each operates on an independent property of the model — this is the same "independent, compounding axes" relationship M4-02 and M4-03 established individually, now demonstrated as a full three-lever stack. The combined result still needs its own accuracy re-measurement on a held-out evaluation set, exactly as every earlier lesson in this module insisted, because each lever changes the starting point the next one operates on and the effects do not simply sum without checking.

Work through each item before checking the answer key. Every wrong option describes something real about some model or lever covered in this module — the task is matching the described scenario to the correct concept, not spotting an obviously invented distractor.

1. Which training objective predicts a hidden token using context from both directions?
   • A. Causal language modeling.
   • B. Masked language modeling.
   • C. Knowledge distillation.
   • D. Post-training quantization.
2. Why does masked language modeling not fit a decoder-only architecture?
   • A. Decoder-only models cannot process text at all.
   • B. A decoder's causal attention mask structurally forbids attending to tokens that come after the current position, which MLM's bidirectional prediction requires.
   • C. MLM requires labeled data, which decoders cannot use.
   • D. Decoder-only models have no attention mechanism.
3. DistilBERT is best understood, in the context of this lesson, as:
   • A. A generic illustration of distillation with no particular connection to encoder foundation models.
   • B. The canonical, validated distillation recipe specifically for shrinking an MLM-pretrained encoder foundation model.
   • C. A pruning technique applied to BERT.
   • D. A quantization scheme for encoder models.
4. A team has no training budget and no confirmed sparse-hardware support, and needs to shrink a BERT-class encoder. Which lever fits best?
   • A. Knowledge distillation.
   • B. Structured 2:4 sparsity.
   • C. PTQ.
   • D. QAT.
5. Which three properties do quantization, distillation, and pruning respectively change?
   • A. All three change the same property: overall model size.
   • B. Numeric precision of existing weights; architecture and parameter count; which weights survive.
   • C. Serving infrastructure; training data; evaluation metrics.
   • D. GPU architecture target; batch size; sequence length.
6. Why do quantization, distillation, and pruning all apply to an encoder foundation model exactly as they would to a decoder-only model?
   • A. They do not — these levers only apply to decoder-only models.
   • B. None of the three levers depend on the pretraining objective that produced the model's weights; they operate on the weights and architecture shape directly.
   • C. Encoder and decoder models have identical architectures.
   • D. MLM and causal language modeling are the same objective.
7. What is the risk of applying every lever in this module to a model regardless of the stated constraint?
   • A. There is no risk; more levers always produce a better result.
   • B. A lever can fail to deliver its expected benefit, or be inapplicable outright, if the resource it needs (a training budget, a teacher model, confirmed sparse-hardware support) is not actually available.
   • C. Applying multiple levers always cancels out their individual benefits.
   • D. Only one lever is ever legally combinable with another.
8. Why does a multi-lever stack (distillation, then pruning, then quantization) need its own accuracy re-measurement rather than summing each lever's individually-reported accuracy cost?
   • A. Accuracy costs always sum exactly, so re-measurement is a formality.
   • B. Each lever changes the starting point the next one operates on, so the combined effect cannot be assumed to equal the sum of individually-measured costs.
   • C. Only the first lever in a stack affects accuracy.
   • D. Re-measurement is required only when quantization is the first lever applied.

Answers

1. B. This is MLM's identity statement from section 1: bidirectional context is exactly what the objective requires and exactly what a causal decoder mask forbids.
2. B. This is the architecture-objective coherence point section 1 and M1-03 both establish: causal masking and bidirectional prediction are structurally incompatible.
3. B. Section 4 makes this connection explicit: DistilBERT's teacher (BERT) is itself an MLM-pretrained encoder, making it the validated recipe for exactly this scenario, not a generic example.
4. C. This mirrors section 7's worked comparison: no training budget rules out distillation, no confirmed sparse-hardware support weakens 2:4 sparsity's value proposition, and PTQ needs neither.
5. B. This is section 8's synthesis table: precision (quantization), architecture/parameter count (distillation), and weight survival (pruning) are three independent axes.
6. B. Section 2 states this directly: none of the shrinking levers care what pretraining objective produced the weights — they operate on the weights and architecture shape, not the training history.
7. B. This is the exact lesson section 7's worked example is built to teach: match each lever to the resources actually available rather than applying all of them by default.
8. B. This is the closing discipline restated from M4-02 and M4-03 and reiterated in the "Can quantization, distillation, and pruning all be applied" FAQ above: each lever changes the starting point for the next, so the stack's real accuracy cost must be measured, not summed.

• Masked language modeling predicts hidden tokens using bidirectional context, which is exactly why MLM is an encoder-appropriate objective and does not fit a decoder-only, causally-masked architecture.
• Objective 4.7 pairs encoder pretraining with the shrinking levers deliberately — an encoder foundation model pretrained at scale is precisely the kind of asset quantization, distillation, and pruning exist to compress for deployment.
• DistilBERT (M4-02) is not merely an example of distillation — it is the canonical, validated result for shrinking exactly this kind of model, an MLM-pretrained encoder foundation model.
• Quantization, distillation, and pruning are architecture-agnostic: none of them care what pretraining objective produced the weights they operate on, and all three apply to an encoder model exactly as M4-01 through M4-03 describe for any model.
• The levers compound because each operates on an independent axis — architecture and parameter count (distillation), which weights survive (pruning), and how many bits represent each surviving weight (quantization) — not because any one of them does the whole job alone.
• Matching a lever to a stated constraint (a training budget, a teacher model, confirmed sparse-hardware support) is the actual tested skill, not applying every lever regardless of what resources exist.
• Model Optimization is 17% of the NCP-GENL blueprint, the single largest domain, and this closing lesson's synthesis — six levers, six different properties of a model, one recurring constraint-matching discipline — is the single idea the rest of the domain's scenario questions test in different costumes.

This module has been about making an already-trained model smaller, faster, and cheaper to run, without changing what it fundamentally knows how to do. The next module turns to the opposite kind of intervention: deliberately changing a model's weights to teach it something new — a skill, a style, a behavior — rather than compressing what it already learned. The same discipline this module built, measuring a tradeoff rather than assuming one, carries forward directly: fine-tuning trades compute and data for capability, exactly as this module's levers traded memory or latency for accuracy risk, and the next module's central distractor pair turns on knowing precisely which piece of the training setup each fine-tuning method needs and which it can do without.

Next: M5-01 opens Module 5, Fine-Tuning, with parameter-efficient fine-tuning — LoRA, adapters, and P-tuning — and the property that makes LoRA specifically add no inference latency, unlike bottleneck adapters, once the low-rank matrices it trains are merged back into the frozen base model this module has spent its whole scope learning to shrink.