Multimodal Transfer Learning: Pretrained Encoders, Full Fine-Tuning vs. PEFT

Identity statement: transfer learning reuses a model pretrained on a large dataset and adapts it to a new task with less data and compute than training from scratch would require. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states this directly, and names the specific multimodal instance: "In multimodal work you often start from strong pretrained encoders (e.g., a vision backbone plus a text/LLM backbone, or a CLIP encoder) and fine-tune or add lightweight adapters."

When it matters: any scenario describing a multimodal model's starting point, or comparing the cost and outcome of building a component from scratch versus adapting an existing one.

The reasoning behind treating this as a default rather than an occasional shortcut is straightforward once stated plainly: a vision backbone pretrained on a very large, general image dataset has already learned to detect edges, textures, shapes, and a wide range of general visual patterns — exactly the hierarchical feature-building process M1-06 described convolutional stacking as producing. A text/LLM backbone pretrained on a very large text corpus has already learned general language structure, common patterns of usage, and broad world knowledge encoded in its weights. A CLIP encoder has already learned a shared, general-purpose text-image embedding space, the mechanism M1-08's contrastive-loss section covered. Starting a new multimodal project by throwing away all of that and training every component from a random initialization discards a large amount of already-learned structure that would otherwise need to be relearned, at real cost in data and compute, for essentially no benefit — which is exactly why the default instinct for a new multimodal project is to reach for an existing pretrained encoder first, and to consider training from scratch only when no suitable pretrained option exists for the modality or domain in question.

Identity statement: full fine-tuning updates every weight in a pretrained model during adaptation to a new task; parameter-efficient methods (adapters, LoRA) update only a small subset of parameters, leaving the bulk of the pretrained weights frozen. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this distinction directly.

L1 — Intuition

Full fine-tuning treats the pretrained model as a good starting point for a fresh, comprehensive training run — every weight is free to move, and the training loop from M1-05 updates the entire network, just starting from an already-informed position rather than a random one. Parameter-efficient adaptation instead treats the pretrained model as mostly finished and mostly correct already, adding a small number of new, trainable parameters — or unfreezing only a tiny fraction of the existing ones — while leaving the vast majority of the pretrained weights untouched, on the premise that most of what the pretrained model already knows should be preserved exactly as it is.

L2 — Mechanism

Full fine-tuning's mechanism is the ordinary training loop, applied to a network that happens to start from pretrained rather than random weights: backpropagation computes gradients for every parameter, and gradient descent updates every one of them. The only difference from training from scratch is the starting point; the update mechanism itself is unchanged.

Parameter-efficient fine-tuning (PEFT) methods change what gets updated, not how updates are computed. Adapters insert small, new trainable modules — typically compact bottleneck layers — at specific points inside the pretrained network's architecture, and only those newly inserted modules' weights (plus, sometimes, a small number of the original network's own parameters) are trained; the vast bulk of the original pretrained weights are frozen and never receive a gradient update at all. LoRA (low-rank adaptation) instead represents the update to certain existing weight matrices as a product of two much smaller matrices, training only those two small matrices while leaving the original, large weight matrix frozen and unmodified — the effective change to the model's behavior comes from adding this small, learned low-rank update on top of the frozen original weights, rather than modifying the original weights directly.

L3 — The exam-relevant tradeoff, stated precisely

The tradeoff is not "PEFT is strictly better" or "full fine-tuning is strictly better" — each makes a different bet. Full fine-tuning has more flexibility to adapt every part of the network to the new task, which can matter when the new task is substantially different from whatever the pretrained model originally learned, but it requires updating and storing gradients for every parameter, at real memory and compute cost, and carries a real risk of catastrophic forgetting — overwriting useful pretrained knowledge that the new, narrower fine-tuning data does not reinforce, particularly when the fine-tuning dataset is small relative to what the model originally learned from. Parameter-efficient methods update far fewer parameters, at dramatically lower memory and compute cost, and largely sidestep catastrophic forgetting because the original pretrained weights never change — but they have less flexibility to fundamentally alter the pretrained model's behavior, since most of the network is frozen by design. Which is appropriate depends on how much the new task genuinely diverges from what the pretrained model already knows, and how much data and compute the project can actually spend.

Why LoRA's memory savings come specifically from parameter count during training, not just at inference

It is worth being precise about exactly where LoRA's efficiency comes from, since "it's more efficient" alone does not distinguish LoRA's specific mechanism from adapters' or from any other parameter-reduction idea. The memory a training run needs scales with the number of trainable parameters, not merely the model's total parameter count, because the optimizer (recall M1-05's optimizer-state framing) has to track additional per-parameter bookkeeping — momentum terms, adaptive learning-rate statistics — for every parameter it is actually updating, and backpropagation has to store intermediate values needed to compute each trainable parameter's gradient. A LoRA adaptation might introduce two small matrices representing, say, a few million trainable parameters layered on top of an underlying model with several billion frozen parameters; the optimizer only needs to track state for those few million, not the several billion, which is where the bulk of LoRA's training-time memory savings actually comes from — the frozen billions of parameters still occupy memory to store and to compute the forward pass through, but they need none of the additional gradient and optimizer-state bookkeeping that trainable parameters require.

Partial fine-tuning: a middle ground worth naming

Between full fine-tuning (every weight trainable) and PEFT (a small number of new or reparameterized weights trainable) sits a third, simpler option worth naming even though the source material's own framing centers on the first two: partial fine-tuning, which freezes most of a pretrained network's original layers outright and unfreezes only a subset of the existing layers — commonly the last few, closest to the output — for ordinary full-precision training. This differs from adapters and LoRA in that it introduces no new parameters or reparameterized update structure at all; it simply trains a subset of the model's own original weights and leaves the rest exactly as pretrained. Partial fine-tuning sits closer to full fine-tuning's memory profile than adapters or LoRA's (since the unfrozen layers, however few, are ordinary full-size weight matrices), but it shares PEFT's basic premise that most of the pretrained network's knowledge is worth preserving unchanged. Recognizing this as a genuinely distinct third option, rather than collapsing everything that is "not full fine-tuning" into "PEFT," is useful for reading a scenario precisely — a description of "we only unfroze the last two layers and trained those directly" is partial fine-tuning, not PEFT in the adapters/LoRA sense the source material names, even though both share the family resemblance of updating fewer than all of a model's parameters.

M1-07 established that a multimodal model's architecture is nonsequential by necessity, built from separate branches, one per modality, that eventually merge. Transfer learning interacts with that branch structure in a specific way worth making explicit: a multimodal model does not have to make one single, uniform transfer-learning decision across its entire architecture — each branch can independently be pretrained-and-frozen, pretrained-and-fine-tuned, or trained from scratch, and real multimodal projects frequently mix these choices across branches within the very same model.

Consider a model fusing a pretrained vision backbone with a from-scratch audio branch — exactly the asymmetric setup M1-09's training-stability lesson used as its running example. The vision branch might use a pretrained encoder with parameter-efficient adaptation (since strong, general vision backbones are widely available and the task likely does not need to relearn basic visual structure), while the audio branch, if no comparably strong pretrained audio encoder exists for the specific kind of audio in question, might be trained entirely from scratch. This is not an inconsistency to resolve — it is a direct, correct application of the same underlying decision framework from section 2, applied independently to each branch based on that branch's own data volume, task divergence, and the availability of a suitable pretrained starting point for that specific modality.

This branch-level independence is also precisely why M1-09's per-branch learning-rate framing exists: a pretrained-and-frozen-mostly branch under PEFT typically needs a very different, usually much smaller, learning rate applied to its small number of trainable parameters than a from-scratch branch needs applied to its entire parameter set, and a single global learning rate across the whole fused model would be simultaneously wrong for both branches at once — the exact multimodal training-stability risk that lesson named directly.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names three specific kinds of pretrained encoder a multimodal project typically starts from, and recognizing each by name is worth doing explicitly rather than treating "a pretrained encoder" as one undifferentiated category.

A multimodal project frequently combines more than one of these — a vision backbone for the image branch, a text/LLM backbone for the text branch, both feeding into a fusion point trained (fully or with PEFT) for the specific task at hand — or, when the task benefits specifically from a shared embedding space rather than two separately-aligned encoders, starts directly from a CLIP encoder instead of assembling the vision and text backbones separately. The choice among the three is itself task-dependent: a task needing fine-grained, task-specific visual detail benefits from a strong, general vision backbone; a task needing cross-modal comparison (does this image match this text) benefits from starting directly from CLIP's already-aligned shared space, since building that alignment from a separately pretrained vision backbone and text backbone would require re-solving the alignment problem CLIP's pretraining already solved.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names three specific benefits, and each traces back to a distinct mechanism worth separating rather than treating as one vague "it's better" claim.

Faster convergence. A pretrained encoder's weights already encode useful structure, so the training loop starts from a position much closer to a good solution than a random initialization would — fewer gradient-descent steps are needed to reach comparable performance, because the model is not spending early training simply discovering basic structure (edges, common word patterns) that the pretrained weights already captured.

Better accuracy with limited labeled data. A task with a small labeled dataset often cannot, on its own, teach a from-scratch model the general structure a large pretrained dataset already taught a pretrained encoder — transfer learning lets a small, task-specific dataset be used for what it is actually suited for (adapting general knowledge to a specific task) rather than for what it is poorly suited for (teaching general structure from very little data).

Lower energy cost. Training a large model from scratch requires substantially more compute — and therefore more energy — than adapting an already-pretrained model, particularly with parameter-efficient methods that update only a small fraction of the total parameters. This connects directly to Domain 5's efficiency framing later in this course: reusing a pretrained encoder is a direct, immediate efficiency win, independent of any later precision or pruning technique layered on top.

Project A. A team has 500,000 labeled examples for a new multimodal fraud-detection task, believes the task requires the model to learn genuinely new visual patterns specific to fraudulent product photos that a general vision backbone would not have encountered in its original pretraining, and has ample GPU budget for a multi-week training run.

Data volume: large (500,000 examples)
Task divergence from pretraining: high (novel, task-specific visual patterns)
Compute budget: ample

-> FULL FINE-TUNING is the more defensible choice. Enough data exists
   to fine-tune the entire network without a severe catastrophic-
   forgetting risk, the task's divergence from general pretraining
   benefits from the full flexibility of updating every weight, and
   the compute budget can absorb the higher cost.

Project B. A different team has 2,000 labeled examples for a narrower task — classifying a specific product category's images using an existing, well-suited vision backbone — and needs to deploy quickly on a constrained compute budget, with no reason to believe the task requires fundamentally new visual patterns beyond what the pretrained backbone already captures.

Data volume: small (2,000 examples)
Task divergence from pretraining: low (narrow adaptation of existing knowledge)
Compute budget: constrained

-> PARAMETER-EFFICIENT ADAPTATION (adapters or LoRA) is the more
   defensible choice. The small dataset carries real catastrophic-
   forgetting risk under full fine-tuning, the task does not appear
   to need the full flexibility of updating every weight, and the
   constrained compute budget favors a method that trains far fewer
   parameters.

This is a constructed scenario with illustrative project descriptions, not measurements from real deployments. The generalizable reading: the decision is not a fixed rule ("always use PEFT," "always fully fine-tune") but a function of three factors evaluated together — how much labeled data is available, how far the new task diverges from what the pretrained encoder already knows, and how much compute the project can spend. A scenario naming any two of these three factors is usually supplying enough information to reason toward the correct choice.

⭐ THE EARNED INSIGHT Full fine-tuning and parameter-efficient adaptation are not a strength-versus-weakness comparison where one method is simply "worse." They encode two different bets about how much of a pretrained model's existing knowledge should be trusted versus overwritten. Full fine-tuning bets that the new task has enough data and enough genuine divergence from the original pretraining to justify letting every weight move, accepting the forgetting risk that comes with it. Parameter-efficient adaptation bets that most of what the pretrained model already knows is still correct and worth preserving exactly as-is, and that the new task's needs can be captured by a small number of additional, targeted parameters layered on top. Neither bet is universally correct — the right one depends on reading the specific task's data volume and divergence from pretraining honestly.

A team fully fine-tunes a pretrained vision-language model — originally trained on a broad, general image-caption dataset — on a narrow, 800-example dataset of medical X-ray images and captions, updating every weight for 30 epochs.

Before fine-tuning: model correctly captions a wide variety of general
images ("a dog on a beach," "a red sports car," "a plate of food").

After 30 epochs of full fine-tuning on 800 X-ray examples:
  - Captions X-ray images accurately (the intended improvement).
  - Given a general, non-medical image, produces a nonsensical or
    medically-flavored caption unrelated to the actual image content
    -- the model has partially "forgotten" its original general
    captioning ability.

Constructed scenario, illustrative outcome. This is catastrophic forgetting in its most direct form: because every weight was free to move, and the fine-tuning dataset (800 examples, all X-rays) contained no examples reinforcing the model's original general-image knowledge, gradient descent had no reason to preserve that knowledge — the loss function it was minimizing said nothing about general images at all, only about X-ray captioning accuracy. A parameter-efficient approach on the same 800-example dataset would have left the vast majority of the original pretrained weights frozen, making this kind of broad, general-capability forgetting structurally much less likely. Those frozen weights are simply never touched during the adaptation, regardless of how narrow or small the new dataset is.

The mistakes table below collects this failure alongside the other traps this lesson names.

Every row maps a specific symptom to a specific fix. Exam weight explains why the underlying decision framework matters this much.

Core Machine Learning and AI Knowledge carries 20% exam weight, and [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) frames pretrained encoders as the explicit multimodal default — "you often start from strong pretrained encoders... rather than training from scratch." This lesson's material also connects forward to two later modules directly: M5-05 in Performance Optimization returns to transfer learning specifically as an efficiency technique, and the parameter-efficient methods named here (adapters, LoRA) recur by name across this exam's broader treatment of adapting large pretrained models economically.

The question tends to arrive in a small number of recognizable shapes.

1. Naming the default starting point. "What is the typical starting point for a new multimodal model?" with the keyed answer naming a pretrained encoder (vision backbone, text/LLM backbone, or CLIP), against distractors describing training from scratch as standard.
2. Full fine-tuning versus PEFT, matched to a described scenario. A scenario states a data volume, a task-divergence level, and a compute budget, and asks which adaptation approach is more appropriate.
3. Catastrophic forgetting, recognized from a symptom. A scenario describes degraded general capability after fine-tuning on a narrow dataset, asking for the cause — the keyed answer names catastrophic forgetting from full fine-tuning on limited, narrow data.
4. Benefit-to-mechanism matching. "Why does transfer learning reduce energy cost?" with the keyed answer connecting reduced training compute to the specific benefit named, rather than a vague "it's more efficient" non-answer.

What the distractors typically look like

The reliable distractor families: describing training from scratch as the multimodal norm rather than the exception; asserting one adaptation method (full fine-tuning or PEFT) is unconditionally superior regardless of data volume or task divergence; and describing catastrophic forgetting as an inherent property of pretrained models generally, rather than a consequence specifically tied to updating every weight on narrow, limited fine-tuning data.

Favor parameter-efficient methods (adapters, LoRA) when the available labeled data for the new task is limited relative to what the model originally learned from, when the new task does not appear to require fundamentally new capability beyond what the pretrained encoder already provides, or when compute and memory budget are constrained. Under these conditions, updating every weight via full fine-tuning carries a real risk of catastrophic forgetting — overwriting useful general knowledge the narrow fine-tuning data does not reinforce — while parameter-efficient methods largely sidestep that risk by leaving the bulk of the pretrained weights frozen, at a fraction of the compute and memory cost. Reach for full fine-tuning instead when the new task genuinely diverges from the pretrained encoder's original knowledge and enough data and compute are available to support updating the entire network without excessive forgetting risk.

Because a pretrained vision backbone, text/LLM backbone, or CLIP encoder has already learned general, broadly useful structure — visual feature extraction, language patterns, or a shared text-image embedding space — from a very large amount of data, and discarding that already-learned structure to train from a random initialization means re-learning it from whatever smaller, task-specific dataset the new project has available, at real cost in data, compute, and time for no corresponding benefit. Training from scratch only makes sense when no suitable pretrained encoder exists for the modality or domain in question, which is the exception rather than the rule for the common modalities (text, image, and increasingly audio) this exam's multimodal material covers.

Yes, and in practice this is the common case rather than an unusual arrangement. A multimodal model's separate modality branches are architecturally independent up to their fusion point, so each branch can independently be pretrained-and-frozen, adapted with PEFT, fully fine-tuned, or trained entirely from scratch, based on that specific branch's own data volume, task divergence, and whether a suitable pretrained encoder even exists for that modality. A common real pattern pairs a strong pretrained vision backbone under parameter-efficient adaptation with a from-scratch branch for a less-standard modality lacking an equally mature pretrained option, and the resulting asymmetry in each branch's trainable-parameter count is precisely why per-branch learning-rate treatment, covered in M1-09, matters as much as it does — a single global learning rate would be simultaneously wrong for a nearly-frozen PEFT branch and a fully-trainable from-scratch branch trained side by side in the same model.

• Starting from a pretrained encoder — a vision backbone, a text/LLM backbone, or a CLIP encoder — is the multimodal default, not an occasional shortcut; training from scratch is the exception.
• Full fine-tuning updates every weight; parameter-efficient methods (adapters, LoRA) update only a small subset, leaving most of the pretrained weights frozen.
• The choice between them depends on data volume, task divergence from the pretrained encoder's original knowledge, and compute budget — neither method is universally superior.
• Catastrophic forgetting is the specific risk of full fine-tuning on narrow, limited data — overwriting general pretrained knowledge the fine-tuning data does not reinforce; parameter-efficient methods largely sidestep it by freezing most weights.
• Transfer learning's three named benefits — faster convergence, better accuracy with limited data, and lower energy cost — each trace to a distinct mechanism: reusing already-learned structure, needing less task-specific data to adapt rather than learn from scratch, and requiring less total compute.
• A multimodal model's separate branches can each independently use a different transfer-learning strategy — mixing a PEFT-adapted pretrained branch with a from-scratch branch in the same model is a common, correct pattern, not an inconsistency.
• LoRA's memory savings come from reducing the number of trainable parameters the optimizer must track, not merely from the model's total parameter count — frozen weights still occupy memory but need no gradient or optimizer-state bookkeeping.
• Partial fine-tuning is a distinct third option from PEFT — it unfreezes a subset of a model's own original layers with no new parameters introduced, rather than adding adapters or a low-rank update on top of frozen weights.
• Evaluate data volume, task divergence, and compute budget together for any described project, since all three factors jointly determine whether full fine-tuning or a parameter-efficient method is the more defensible choice.

This module has now covered how a multimodal model gets its starting weights and how it is adapted from there, whether uniformly or branch by independent branch. What remains is the question of when, architecturally, a multimodal model actually combines its separate modalities into one decision — the exact point where the pretrained-or-not branches this lesson described eventually have to meet. Next: M1-11 covers model fusion and orchestration — early, intermediate, and late fusion, and the distinction between fusing data types and coordinating agents.