LoRA vs. Adapters vs. P-Tuning: Parameter-Efficient Fine-Tuning Compared

Full fine-tuning updates every weight in the model. That is expensive in a specific, countable way: under Adam, each trainable parameter needs a gradient tensor and two optimizer-moment tensors, so training memory scales with the total parameter count, and every one of those weights is a candidate for catastrophic forgetting — a large-enough update to any given weight can degrade a capability that weight was quietly supporting, with no signal that it happened until you test for it.

Parameter-efficient fine-tuning (PEFT) is the family of methods that gets most of a full fine-tune's behavioral benefit without paying that price, by freezing the pretrained weights and training only a small, newly introduced parameter set. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the mechanism for the family plainly: full fine-tuning "updates every weight — expensive and prone to catastrophic forgetting," while "PEFT updates only a small set of new parameters while freezing the base model." That sentence is worth reading twice, because it names the one property every PEFT method shares and rules out a misreading that recurs constantly on this domain's questions.

The misreading is this: PEFT does not compress, prune, or shrink the base model in any way. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the trap directly as a common exam misconception: "LoRA reduces trainable parameters; it does not shrink the base model." The frozen weights are exactly as large after PEFT as before it — every one of them still has to sit in memory and participate in the forward pass, because the model still needs its full pretrained capability to produce a coherent output at all. What gets small is the new thing you train alongside that unchanged base: a pair of low-rank matrices, a bottleneck layer, or a handful of continuous embedding vectors. Confusing "the trainable set is small" with "the model is small" is the single most exploitable misconception in this section of the domain, and a scenario question that describes model-size reduction as PEFT's benefit is testing exactly this confusion.

Three methods dominate the professional-level treatment of this family, and each intervenes at a different point in the network:

• LoRA intervenes inside a weight matrix, adding a learned low-rank correction to what that matrix computes.
• Bottleneck adapters intervene between layers, inserting an entirely new sequential block the forward pass must run through.
• P-tuning / prompt tuning intervenes at the input, prepending learned continuous vectors to the sequence the frozen model reads.

Holding those three intervention points distinct — inside a weight, between layers, at the input — is the organizing idea for everything that follows, because it is also what predicts each method's cost at inference time, which is this lesson's central professional-depth thread.

L1 — Intuition: a correction inside the wall, a checkpoint in the hallway, or a note handed in at the door

Picture the frozen model as a building whose internal walls (its weight matrices) cannot be touched. LoRA writes a small, structured correction directly onto one wall's surface — a thin overlay that changes what that wall does to anything passing through it, without demolishing or rebuilding the wall itself. Because the overlay and the wall occupy the same physical footprint, they can eventually be fused into a single wall with the correction baked in permanently.

A bottleneck adapter is different: instead of altering a wall, it installs a small new checkpoint booth in the hallway between two rooms that already existed. Every person walking from room to room now has to stop at the booth, get processed, and continue. The original rooms are untouched, but the hallway itself is now longer, and it stays longer for every future walk-through unless someone physically removes the booth.

P-tuning does neither. It hands the building's frozen receptionist — who has fixed instructions and cannot be retrained — a note to read before every visitor's request. The note is written in a language only the receptionist's internal process understands (a continuous vector, not English), and it occupies one of the receptionist's limited attention slots for every single visit. Nothing inside the building changes at all; what changes is what gets handed in at the door.

L2 — Mechanism: where each method's new parameters live and how they act on the forward pass

LoRA targets a chosen frozen weight matrix W and introduces two small matrices, A and B, whose product B·A is added to whatever W alone would compute for a given input. The frozen matrix runs unmodified; the low-rank product runs in parallel and its output is summed with the frozen path's output before continuing to the next operation. Because B·A has the exact same shape as W, the two can later be added together as ordinary matrices — W_new = W + B·A — a step called merging. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) states the payoff of that shape match directly: LoRA "adapters can be merged into the base weights, so there is no added inference latency." Until merged, LoRA still adds a small parallel computation at inference; the zero-latency property specifically describes the merged state, not the unmerged one, and that distinction is easy to blur and worth holding onto.

Bottleneck adapters take a different shape entirely: a small feed-forward block — typically a down-projection to a narrow bottleneck dimension, a nonlinearity, and an up-projection back to the original width — inserted as a new sequential layer between two existing transformer sub-layers (commonly after the attention block and after the feed-forward block, once per transformer layer). Unlike LoRA's parallel correction, this new block sits directly in the path the forward pass must traverse. Every token, at every one of those insertion points, now has to be processed by the adapter's extra matrix multiplications before the residual stream continues onward. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) is explicit about the consequence: adapters are "small trainable modules inserted between transformer layers ... (can add inference latency if not merged)." Note the qualifier — "if not merged" — because it signals adapters are not structurally forbidden from ever being folded away, but the common bottleneck-adapter design (a nonlinearity sits between the down- and up-projection) means the two projections cannot be collapsed into a single linear term the way LoRA's product can; the nonlinearity is exactly what makes B·A-style merging mathematically unavailable to a typical adapter block.

P-tuning / prompt tuning does not touch any weight matrix at all. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) names the mechanism concisely: P-tuning and prompt tuning "learn soft prompt embeddings while keeping the model frozen." Concretely, a fixed number of trainable continuous vectors — soft prompt tokens — are learned in the model's embedding space and prepended to the sequence of ordinary token embeddings before the (fully frozen) transformer processes them. These vectors are not word embeddings that correspond to any vocabulary token; they are free parameters, optimized by ordinary backpropagation through the frozen model, that the network learns to interpret as steering context. Because they behave like extra input positions rather than a term added to any weight, this stays labelled inference beyond the source's one-line definition: the deeper mechanics of how a soft prompt is initialized or reparameterized are implementation detail the domain file does not specify, and a candidate should carry the one-line definition — continuous, learned, prepended, weights frozen — rather than a specific implementation recipe as the examinable fact.

Two properties follow immediately from "prepended to the sequence" rather than "added inside a weight": first, soft prompt tokens consume real context-window budget on every single request, the same way a few-shot example would, for as long as the model is used — there is no equivalent of merging that ever removes this cost. Second, soft prompts cannot be composed the way multiple LoRA adapters can be swapped against one base, because they are tied to specific input positions rather than to a specific weight matrix's behavior.

L3 — The exam-relevant edge case: mergeability is a property of the shape of the update, not of PEFT as a category

The professional-level distinction this domain rewards is narrower than "PEFT is efficient." It is: whether a given PEFT method's learned update has the same mathematical shape as something already in the frozen forward pass. LoRA's update is a matrix of the same shape as the weight matrix it augments, so the two can be summed into one matrix and the correction disappears as a separate computation. A bottleneck adapter's update is a nonlinear function wrapped around two projections, which has no matching shape inside the frozen network to merge into — the nonlinearity is precisely what blocks the algebraic trick LoRA relies on. P-tuning's update lives in input space, not weight space, so "merging" is not even a meaningful operation to ask for; there is no weight matrix for a sequence of extra tokens to be folded into.

This is why the honest professional-level statement is narrower than "adapters are slow and LoRA is fast." It is: LoRA is latency-neutral once merged because its update algebraically collapses into an existing matrix; a classic bottleneck adapter is not, because its nonlinearity prevents that collapse; and P-tuning's cost is not latency at all but permanent context-budget consumption, a different kind of overhead than either of the other two. A scenario question that asks "which of these adds no inference overhead under any configuration" is really asking which method's update shares the frozen network's linear-algebra shape, and only LoRA, among the three, has that property.

All figures below are a constructed scenario built to illustrate the mechanism, not a measured benchmark of any named product. Suppose a 13-billion-parameter decoder-only model with 40 transformer layers and hidden size 5,120 is being served for two customers, each needing a different task-specific behavior layered on top of the same frozen base.

Step 1 — LoRA's added compute per token, merged versus unmerged.

Per targeted matrix, unmerged LoRA at rank r=16, hidden size d=5,120:
  extra multiply-adds per token ≈ 2 x r x d = 2 x 16 x 5,120 = 163,840

Compare to the frozen matrix's own multiply-adds per token:
  d x d = 5,120 x 5,120 = 26,214,400

Unmerged LoRA overhead as a fraction of that one matrix's own compute:
  163,840 / 26,214,400 ≈ 0.62%

Merged LoRA overhead:
  0%  (the correction is now inside W_new; no separate computation exists)

Unmerged, LoRA's own added compute is already small — well under one percent of the matrix it augments — and merging removes even that. This is the arithmetic behind the domain's stated property: a merged LoRA adapter is not "very cheap," it is exactly as expensive as running the base model alone, because after merging there is no longer a separate LoRA computation to account for.

Step 2 — a bottleneck adapter's added compute per token, which cannot merge away.

Adapter design: down-projection d -> b, nonlinearity, up-projection b -> d
Bottleneck width b = 64, hidden size d = 5,120, one adapter per layer, 40 layers

Multiply-adds per adapter, per token:
  down-projection: d x b = 5,120 x 64 = 327,680
  up-projection:   b x d =    64 x 5,120 = 327,680
  total per adapter                       = 655,360

Across 40 layers (one adapter inserted per layer):
  40 x 655,360 = 26,214,400 extra multiply-adds per token

Step 3 — compare that adapter cost against the full model's per-token compute.

Rough full-model compute per token (dominant term, attention + feed-forward
across 40 layers at hidden size 5,120): on the order of several billion
multiply-adds per token for a 13B-parameter model — this course does not
quote a specific FLOPs-per-token figure for an unnamed model, so treat the
comparison qualitatively rather than as a precise ratio. ⚠️ UNVERIFIED beyond
the qualitative point that follows.

Qualitative point that is grounded, not the specific ratio:
  the adapter total (≈26.2M multiply-adds/token) is a small fraction of a
  multi-billion-multiply-add forward pass, but unlike LoRA merged, it is a
  cost that is paid on every single token, every single request, forever,
  because it cannot be algebraically folded into the frozen weights.

Step 4 — the serving-architecture consequence. Customer A's LoRA adapter, once merged, is served as an ordinary model with no distinguishable extra latency from the base — the operator could not tell from a latency trace alone that a LoRA fine-tune happened at all. Customer B's bottleneck adapter, however small its individual contribution, adds a fixed sequential cost to every layer of every forward pass for as long as it stays deployed, because there is no merge operation available to remove it. At high query volumes, that recurring, unremovable cost is exactly the number a capacity-planning exercise has to multiply by requests-per-second — and it is the reason production teams default to LoRA when the same behavioral change can be expressed either way.

⭐ THE EARNED INSIGHT

The exam's favorite inversion in this section is treating "small number of trainable parameters" as synonymous with "small inference cost." They are not the same claim. LoRA's trainable parameter count and its inference cost are both small, but for two different reasons — the parameter count is small because the rank is chosen small, and the inference cost is small (eventually zero) because the update's matrix shape allows an algebraic merge. A bottleneck adapter can have an equally small or smaller trainable parameter count than LoRA and still impose a permanent, non-mergeable inference cost, because its nonlinearity — not its size — is what blocks the merge. Judge inference cost by mergeability, never by parameter count.

All figures below are a constructed scenario, not a measured benchmark. A team wants a frozen 13B model to consistently answer in a fixed structured format for one internal tool. Two candidate approaches: a LoRA adapter targeting the attention projections, or a P-tuning soft prompt of 20 learned virtual tokens. Both are trained on the same 2,000-example dataset. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) frames P-tuning's soft-prompt mechanism as the minimal-machinery option among the three PEFT methods this lesson covers, and the arithmetic below shows exactly why.

Step 1 — trainable parameter count for each.

LoRA, rank r=8, targeting q_proj and v_proj across 40 layers, hidden size 5,120:
  per matrix: r x (d_in + d_out) = 8 x (5,120 + 5,120) = 81,920
  2 modules x 40 layers x 81,920  = 6,553,600 trainable params  ≈ 6.55M

P-tuning, 20 soft-prompt tokens at hidden size 5,120:
  20 x 5,120 = 102,400 trainable params  ≈ 0.10M

P-tuning's trainable count here is roughly 64 times smaller than the LoRA configuration above.

Step 2 — but count the recurring cost, not just the one-time trainable count. Trainable-parameter count is a training-time number, paid once. What actually matters for a production deployment is the cost paid on every request, forever:

LoRA, merged:
  extra context tokens consumed per request        = 0
  extra sequential compute per request (post-merge) = 0

P-tuning:
  extra context tokens consumed per request        = 20, every single request
  at 1M requests/day, average 500 output tokens each:
    baseline daily token volume ≈ 500,000,000 tokens
    soft-prompt overhead        ≈ 20,000,000 tokens/day of pure input overhead
    that is roughly 4% of the daily token volume spent on tokens that carry
    no user content at all, for as long as the deployment runs

Step 3 — the reversal this comparison is built to show. P-tuning trained a footprint 64 times smaller than LoRA, and yet at scale it is P-tuning — not LoRA — that carries the larger ongoing cost, because its cost compounds with every request while a merged LoRA's cost was paid once, in training, and is now zero. This is the same shape of trap as the earned insight above, applied to a different axis: small training footprint does not predict small serving footprint any more than small parameter count predicts small inference-latency footprint. The three numbers that actually matter for a deployment decision — trainable-parameter count, mergeability, and recurring per-request cost — can point in three different directions, and a professional-level question is testing whether you check all three rather than stopping at the first.

PEFT is a training-time decision about how many parameters to update; it does not replace the inference-time optimization levers Model Optimization covers, and conflating the two is a recurring category error. Quantization, distillation, pruning, and KV caching all act on a model that has already finished training — they answer "how do we run this model cheaply," while PEFT answers "how do we change this model cheaply." The two compose rather than compete: a LoRA-adapted model can still be quantized for serving, and its KV cache behaves exactly as it would for any other decoder, because a merged LoRA adapter is, from the serving stack's point of view, an ordinary set of weights.

The same composability applies to Module 7's parallelism levers. Because PEFT trains a small fraction of the model's parameters, gradient and optimizer-state memory shrink dramatically relative to full fine-tuning — the frozen base still occupies its full memory footprint, but the tensor-parallelism and memory-sharding decisions in that module become far less pressing for a PEFT run than for a full fine-tune of the same base model, simply because there is so much less trainable state to shard.

Fine-Tuning is tied for third-largest domain on the NCP-GENL blueprint at 13% of the exam, and PEFT is its opening subject because it is the precondition for almost everything else the domain covers: alignment methods, discussed next in this module, are typically applied on top of an already-adapted or already-aligned model, and the memory arithmetic that makes alignment tractable at all assumes some PEFT method is doing most of the heavy lifting. Within the domain's own framing, professional-level questions "probe the distinguishing properties of each method — e.g., LoRA adds no inference latency" [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md), which is precisely the property this lesson has spent its worked examples pricing out in arithmetic rather than asserting as a slogan.

Expect this material to appear in a few recurring shapes: a direct identification question ("which PEFT method's update can be merged into the base weights with no added inference cost?"), a trap-reversal question that states the "PEFT shrinks the model" misconception as fact and asks you to evaluate it, and a scenario question naming a specific constraint — multi-tenant serving, a strict latency SLA, a minimal-infrastructure requirement — and asking which of the three methods fits. The distractors in this domain's house style tend to be a real, correctly-described technique attached to the wrong constraint: offering "adapters" as the answer to a zero-latency requirement, or offering "P-tuning" as the answer to a multi-tenant swapping requirement it does not naturally support as cleanly as LoRA's adapter-swap pattern.

No, not in the same sense, and this is worth stating precisely because the two failure modes are genuinely different. A bottleneck adapter adds latency because the frozen network itself must execute extra sequential computation at every insertion point on every forward pass. P-tuning adds no such computation inside the network — the transformer's own layers run exactly as they would for any input. What P-tuning costs instead is context-window budget: its soft prompt tokens occupy input positions that could otherwise carry user content, and because attention cost scales with sequence length, a longer effective input (soft prompt plus user content) does cost somewhat more compute than the user content alone would — but that is the ordinary cost of a longer sequence, the same cost a human-written few-shot example would impose, not a new architectural insertion the way an adapter block is.

Generally no, and the reason is structural rather than incidental. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) notes adapters "can add inference latency if not merged," which leaves the door open in principle, but a typical bottleneck adapter's down-projection, nonlinearity, and up-projection cannot be algebraically collapsed into a single linear term the way LoRA's B·A product can be added to W, because the nonlinearity sitting between the two projections is not a linear operation and therefore has no equivalent inside the frozen weight matrix to fold into. For the professional exam, the safe reading of [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md) is that LoRA's mergeability is the property stated to generalize, while an adapter's mergeability is conditional — "if not merged" implies some designs might allow it, but the source names no specific mergeable adapter variant, so treat any adapter beyond the classic inserted-layer description as ⚠️ UNVERIFIED rather than assume it merges as cleanly as LoRA.

• PEFT never shrinks the base model. [GROUND TRUTH] (Sources/ncp-genl/domain-5-fine-tuning.md): it trains a small new parameter set while the frozen weights stay fully resident and unchanged in size.
• Only LoRA's update is generally mergeable at zero added inference cost, because its low-rank product shares the exact shape of the weight matrix it augments and can be algebraically summed into it.
• A bottleneck adapter's nonlinearity blocks that same algebraic merge, which is why adapters can add measurable inference latency that persists for as long as they stay deployed, unlike a merged LoRA adapter.
• P-tuning's cost is context-window consumption, not per-layer computation — its soft prompts occupy input positions on every request, permanently, with no equivalent of merging available to remove that cost.
• Judge inference cost by mergeability and intervention point, never by trainable-parameter count alone — the earned insight this lesson's worked example demonstrates directly.
• All three methods compose with the inference-time optimization levers — quantization, distillation, pruning, KV caching — because a merged PEFT adapter is, from the serving stack's perspective, an ordinary set of weights.

Choosing among LoRA, adapters, and P-tuning settles which small parameter set to train for a behavioral change. It says nothing about what kind of behavioral change is being requested — and the domain's next and most heavily trapped subject is exactly that question, applied to the specific case of aligning a model with human preferences. Next: M5-02 walks the four alignment methods — SFT, RLHF, DPO, and GRPO — and the single distinguishing fact each one hinges on: which of them needs a separate reward model, which needs a critic network, and which needs neither.