LoRA and Parameter-Efficient Fine-Tuning (PEFT) Explained

Parameter-efficient fine-tuning (PEFT) is the family of adaptation methods that update only a small subset of a model's parameters — or add a small number of new ones — while leaving the vast majority of the pretrained weights frozen. The goal is to obtain most of a full fine-tune's behavioural benefit at a small fraction of its memory, storage, and operational cost. PEFT is a category, not a single technique; LoRA, adapters, prompt tuning, p-tuning, and prefix tuning are all members of it.

LoRA (Low-Rank Adaptation) is the PEFT method that dominates practice. For a chosen weight matrix W of shape (d_out × d_in) in the frozen model, LoRA introduces two new matrices, B of shape (d_out × r) and A of shape (r × d_in), where r — the rank — is small. During the forward pass the layer computes:

h = W·x  +  (α/r) · B·A·x
     │              │
  frozen         trainable

W never changes. A and B do. The product B·A has the same shape as W, so its contribution can be added directly to the layer's output, but it contains only r·(d_out + d_in) parameters instead of d_out · d_in. When r is small relative to the dimensions, that is a dramatic reduction.

The name is literal. B·A is a matrix whose rank is at most r, so LoRA constrains the update to W to be low-rank. The underlying bet is that the change a fine-tune needs to make to a pretrained weight matrix is itself low-dimensional — that adapting a model to a task is a small, structured nudge rather than an arbitrary rewrite. The empirical success of the method is evidence for that bet, and it is worth stating as a hypothesis the method rests on rather than as a proven fact about all tasks.

Three properties follow immediately and are what make LoRA operationally attractive:

1. Training memory collapses, because gradients and optimizer state exist only for A and B.
2. The artifact is tiny. You save A and B, not a whole model. Megabytes instead of gigabytes.
3. It is reversible and composable. The base weights are untouched, so you can detach the adapter to get the original model back exactly, or keep several adapters for several tasks and swap them against one shared base.

L1 — The intuition: a sticky note on a frozen page

Imagine the model's weight matrix as a printed page you are not allowed to edit. A full fine-tune reprints the whole page. LoRA writes a small correction on a sticky note and instructs the reader to add the note's contents to whatever the page says. The page stays pristine; the note is small; you can peel it off; you can keep a different note for a different reader.

The reason the note can be small is the low-rank bet: the corrections a task needs are not scattered arbitrarily across every entry of the matrix. They are structured, and a structured correction can be written compactly as the product of two thin matrices.

The reason this saves so much memory is not the note's size directly — 14 GB of frozen page is still 14 GB. It is that gradients and optimizer state are only ever computed for things that can change, and the only thing that can change is the note.

L2 — The mechanics: rank, alpha, target modules, initialisation, merging

Rank (r) sets the width of the bottleneck and therefore the number of trainable parameters. For a weight matrix of shape (d_out × d_in):

full fine-tune trainable params for this matrix = d_out × d_in
LoRA trainable params for this matrix           = r × (d_out + d_in)

Higher rank means more capacity to express the update and more parameters to train. Lower rank means fewer parameters and a tighter constraint on what the adaptation can represent. There is no universal correct rank, and any source presenting one as a standard is overstating; what exists is a trade-off you resolve empirically for your task, model, and data volume, with an eval set to tell you when raising it stopped helping.

Alpha (α) is a scaling constant. The adapter's contribution is multiplied by α/r before being added. The purpose of dividing by r is to keep the magnitude of the adapter's effect roughly stable as you change the rank, so that raising rank changes capacity without simultaneously changing effective step size — which would confound any rank sweep you run. The ratio α/r is what actually matters; two configurations with the same ratio have comparable adapter influence. Treat specific alpha values as configuration to tune, not as recommended constants.

Target modules are the layers you attach adapters to. In a transformer, the candidates are the attention projections — query, key, value, and output — and the feed-forward layers. Attaching to more modules means more trainable parameters and more expressive adaptation; attaching to fewer means a smaller, cheaper adapter. The attention projections are the most commonly targeted, and the query and value projections in particular are a frequent starting choice. Which set is best is task-dependent and is exactly the kind of thing your eval set should decide.

Initialisation matters and is elegantly chosen. A is initialised randomly and B is initialised to zeros. Therefore B·A = 0 at step zero, so the adapter contributes nothing and the model starts exactly as the pretrained model behaves. Training then moves the adapter away from zero. This is why a LoRA run cannot make the model worse at initialisation — a genuinely useful property, and a nice contrast with a full fine-tune where the first optimizer step already perturbs everything.

Merging. Because the adapter's contribution is an additive term of the same shape as W, you can fold it in permanently:

W_merged = W + (α/r) · B·A

After merging there is no adapter and no extra computation at inference — the model is an ordinary model with slightly different weights. Merging removes the small inference overhead of the extra matrix multiplications and removes the ability to detach. Keeping the adapter separate preserves reversibility and lets you serve many adapters against one base. That is a real deployment decision, and the table in section 5 covers it.

What the training loop looks like. Nothing exotic: the same supervised fine-tuning procedure as 11-02, same data shape, same loss, same masking. Gradients flow backward through the frozen weights — you still need the backward pass to reach the adapter — but no gradient is stored for the frozen weights and no optimizer state is allocated for them. That distinction is the source of the saving and it is worth being precise about: the backward pass still traverses the whole network; it just does not accumulate state for the frozen parts.

L3 — Where the savings come from, term by term, and where they do not

Take the four memory terms from 11-04 and mark what LoRA does to each:

Two of these deserve emphasis because they are where people's expectations are wrong.

LoRA does not reduce the weight term. The frozen base must be in memory to compute the forward pass. This is why, after applying LoRA, the frozen weights become the new dominant term — and why the next lever is to quantise them. Loading the frozen base in 4-bit while training a bf16 adapter on top is the recipe usually called QLoRA, and it is a direct consequence of this table: attack whatever term is now largest. 12-02 covers quantisation's accuracy trade-offs.

LoRA does not much reduce activation memory. Activations depend on batch size, sequence length, hidden size, and depth, and LoRA changes none of those. So gradient checkpointing remains a necessary companion lever when sequences are long or batches are large. A candidate who believes LoRA solves all memory problems will be surprised by an OOM at step 40 with a long example.

A third L3 point, on the quality question. The honest framing: LoRA is widely used precisely because it recovers most of the benefit of full fine-tuning for typical adaptation tasks at a fraction of the cost, and the original work motivating it argues that the required weight update is low-rank. This course does not quote specific quality comparisons, because published numbers are task-, model-, and configuration-specific and reproducing them as general claims would be misleading. What you should carry is the shape of the trade-off: LoRA constrains the update, and a constrained update cannot in general match an unconstrained one; for behavioural adaptation the constraint is usually not the limiting factor; for adaptation that genuinely requires broad representational change — a large distribution shift, a new language, a new modality — full-parameter training or continued pretraining has headroom LoRA does not. Decide with your eval set, not with a rule of thumb.

Finally, a structural benefit that is easy to undervalue: LoRA reduces catastrophic-forgetting exposure, because the weights that encode general capability are physically frozen and cannot be overwritten. This is a qualitative argument from the method's structure, not a measured claim, and it does not eliminate the need for the baseline eval run 11-03 insists on — a strongly-trained adapter can still suppress an existing behaviour additively. But the mechanism that causes the worst forgetting is unavailable to a LoRA run, and detaching the adapter is a true rollback.

The customisation ladder in cost order is: prompt → RAG → prompt learning (prompt tuning, p-tuning) → PEFT/LoRA/adapters → full fine-tune → alignment. Here is the comparison across the rungs that matter for this lesson.

Two contrasts inside that table are exam-grade confusables in their own right.

LoRA vs prompt tuning / p-tuning. Both are PEFT and both leave the base weights alone, but they intervene at different places. Prompt-learning methods prepend trainable continuous vectors to the input sequence — soft prompts that are optimised by gradient descent but are not real tokens and are not human-readable. They act at the input. LoRA acts inside the network, modifying what weight matrices compute. Consequences: prompt tuning consumes context positions on every request, has fewer trainable parameters, and generally has less capacity to reshape behaviour; LoRA consumes no context, has more capacity, and can be merged into the weights. On the ladder, prompt learning sits below PEFT — cheaper and less powerful.

LoRA vs classic adapters. Classic adapter methods insert small new layers into the network, so the forward pass has additional sequential blocks to run — which adds inference latency that cannot be removed. LoRA's contribution is a parallel additive term that can be merged into the existing weight matrix, so a merged LoRA has zero inference overhead. That mergeability is a large part of why LoRA became the default rather than one option among several.

And the one that is not a comparison at all: LoRA vs RAG. These are not competing options and a question that frames them as such is testing whether you know the difference between changing the model and changing its input. LoRA changes behaviour; RAG supplies knowledge. The production answer is usually both. 11-08 settles this formally.

You are going to adapt a 7-billion-parameter model to produce your team's report format. Before launching, predict the trainable parameter count, the adapter file size, and peak static memory. Every number below is derived from the stated assumptions; the architecture figures are a constructed illustrative configuration, not a specific product's specification.

Assumptions:

Total parameters P_total    = 7.0e9
Layers L                    = 32
Hidden size d               = 4,096
Attention projections       = q, k, v, o, each (4,096 × 4,096)
Target modules              = q_proj and v_proj only
Rank r                      = 8
Alpha α                     = 16   (so α/r = 2.0)
Precision                   = bf16 (2 bytes) for adapter and frozen base
Optimizer                   = Adam, fp32 moments (4 bytes each, two of them)

Step 1 — trainable parameters per adapted matrix.

For one (4,096 × 4,096) matrix at rank 8:
  A is (r × d_in)  = (8 × 4,096)   = 32,768 params
  B is (d_out × r) = (4,096 × 8)   = 32,768 params
  total per matrix                 = 65,536 params

Compare that with the matrix it is adapting:

full matrix = 4,096 × 4,096 = 16,777,216 params
adapter     =                     65,536 params
ratio       = 65,536 / 16,777,216 = 0.39%

Step 2 — total across all target modules.

2 target modules (q, v) × 32 layers = 64 adapted matrices
64 × 65,536 = 4,194,304 trainable params ≈ 4.19 M

Step 3 — the trainable fraction.

4.194e6 / 7.0e9 = 0.0599% ≈ 0.06% of the model

Six hundredths of one percent. That is the number that does all the work.

Step 4 — adapter artifact size on disk.

4.194e6 params × 2 bytes (bf16) = 8.39e6 bytes ≈ 8.4 MB

An 8.4 MB file versus a 14 GB model checkpoint — a ratio of about 1,670 to 1. This is why "one base model, fifty customer adapters" is a viable architecture and "fifty fine-tuned models" is not.

Step 5 — predicted peak static training memory.

frozen base weights   7.0e9  × 2 B = 14,000  MB  = 14.00 GB
adapter weights       4.19e6 × 2 B =      8.4 MB
adapter gradients     4.19e6 × 2 B =      8.4 MB
Adam moment 1         4.19e6 × 4 B =     16.8 MB
Adam moment 2         4.19e6 × 4 B =     16.8 MB
────────────────────────────────────────────────
static total                       ≈ 14,050 MB ≈ 14.05 GB (13.09 GiB)

Step 6 — compare against the full fine-tune from 11-04.

Full fine-tune static total   ≈ 84.00 GB
LoRA static total             ≈ 14.05 GB
Reduction factor              ≈ 5.98×

Of the 70 GB saved:
  gradients          14.0 GB → 0.008 GB
  optimizer state    56.0 GB → 0.034 GB
  weights            14.0 GB → 14.0 GB   (unchanged)

Every byte of the saving came from the two terms that scale with trainable parameters. The weight term is untouched, and it is now 99.6% of the total — which tells you exactly what to attack next.

Step 7 — add activations and check the fit. Suppose micro-batch 2 at sequence length 2,048, with gradient checkpointing enabled. Activations are the dynamic term and depend on framework details, so this is an estimate to calibrate rather than a prediction to trust:

static             ≈ 14.05 GB
activations (est.) ≈ 1–3 GB with checkpointing on
framework overhead ≈ 1 GB
────────────────────────────
peak (est.)        ≈ 16–18 GB

On a 24 GB card, that fits with headroom. On a 16 GB card it is marginal, which brings us to step 8.

Step 8 — the 4-bit base variant (QLoRA-style), to reach a 16 GB device.

frozen base at ~4 bits: 7.0e9 × 0.5 B = 3,500 MB = 3.50 GB
adapter + grads + Adam state (unchanged) ≈  0.05 GB
────────────────────────────────────────────────
static total                            ≈  3.55 GB
+ activations with checkpointing (est.)  ≈  1–3 GB
+ overhead                               ≈  1 GB
────────────────────────────────────────────────
peak (est.)                              ≈  6–8 GB

Now a 16 GB card is comfortable and even a 12 GB card is plausible. Note the progression across the three configurations, because it is the module's whole argument in one block:

Full fine-tune, bf16, Adam         ≈ 84.0 GB   → multi-GPU job
LoRA, bf16 frozen base             ≈ 14.1 GB   → one 24 GB card
LoRA, 4-bit frozen base            ≈  3.6 GB   → one modest card

Step 9 — decide what to measure. Before launching, write down the predicted peak. After launching, read the actual peak from the framework's memory report. The gap between them is your activation-and-overhead calibration, and it is the number that makes your next prediction accurate. This is the habit the lesson wants you to leave with: predict, then measure, then reconcile.

Step 10 — run the baseline eval first. 11-03 is unconditional. LoRA reduces forgetting exposure; it does not remove the need to know what the model could do before you touched it. Record the per-slice baseline, then train, then re-score with identical decoding settings.

Merged versus unmerged at serving time is its own decision:

LoRA is explicitly core associate-level content on this exam, not an advanced aside. Two independent signals establish that. First, the customisation ladder — prompt → RAG → prompt learning → PEFT/LoRA/adapters → full fine-tune → alignment — is in the must-know content for the Core Machine Learning and AI Knowledge domain, which is 30% of the exam and the largest domain on it. Second, the official study guide's suggested-reading list names the LoRA paper directly, which this course teaches inline rather than deferring.

Beyond the ladder, LoRA serves the blueprint's objectives on identifying the hardware and software components required to meet user needs, and on assisting with deployment and evaluation of scalability and performance. It is the answer to "we have one GPU and a customisation requirement," which is the single most common real-world version of that objective.

The exam's calibration matters for how you study it. Candidate reports describe questions as general-level rather than deep-technical, so expect to be tested on identity and when-to-use, not on rank-selection heuristics or optimiser configuration. Know that LoRA freezes the base and trains low-rank matrices. Know that the memory saving comes from gradients and optimizer state, not from a smaller model. Know its position on the ladder. Know that the artifact is tiny and swappable.

Question phrasings to expect:

• "What is the primary characteristic of LoRA?" — the original weights are frozen and small low-rank matrices are trained alongside them.
• "Why does LoRA require far less GPU memory than full fine-tuning?" — gradients and optimizer state scale with trainable parameters, and LoRA makes that count tiny.
• "Which approach lets one deployed base model serve several task-specific behaviours?" — multiple LoRA adapters against a shared base.
• "Which of these does NOT change the model's weights?" — prompting, RAG, and (in the sense that the base is untouched) prompt tuning; note carefully how the option is worded.
• "Place these customisation approaches in order of increasing cost." — prompt, RAG, prompt learning, PEFT/LoRA, full fine-tune, alignment.
• "A team must fine-tune a 7B model on a single 24 GB GPU. What is the most appropriate approach?" — parameter-efficient fine-tuning, optionally with a quantised base.
• "What does the rank parameter in LoRA control?" — the size of the low-rank bottleneck, hence the trainable parameter count and the adaptation's capacity.
• "What is one advantage of merging a LoRA adapter into the base weights?" — no additional inference overhead.

Distractor families:

Full fine-tuning updates every parameter in the model; LoRA freezes all of them and trains a small pair of low-rank matrices beside each targeted weight matrix. Everything else that differs follows from that one structural choice.

The honest summary of the quality question: LoRA constrains the update, so it cannot in general match an unconstrained one, and for typical behavioural adaptation that constraint is usually not what limits you. Where the required change is broad and representational rather than behavioural, full-parameter training has headroom. Do not accept a general claim in either direction — including from this page — over your own eval set.

Training memory, primarily and dramatically. Inference memory is essentially unchanged, and can be marginally higher if you serve the adapter unmerged, because the adapter's parameters are additional resident tensors and the forward pass performs a few extra small matrix multiplications.

This is one of the most common misconceptions about the method, and it is worth being blunt about the mechanism. The frozen base weights are needed to compute the forward pass, so they are in memory at both training and inference time. What LoRA removes is the gradient tensor and the optimizer moment tensors — objects that exist only during training. Inference never allocated them, so there is nothing for LoRA to save there.

If your problem is inference memory, the levers are different ones:

Where LoRA does help operationally at serving time is storage and deployment economics rather than device memory: fifty behaviours as fifty 8 MB adapters against one shared base is a completely different infrastructure proposition from fifty 14 GB models, in registry size, in deployment time, and in how many distinct models a single GPU can effectively serve.

Empirically, with an eval set, starting small and escalating only on evidence — and with the explicit understanding that there is no published standard value to look up. Anyone who hands you a canonical rank is giving you their task's answer, not yours.

A defensible procedure:

1. Start with a small rank and the attention query and value projections. This is a common, cheap starting configuration that gives you a working run and a baseline data point fast.
2. Hold α/r constant while you sweep r. Otherwise you are changing capacity and effective adapter magnitude simultaneously and cannot attribute the result.
3. Widen target modules before pushing rank very high. Adding the key and output projections, or the feed-forward layers, often buys more than the same parameter budget spent on a wider bottleneck — but this is a hypothesis for your task, not a law.
4. Watch the general-capability slices, not just the target. More trainable parameters means more capacity to suppress existing behaviour. The forgetting table from 11-03 is the instrument.
5. Stop when the eval gain flattens. Extra rank costs memory and forgetting exposure; if the target metric has plateaued you are paying for nothing.
6. Record the whole configuration with the adapter. Base checkpoint, rank, alpha, target modules, learning rate, epochs, data version. An adapter without its configuration is not reproducible, and reproducibility discipline is 09-11.

Data volume is the constraint people most often ignore. A high-rank adapter has more parameters to fit, and fitting more parameters on a small dataset is the standard recipe for overfitting (01-07). If you have a few hundred examples, a small rank is not a compromise — it is the appropriate amount of capacity for the amount of signal you have.

Yes, and it is one of the method's most useful operational properties, with two distinct patterns that should not be confused.

Swapping is the well-behaved pattern: one base model resident on the GPU, many adapters on disk, and the serving layer attaches whichever adapter the request needs. Because each adapter is megabytes, hundreds of behavioural variants can share one deployment. This is the multi-tenant story — per-customer tone, per-team format, per-task behaviour — and it is impossible with full fine-tuning, where each variant is a separate multi-gigabyte model with its own memory footprint.

Composing — applying two or more adapters simultaneously so their contributions both add into the forward pass — is arithmetically straightforward and behaviourally unpredictable. Two adapters trained independently for different objectives were never optimised to coexist, and their combined effect is not the union of their intentions. Treat composition as an experiment requiring its own evaluation, not as a feature you can assume works.

The operational costs to budget for either pattern:

• Adapter-swap latency. Loading and switching is fast relative to loading a model, but it is not free, and under high churn it shows up in tail latency. Measure it (12-10).
• Version pinning. Every adapter is tied to a specific base checkpoint. Upgrade the base and every adapter needs re-validation at minimum, retraining at worst.
• Evaluation multiplication. N adapters is N eval runs. The regression suite from 10-04 is what keeps that tractable.

• LoRA freezes the pretrained weights and trains a pair of small low-rank matrices alongside each targeted weight matrix. No original weight is updated. Say it exactly that way on the exam.
• The memory saving comes from gradients and optimizer state, not from a smaller model. Those two terms scale with trainable parameters; the frozen base is still fully resident.
• The arithmetic is the argument. In the constructed 7B example, a 0.06% trainable fraction took static training memory from about 84 GB to about 14 GB, and to about 3.6 GB with a 4-bit frozen base.
• LoRA does not reduce activation memory or inference memory. Gradient checkpointing handles the first; quantisation handles the second.
• The artifact is megabytes, not gigabytes, which makes one-base-many-adapters a real architecture and detach-to-roll-back a real operation.
• Rank sets capacity and cost; α/r sets the adapter's effective magnitude. There are no standard values — sweep with your eval set and hold α/r while varying r.
• Zero-initialising B means the model starts exactly as the base model behaved, so a LoRA run cannot be worse at step zero.
• Merging removes inference overhead and removes detachability. Choose deliberately; keep it unmerged when rollback or multi-tenancy matters.
• Forgetting exposure is lower because the general-capability weights are frozen — a structural argument, not a licence to skip the baseline eval.
• It is still fine-tuning. Behaviour, format, tone, and register: yes. Facts, freshness, citations, deletability: no — that is retrieval's job.
• Its rung on the ladder is above prompt learning and below full fine-tuning, and you escalate only when your eval set says the cheaper rung fell short.

You can now change a model's behaviour cheaply, reversibly, and on hardware you actually have. But every method so far has needed someone to write down the right answer — a demonstration to imitate. There is a whole class of qualities nobody can write a target for: which of two acceptable summaries is more helpful, which refusal is better calibrated, which explanation is clearer. For those you need a different kind of supervision, one where humans compare rather than author. Next: 11-06 walks the RLHF pipeline in its strict order — supervised fine-tuning, then a reward model trained from human preference labels, then policy optimisation — and shows why it is the only mechanism in the stack that moves human judgement into weights.