LLM Parameters: What They Are and Where Knowledge Is Stored

An LLM's parameters are the learned numbers inside the model — the weight matrices and bias vectors of its embedding table, attention projections and feed-forward layers. They are the sole storage for everything the model learned during training. A prompt, a retrieved passage or a system message never becomes a parameter; it is input, consumed and discarded.

That distinction is the first pole of a spine that runs the length of this course. Every customisation technique on the exam sorts into exactly two buckets: the ones that change parameters, and the ones that change what you feed the model. Knowing which bucket a technique falls into answers a large share of the scenario questions in the Core ML domain, and it is the difference between an engineer who proposes fine-tuning for a daily-changing price list and one who proposes retrieval.

1. What LLM parameters are

Identity statement: a parameter is a single number whose value was learned by training — adjusted by gradient descent to reduce prediction error. "A 7-billion-parameter model" means 7 billion such numbers.

When the term matters: whenever you need to answer "will this fit on my GPU?", "does this technique update the model?", or "why does the model still not know our internal product names?"

Parameters are not scattered randomly. They live in named structures:

Structure	What it holds	Rough share of total
Embedding matrix	One learned vector per vocabulary token	Small in large models, significant in small ones
Attention projections (Q, K, V, output)	How tokens attend to one another	About one third of each transformer block
Feed-forward / MLP layers	The per-position transformation after attention	About two thirds of each transformer block
Layer-norm scales and biases	Per-layer scaling	Negligible
Output / unembedding matrix	Hidden state back to vocabulary scores	Same order as the embedding matrix

The distinctive number for this page: in the standard transformer block where the feed-forward inner dimension is 4× the hidden size, the four attention projections contribute 4·d² parameters and the two feed-forward matrices contribute 8·d², so the feed-forward layers hold roughly two thirds of a transformer block's parameters and attention holds one third. Attention gets the attention; the MLP holds the bulk of the storage. You will not be asked to derive that, but it usefully corrects the common mental picture that "the knowledge is in the attention."

Weights, biases, and what "trainable" means

Two vocabulary items get used interchangeably and should not be. A weight is a multiplicative parameter — an entry in a matrix that scales an input. A bias is an additive parameter — a per-output offset. "Parameters" is the umbrella term covering both, and it is what a model's advertised size counts. Many modern LLM architectures drop most biases entirely for efficiency, which is why you will see "weights" used as a loose synonym for "parameters" in practice.

Trainable is the other word to pin down. A parameter is trainable if the current training run is allowed to update it. The same number can be trainable in one run and frozen in the next — that is precisely the mechanism behind parameter-efficient fine-tuning, where the base checkpoint is marked frozen and only a small added set is trainable. So "the model has 7B parameters and we fine-tuned 0.1% of them" is a coherent sentence: 7B is the storage, 0.1% is the trainable subset for that run.

2. How knowledge gets into parameters and how it comes back out

L1 — Weights as compressed regularities

Training exposes the model to text and adjusts parameters so predictions improve (01-01). Regularities that repeat across the corpus — that Paris follows "the capital of France is," that Python functions start with def, that a polite reply follows a polite question — get baked into weight values. The model is not a database with rows; it is a lossy compression of statistical structure in its training data. That is why it recalls widely repeated facts reliably and rare ones badly, and why it cannot cite where a fact came from.

If you take one image away, take this one: the weights are a lossy compression of the training corpus, not an index into it. Lossy explains the failure modes. Compression explains why a 14 GB file can discuss almost any topic. Not an index explains why it cannot tell you its source.

L2 — Frozen at inference

Once training ends, parameters are frozen. Serving a model is a read-only operation over those numbers. Nothing you send at inference writes back. Consequences the exam cares about:

• The knowledge cutoff is a property of the weights. No amount of prompting adds post-training knowledge; you must supply it in the context, which is precisely what RAG does.
• Conversation "memory" is re-sent text. A chatbot that appears to remember earlier turns is being handed those turns again inside its context window on every request.
• Two identical requests to a frozen model differ only through decoding randomness, not through the model having changed.
• The same checkpoint on two different GPUs is the same model. Behaviour differences at that point come from precision, kernel implementation or decoding settings, not from knowledge.

Frozen also has an operational meaning worth carrying: because the file does not change, a checkpoint is a versionable artefact. You can hash it, store it, roll back to it, and state exactly which weights produced a logged output. That property is what makes model versioning and reproducibility possible at all, and it is why the evaluation discipline in 01-08 insists on recording which checkpoint produced a score.

L3 — Precision, and the memory arithmetic

Each parameter is stored in a numeric format, and the format sets its byte cost:

Precision	Bytes per parameter	Weight memory for a 7B model
FP32	4	~28 GB
FP16 / BF16	2	~14 GB
INT8 / FP8	1	~7 GB
INT4	0.5	~3.5 GB

The rule to memorise: weight memory ≈ parameter count × bytes per parameter. At the widely used FP16/BF16 serving default that collapses to a two-second estimate — double the billions and read the answer in gigabytes: 7B → ~14 GB, 13B → ~26 GB, 70B → ~140 GB.

Two honesty notes. First, units: 7 × 10⁹ × 2 bytes is 14 GB decimal but 13.0 GiB, and a GPU advertised as "16 GB" is typically 16 GiB. The gap is about 7% and it decides whether a model fits — this is the arithmetic set up in M0.4. Second, weights are not the whole footprint. Serving also needs the KV cache (which grows with batch size and sequence length) and activation working space; training needs optimizer state and gradients on top, which is why training a model takes several times the memory of serving it. Capacity planning gets its own treatment later in the course; here you only need the weights term and the reason the other terms exist.

The lesson stops at L3 deliberately. You do not need to know how BF16's exponent range differs from FP16's, or which kernels are available at which precision. What you need is that precision is a bytes-per-parameter choice, that lowering it is the standard lever for fitting a model into less memory, and that the cost of lowering it is accuracy you have to measure rather than assume.

3. Parameters vs hyperparameters vs context vs decoding settings

Four things are routinely blurred, and distractors exploit all four.

	Parameters	Hyperparameters	Context (the prompt)	Decoding settings
Who sets the value	Learned by training	Chosen by a human before/around training	Supplied by the caller at request time	Supplied by the caller at request time
Examples	Attention weights, embedding vectors	Learning rate, batch size, epochs, layer count	System prompt, user question, retrieved passages, few-shot examples	Temperature, top-k, top-p, max tokens
Persists after the request?	Yes — stored in the checkpoint	Yes, as a training recipe	No — discarded when the request ends	No
Changed by prompting?	No	No	Yes, that is what prompting is	Separately configurable
Counted in "7B parameters"?	Yes	No	No	No
Affects what the model knows?	Yes	Indirectly, via how training went	No — adds what it can see	No

The fourth column is the one people get wrong in casual speech. Temperature and top-p are neither parameters nor training hyperparameters — they are decoding settings applied at inference to an already-frozen model. Calling temperature a "model parameter" is a common informal slip and a clean distractor.

And the pairing that matters most for scenario questions:

Technique	Changes parameters?	What it actually changes	Typical data volume
Prompt engineering	No	The input text	A handful of examples
RAG	No	The input text, augmented with retrieved evidence	A corpus to index, no labels
Prompt tuning / p-tuning	Adds a small set of new trainable vectors; base weights frozen	A learned soft prefix	Hundreds to thousands of examples
LoRA / adapters (PEFT)	Yes, a small added set; base weights frozen	A low-rank update applied to the base	Thousands of examples
Full fine-tuning	Yes, all of them	The whole checkpoint	Tens of thousands and up
Alignment / RLHF	Yes	The whole policy's weights	Large preference-labelled set

Read the table top to bottom and it is a ladder of increasing cost, increasing data requirement, and increasing commitment. Read the "changes parameters" column on its own and it is the single test that resolves most scenario items.

4. Parameters vs the other things measured in a model spec

A model card lists several numbers and they measure unrelated properties. Mixing them up is a reliable way to answer a sizing question wrong.

Quantity	What it measures	Units	Changed by
Parameter count	How much learned storage the model has	Count (7B, 70B)	Choosing a different model
Context window	How many tokens the model can read at once	Tokens (8k, 128k)	Choosing a different model, or a long-context variant
Precision	Bytes used to store each parameter	Bytes (4, 2, 1, 0.5)	Quantisation
Vocabulary size	How many distinct tokens exist	Count (32k, 128k)	The tokenizer, fixed with the model
Hidden size	Width of each token's internal vector	Count (4096)	Architecture
Layer count	Depth of the stack	Count (32)	Architecture

Two of these interact in a way worth knowing. Vocabulary size and hidden size together determine the embedding table's parameter count (vocabulary × hidden size), which is why a small model with a large vocabulary can spend a surprising fraction of its parameters just on embeddings. And context window is a runtime memory question, not a storage question: doubling the context you actually use roughly doubles the KV cache, while leaving the weight file byte-identical.

5. Worked example: sizing a 13B model for a 40 GB GPU

A team wants to serve a 13-billion-parameter model on a single 40 GB accelerator.

FP32:      13e9 × 4 bytes = 52 GB   → does not fit
FP16/BF16: 13e9 × 2 bytes = 26 GB   → weights fit, ~14 GB left for KV cache + activations
INT8:      13e9 × 1 byte  = 13 GB   → comfortable headroom for larger batches

Read it as an engineer, not an arithmetician. FP32 is eliminated outright. BF16 fits the weights with roughly 14 GB of working room — enough for a modest batch, and the batch size and context length you can then serve depend on how much of that the KV cache consumes. INT8 roughly doubles your headroom, at the cost of some accuracy that must be measured rather than assumed. (Quantisation methods and the accuracy question are their own topic later; here the point is that the decision is driven by a multiplication you can do in your head.)

Now do the units check that M0.4 set up, because it changes the BF16 answer's comfort level:

26 GB decimal = 26e9 / 2^30 GiB = 24.2 GiB
A "40 GB" accelerator is typically 40 GiB = 42.9 GB decimal
Headroom = 42.9 - 26 = 16.9 GB decimal ≈ 15.7 GiB

The direction of the correction matters. Because the card is measured in the larger binary unit and the weights in the smaller decimal one, this particular comparison comes out slightly better than the naive subtraction suggested. Reverse the situation — a decimal-advertised file against a binary-advertised card in the other direction — and it comes out worse. The habit, not the answer, is what you carry: convert both sides to the same unit before you conclude anything fits.

Now the same reasoning in reverse. If the team's real complaint is "the model does not know our internal API," no precision choice helps, because that fact was never in the weights. The options are: put the API docs in the context via retrieval, or change the weights via fine-tuning. Which one you pick is a parameters-versus-context decision — the exact distinction this lesson exists to make automatic.

6. Worked example: where the parameters of an illustrative 7B model actually sit

Take a deliberately illustrative configuration — these are round numbers chosen to make the arithmetic legible, not a spec sheet for any named model:

hidden size        d = 4096
layers             L = 32
FFN inner size         4d = 16384
vocabulary         V = 32000

Per transformer block:

attention projections   4 × (d × d)      = 4 × 4096 × 4096  ≈ 67.1M
feed-forward matrices   2 × (d × 4d)     = 2 × 4096 × 16384 ≈ 134.2M
layer norms                                                 ≈ negligible
--------------------------------------------------------------------
per block                                                   ≈ 201.3M

Across the stack and the embeddings:

32 blocks × 201.3M                       ≈ 6.44B
embedding table  V × d = 32000 × 4096    ≈ 0.13B
output projection (if untied) V × d      ≈ 0.13B
--------------------------------------------------------------------
total                                    ≈ 6.7B  → "a 7B model"

Three things to take from the arithmetic. First, the two-thirds/one-third split is visible: 134.2M versus 67.1M per block. Second, the embeddings are about 2% of this model's parameters — real, but not where the storage is. In a much smaller model with the same vocabulary, that fraction becomes large, which is why small models sometimes tie the input embedding and output projection to the same matrix to avoid paying twice. Third, "7B" is a marketing round number; the actual count is whatever the configuration produces.

Then apply the memory rule:

6.7e9 × 2 bytes (BF16) = 13.4 GB decimal = 12.5 GiB of weights

Which is the number that decides whether this checkpoint loads on a 16 GiB card with room left for a KV cache. It does — barely, and only at small batch sizes.

7. Why LLM parameters are on the NCA-GENL exam

Objective 1.5 (ML fundamentals) and objective 1.7 (reading research papers to spot emerging LLM trends) both assume this vocabulary — the official study guide's suggested-reading list includes LoRA, which is unreadable if you do not know what a parameter is. Core ML is 30% of the blueprint, the largest single domain.

The concept is questioned indirectly, in four recurring shapes:

1. Customisation-choice scenarios. "A team needs the assistant to answer from a document set updated daily." The keyed answer is retrieval, not fine-tuning, because retrieval changes context while fine-tuning changes weights — and daily-changing facts should not be baked into weights.
2. Sizing and deployment items. Anything that asks whether a model fits, or why quantisation helps, runs on the weight-memory rule above. Candidate reports say GPU spec-sheet depth did not appear; the arithmetic identity still does, because it is conceptual rather than product trivia.
3. Terminology discrimination. Parameter vs hyperparameter, weights vs context, training vs inference. Cheap points if the vocabulary is precise.
4. Knowledge-cutoff reasoning. "The model confidently describes a product feature released last month that does not exist." The keyed answer names the cutoff and points at grounding.

How the question tends to be phrased

You will typically see a short scenario naming a business need, four plausible technical responses, and one that respects the weights-versus-context boundary. The phrasings to recognise:

• "…knowledge that changes frequently…" → retrieval. Weights are the wrong store for volatile facts.
• "…must adopt our house tone and output format…" → fine-tuning territory, because style is a behaviour you want persisted.
• "…must cite the source document…" → retrieval, because weights cannot attribute.
• "…must run on a smaller GPU without changing models…" → quantisation, a bytes-per-parameter change.
• "…must not retain user data…" → note that inference does not write to weights at all, which is often the point of the item.

What the distractors typically look like

The standard traps are: fine-tune on the daily-changing corpus (technically possible, operationally wrong, and it still cannot cite); increase the context window offered as a fix for missing knowledge (a bigger window with nothing in it changes nothing); raise the temperature offered as a fix for factual gaps; and the model learns from the conversation offered as a mechanism, which contradicts frozen weights. Notice that each distractor is a real technique misapplied — that is the exam's house style, and the weights-versus-context test cuts through all four.

Calibration note, and it is calibration rather than fact: published candidate reports say the exam sits at a general level — know what each thing is and when to use it. NVIDIA publishes no per-item detail and no official passing score, so treat those reports as a study-planning aid only. For this lesson the practical reading is that the identity statements and the decision table earn more marks than any deeper numerical fluency would.

8. Common mistakes about parameters and where knowledge is stored

Mistake	Symptom you would actually observe	Fix
Believing a prompt "teaches" the model	The behaviour you demonstrated in one request is gone in the next session	In-context learning changes output for that request only; nothing persists
Believing a long conversation accumulates knowledge	Quality degrades once the window fills and early turns get truncated	It accumulates context, re-sent each turn, then dropped
Confusing parameter count with context-window size	You size a GPU from the context length, or expect a 7B model to read a book	Storage capacity versus how much text can be read at once — unrelated numbers
Assuming more parameters always means better	You over-provision and still miss the task metric	Data quality, alignment and retrieval frequently beat raw size at a fixed task
Forgetting optimizer state when sizing training	The job OOMs at step 1 despite the weights fitting easily	Training holds gradients and optimizer state on top of weights
Mixing decimal GB with binary GiB	A model that "should fit" fails to load by a few percent	A 14 GB weight file is 13.0 GiB and a "16 GB" card is 16 GiB, per M0.4
Saying "the knowledge is in the attention layers"	You misattribute where capacity lives and mis-answer a mechanism item	Attention routes information between positions; feed-forward layers hold about two thirds of the parameters
Treating quantisation as free	Throughput improves and a subtle quality regression ships unnoticed	Lower precision costs accuracy that must be measured on your own eval set

9. When to change weights and when to change context

A decision rule you can apply directly to scenario items.

The symptom	Root cause	Reach for	Do not reach for
Model does not know a fact that exists in your documents	Fact never entered the weights	Retrieval (RAG)	Fine-tuning; higher temperature
Model does not know a fact that changes daily	Volatile knowledge in a frozen store	Retrieval	Fine-tuning of any kind
Answers must cite a source	Weights cannot attribute	Retrieval with citation	Any weight change
Output format is inconsistent	Behaviour, not knowledge	Prompt constraints first, then fine-tuning if it must persist	Retrieval
Model uses the wrong house tone everywhere	Persistent behaviour requirement	Fine-tuning / PEFT	Retrieval
Model needs a new specialised skill with plenty of examples	Capability gap	PEFT, then full fine-tune if PEFT plateaus	Prompting alone
Outputs vary too much between runs	Sampling	Lower temperature, or greedy decoding	Any weight change
Model will not fit on the available GPU	Bytes per parameter	Quantisation, or a smaller model	Reducing the context window and hoping
Model must stop producing unsafe content	Policy layer	Guardrails plus alignment	Prompt-only patching

The reason this table is worth memorising in shape rather than in detail is that the exam does not ask you to build any of these. It asks you to recognise which one a described situation calls for — exactly the posture the job-role frame describes, where the associate contributes under supervision and is expected to make correct tool choices.

10. Does fine-tuning add knowledge or just change style?

Both, but not equally reliably, and the honest answer is the one the exam rewards.

Fine-tuning demonstrably changes behaviour: format, tone, task framing, willingness to answer in a particular shape. It does that with relatively little data because you are steering something the model can already do.

Fine-tuning can also add knowledge, but it is a poor mechanism for the kind of knowledge business applications usually need. Facts learned this way are diffuse rather than addressable, cannot be cited, go stale the moment the source changes, and need a retraining cycle to update. Worse, fine-tuning a small factual set into a large model risks degrading the surrounding behaviour without a matching evaluation set to catch it.

So the practical rule: fine-tune for behaviour, retrieve for knowledge. Where a scenario needs both — house tone and current facts — the correct answer is usually both techniques together, and an option offering that combination is worth a second look.

11. Why can't an LLM tell you where a fact came from?

Because there is nowhere for the provenance to live. A fact in the weights is not a row with a source column; it is a diffuse pattern distributed across many parameters, contributed to by many documents and reinforced by repetition. There is no pointer back to a document because no document was ever stored.

When a model does produce a citation without retrieval, it is generating text that has the shape of a citation, sampled from a distribution over plausible-looking references. That is the mechanism behind fabricated URLs and invented paper titles: the objective in 01-01 rewards plausible continuations, and a citation is just more text to continue.

Which is why provenance is an architectural property rather than a model property. Retrieval supplies the document, the pipeline passes it in the context, the prompt asks the model to answer only from it, and the citation points at something that actually exists on your side of the system. The model did not gain the ability to attribute; the system gained a source to attribute to.

12. Do parameters and hyperparameters ever swap roles?

Not within one training run, but the boundary is a design choice rather than a law, and knowing that stops the distinction from feeling arbitrary.

Architecture choices — layer count, hidden size, attention head count — are hyperparameters: a human sets them, and they determine how many parameters exist. Optimisation choices — learning rate, batch size, epoch count, weight decay — are hyperparameters that determine what values the parameters end up with. Neither is learned by gradient descent on the training loss, which is the defining test.

The edge cases are worth one sentence each. Learning-rate schedules change a hyperparameter over time by rule, not by learning. Learned positional embeddings are genuine parameters even though "position" feels like a configuration detail. And soft prompts in prompt tuning are new parameters that behave like a prompt — which is exactly why the customisation table above lists them in their own row rather than in either bucket.

13. Glossary recap: the terms this lesson introduced

Term	One-line definition
Parameter	A number learned by training; the umbrella term for weights and biases
Weight	A multiplicative parameter, an entry in a learned matrix
Bias	An additive per-output parameter; often omitted in modern LLMs
Trainable	Whether the current run is permitted to update a given parameter
Frozen	Not updated — the state of all parameters at inference, and of base weights under PEFT
Checkpoint	The stored file of parameter values; the versionable model artefact
Hyperparameter	A value a human sets, not learned by gradient descent — learning rate, batch size, layer count
Context	The text supplied at request time; consumed and discarded, never stored
Precision	The numeric format of each parameter, measured in bytes per parameter
Quantisation	Reducing precision to cut memory, at a measurable accuracy cost
Knowledge cutoff	The point beyond which nothing is in the weights, because training stopped
Embedding matrix	The learned vector per vocabulary token; size is vocabulary × hidden
Unembedding / output projection	The matrix mapping a hidden state back to vocabulary scores
KV cache	Runtime memory holding attention keys and values, growing with batch and sequence length

14. Key takeaways on LLM parameters

• Parameters are learned numbers; they are the model's only knowledge store. Prompts and retrieved text are inputs, not knowledge.
• Parameters are frozen at inference. Nothing a user sends writes back into them.
• Weight memory ≈ parameters × bytes per parameter. FP16/BF16 at 2 bytes gives the double-the-billions shortcut: 7B ≈ 14 GB, 70B ≈ 140 GB. Weights are the floor, not the total.
• Parameters vs hyperparameters vs context vs decoding settings are four distinct things; only the first is counted in a model's size.
• Feed-forward layers hold roughly two thirds of a transformer block's parameters, attention roughly one third.
• Sort every customisation technique by does it change weights — that single test resolves a large class of scenario questions.
• Fine-tune for behaviour, retrieve for knowledge, and expect the best scenario answers to combine them when both are required.
• Weights cannot cite. Provenance is a property of the system you build around the model.

15. Next: tensor shapes in transformers — batch, sequence, and hidden size

You know what the numbers are and how many of them there are. You do not yet know what shape the data flowing through them takes, and that shape is the vocabulary you need to read any model config, error message or architecture diagram for the rest of the course.

Next: 01-03 Tensor shapes in transformers: batch, sequence, and hidden size — the (batch, sequence, hidden) triple that every layer in the stack consumes and returns, and the reason a shape mismatch is the most common error you will ever see in this work.