Neural Network Basics: Neurons, Activation Functions, and the Training Loop

Identity statement: a neuron computes activation(Σ wᵢxᵢ + b) — it takes a set of inputs, multiplies each by a learned weight, sums the results, adds a learned bias term, and passes that sum through a nonlinear activation function. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states this formula directly.

Unpack the formula piece by piece. x₁, x₂, ..., xₙ are the neuron's inputs — they could be raw pixel values, the outputs of a previous layer's neurons, or a tokenized word's embedding, depending on where in a network this neuron sits. Each input xᵢ has an associated weight wᵢ, a learned parameter that determines how strongly that particular input influences this neuron's output — a weight of zero means that input is effectively ignored, a large weight means it dominates. Σ wᵢxᵢ is the weighted sum of every input, and b (bias) is an additional learned constant added to that sum, letting the neuron shift its output up or down independent of the inputs. The entire expression Σ wᵢxᵢ + b — before the activation function — is sometimes called the neuron's pre-activation or its logit.

When it matters: any scenario that asks what a single computational unit inside a network does, or asks you to identify what "weights" and "bias" actually are as opposed to inputs or outputs.

Why the activation function must be nonlinear

The activation function is not optional decoration — it is the single component that makes a multi-layer network capable of learning anything a single-layer network could not. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states the consequence directly: "without it, stacked layers collapse into a single linear function."

Here is the mechanism. A linear function composed with another linear function is still just a linear function — stacking y = a(bx + c) + d where every step is linear produces something that can always be rewritten as one single linear equation, y = mx + k, no matter how many layers you stack. If every neuron in a network computed only Σ wᵢxᵢ + b with no nonlinear step afterward, then a network with a hundred layers would have exactly the same representational power as a network with one layer — all that extra depth would buy nothing, because the whole stack collapses algebraically into one linear transformation.

The activation function breaks that collapse by introducing a nonlinearity between layers. Common choices: ReLU (rectified linear unit, max(0, x) — zero for negative inputs, identity for positive ones), GELU (a smoother variant used heavily in transformer architectures), sigmoid (squashes any input into the range 0 to 1, historically used for binary outputs), tanh (squashes into −1 to 1), and softmax (converts a vector of raw scores into a probability distribution, used at a classification network's final layer). [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this same set directly.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names the training loop's mechanism in a single dense sentence worth unpacking term by term: "A loss function measures error... Gradient descent updates weights in the direction that reduces the loss, scaled by the learning rate. Backpropagation computes the gradients via the chain rule; the optimizer (SGD, Adam) applies the update."

L1 — Intuition

Imagine standing on a hillside in fog, trying to reach the valley floor (the lowest possible loss) by feel alone. You cannot see the whole landscape, but you can feel which direction is downhill from where you are standing right now — that local downhill direction is the gradient, computed by backpropagation. Gradient descent is the decision to step in that downhill direction. The learning rate is how big a step you take. Too large a step and you might overshoot the valley and end up higher on the other side; too small a step and you move toward the bottom so slowly that hours pass with barely any progress. The optimizer is the specific stepping strategy — a plain, fixed-size step every time (SGD) or a strategy that adapts its step size and direction based on the terrain it has recently crossed (Adam).

L2 — Mechanism

The loss function is computed first, at the end of a forward pass: the model makes a prediction, that prediction is compared to the true label or value, and the loss function turns the gap between them into a single number. Cross-entropy is the loss for classification and next-token prediction; MSE is the loss for regression. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names both explicitly.

Backpropagation then works backward from that single loss number through every layer of the network, computing how much each individual weight contributed to the error — mathematically, the partial derivative of the loss with respect to each weight. This is where the chain rule from calculus does the real work: because a deep network is a composition of many functions (layer after layer), the chain rule lets you compute each weight's contribution to the final loss by multiplying together the local derivatives along the path from that weight to the output, rather than needing some entirely separate calculation for every single weight in the network.

Gradient descent is the update principle that says: move each weight a small amount in the direction that decreases the loss — which is the direction opposite the gradient backpropagation just computed, since the gradient points toward increasing loss by definition. The learning rate scales exactly how large that step is: a small positive number, commonly somewhere between 0.1 and 0.00001 depending on the setup, multiplied against the gradient before it is subtracted from each weight.

The optimizer is the concrete algorithm that turns "the gradient, scaled by the learning rate" into the actual number subtracted from each weight. Plain SGD (stochastic gradient descent) does exactly that: new_weight = old_weight − learning_rate × gradient, computed on a random mini-batch of examples rather than the full dataset at once (the "stochastic" part). Adam is a more commonly used optimizer in practice, because it adapts its effective step size per-parameter based on recent gradient history, which tends to converge faster and more reliably than plain SGD's fixed step size across a wide range of architectures.

L3 — Why all five terms recur together, and how they fail

Every one of these five terms can fail independently, and recognizing which failure a described symptom points to is the exam-relevant depth beyond simple definitions.

A learning rate set too high causes the loss to oscillate wildly or diverge outright — each step overshoots the low point of the loss landscape so badly that the next step starts from an even worse position than before, and the loss can grow without bound rather than shrink. A learning rate set too low causes training to crawl — the loss does decrease, but so slowly that a training run that should finish in hours instead needs weeks, or appears to have stalled entirely within any reasonable observation window. A vanishing gradient — the gradient backpropagation computes becoming vanishingly small as it is multiplied backward through many layers — means gradient descent's updates become negligibly small regardless of the learning rate, and the network's early layers effectively stop learning; this is precisely the problem residual connections (M1-07) exist to mitigate. The wrong loss function for the task — MSE on a classification problem, say — produces a number that technically decreases during training but does not actually measure the thing you care about, so a falling loss curve can coexist with a model that is not becoming meaningfully better at its real task.

The worked example in the next section shows one training step computed from a single example, which makes the arithmetic legible but understates how training actually runs in practice. Real training loops process data in three nested units, and each has a specific name the exam expects you to recognize.

A batch (or mini-batch) is a group of training examples processed together in one forward-and-backward pass, rather than one example at a time. Instead of computing a gradient from a single example's loss, the loss is averaged (or summed) across every example in the batch before backpropagation runs — meaning one weight update reflects the combined signal from many examples at once, not just one. Batch sizes commonly range from a handful of examples to several thousand, depending on available GPU memory and the specific architecture.

An epoch is one complete pass through the entire training dataset — if a dataset has 10,000 examples and a batch size of 100, one epoch consists of 100 batches, each producing one weight-update step. Training typically runs for many epochs, repeating passes over the same dataset, because a single pass is rarely enough for the weights to converge to a good solution.

Why batching rather than either extreme. Processing one example at a time (batch size of 1) produces a very noisy gradient estimate — one example's quirks can swing the weight update in a direction that does not reflect the broader dataset at all, and it also fails to exploit a GPU's ability to process many examples in parallel, wasting most of the hardware's available throughput. Processing the entire dataset in one batch (sometimes called full-batch or "vanilla" gradient descent) produces the most accurate possible gradient estimate at each step, but for any dataset larger than trivially small, this becomes both memory-prohibitive (every example's intermediate activations must fit in memory simultaneously) and slow to iterate, because a full epoch's worth of computation happens before a single weight update occurs. Mini-batching is the practical middle ground: a large enough batch to average out individual-example noise into a reasonably stable gradient estimate, small enough to fit in GPU memory and allow many weight updates per epoch rather than just one.

Dataset: 10,000 image-caption pairs, batch size = 100

One epoch = 10,000 / 100 = 100 batches = 100 weight-update steps

Training for 20 epochs = 20 x 100 = 2,000 total weight-update steps,
each one an application of the four-step process from section 2's
worked example, run on 100 examples' averaged loss instead of one.

This is a constructed scenario with illustrative numbers. The takeaway to generalize: every "step" referenced in a loss curve or a training log is one batch's worth of gradient computation and weight update, and "epoch" counts how many times the entire dataset has been swept through — the two units answer different questions (how many updates so far, versus how much of the data has been seen), and scenarios sometimes deliberately conflate them to test whether you keep the distinction straight.

Everything in sections 1 through 3 describes the training loop for a network processing a single kind of input. A multimodal network — one processing text and images together, say — runs the exact same five-term loop (loss function, backpropagation, gradient descent, learning rate, optimizer) with one structural difference worth naming explicitly here, ahead of its full treatment later in this module.

A multimodal network typically has separate encoder pathways for each modality — a text-processing branch built from one stack of layers, an image-processing branch built from a different stack — that eventually combine, at some point in the architecture, into a shared representation the rest of the network continues to process. Backpropagation still computes a gradient for every weight in the entire network, chain-rule fashion, from the final loss back to the very first layer of every branch — the mechanism from section 2 does not change. What changes is that the gradient reaching the text branch and the gradient reaching the image branch can behave very differently from each other, because each branch is processing a different kind of input, at a different natural scale, through a different architecture.

This is the direct setup for two lessons later in this module: M1-08 covers what happens when a multimodal network's loss function itself has to combine signal from more than one modality (a composite, weighted-sum loss), and M1-09 covers the specific training-stability techniques — normalization, learning-rate warmup, loss weighting, gradient clipping — that keep one modality's branch from training faster or more aggressively than another's, purely as a side effect of how each modality's data happens to be scaled or structured. Nothing about the core five-term training loop changes for multimodal training; what changes is that you now have to watch multiple branches' worth of gradient behavior simultaneously, rather than one.

A tiny network has one neuron with two inputs, current weights w₁ = 0.5, w₂ = -0.3, and bias b = 0.1. It receives inputs x₁ = 2.0, x₂ = 1.0, and the activation function is ReLU.

Step 1: Forward pass
  pre-activation = (0.5 × 2.0) + (-0.3 × 1.0) + 0.1
                 = 1.0 - 0.3 + 0.1 = 0.8
  activation (ReLU) = max(0, 0.8) = 0.8      <- the neuron's output

Step 2: Compute the loss
  Suppose the true target value is 1.2, and this is a regression
  problem using MSE for a single example:
  loss = (prediction - target)^2 = (0.8 - 1.2)^2 = 0.16

Step 3: Backpropagation computes the gradient
  d(loss)/d(prediction) = 2 x (prediction - target) = 2 x (0.8 - 1.2) = -0.8
  Since ReLU's derivative is 1 when the pre-activation is positive (0.8 > 0),
  the gradient flows through unchanged to the pre-activation.
  d(loss)/d(w1) = d(loss)/d(pre-activation) x x1 = -0.8 x 2.0 = -1.6
  d(loss)/d(w2) = -0.8 x 1.0 = -0.8
  d(loss)/d(b)  = -0.8 x 1   = -0.8

Step 4: Gradient descent update, with learning rate = 0.05, using SGD
  new_w1 = 0.5  - 0.05 x (-1.6) = 0.5 + 0.08  = 0.58
  new_w2 = -0.3 - 0.05 x (-0.8) = -0.3 + 0.04 = -0.26
  new_b  = 0.1  - 0.05 x (-0.8) = 0.1 + 0.04  = 0.14

This is a constructed scenario with illustrative numbers, not a measurement from any real trained network. Notice the direction of every update: because the gradient at each weight was negative, gradient descent's weight − learning_rate × gradient formula adds a small positive amount to each weight — moving every weight in the direction that would have made this specific prediction closer to the target of 1.2, exactly as the "descent" in gradient descent promises. Run this same four-step process across millions of examples, and this is, mechanically, the entire process by which a neural network "learns."

A team reports the following training loss values, recorded every 100 steps, for three different learning-rate settings on the same architecture and data:

Setting A (learning rate too high):
  step 100: loss = 2.1   step 200: loss = 5.8   step 300: loss = 14.2
  -> Loss is INCREASING, and accelerating. Classic divergence from an
     overly large learning rate — each step overshoots so badly that
     the next step starts from a worse position.

Setting B (learning rate too low):
  step 100: loss = 2.09  step 200: loss = 2.07  step 300: loss = 2.05
  -> Loss is decreasing, but by a tiny amount each time. At this rate,
     reaching a genuinely low loss would take an impractical number
     of steps.

Setting C (learning rate well-tuned):
  step 100: loss = 1.8   step 200: loss = 0.9   step 300: loss = 0.4
  -> Loss is decreasing steadily and substantially. This is the
     healthy pattern a correctly tuned learning rate produces.

Constructed scenario, illustrative numbers. The diagnostic habit to take from this: a rising or wildly oscillating loss almost always points to the learning rate being too high (or, less commonly, a genuine bug in the loss computation itself), while a loss that decreases but barely moves almost always points to the learning rate being too low, or occasionally to a vanishing-gradient problem starving the early layers of any meaningful update at all. Both symptoms live at the same layer of the training loop — the size and direction of the weight update — which is exactly why the learning rate is called out elsewhere in this module as the single most sensitive hyperparameter.

⭐ THE EARNED INSIGHT The five training-loop terms are not five independent facts to memorize — they are one causal chain, and a scenario that seems to be testing one term is very often testing whether you can trace the failure back through the whole chain. A loss that will not fall could be the loss function itself (measuring the wrong thing), backpropagation (a genuine implementation bug, rare in practice), the learning rate (too high or too low), or the optimizer (a poor fit for this architecture). Distinguishing between them requires knowing what each term is actually responsible for, not just what it is named.

The table below names each specific failure and its fix in turn.

Every row maps a specific symptom to a specific fix. Exam weight explains why these five terms are worth holding precisely.

Core Machine Learning and AI Knowledge carries 20% exam weight, and [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) frames the neuron formula and the training-loop terms as foundational vocabulary objective 1.10 tests directly, load-bearing for the residual-connection, multimodal-loss, and training-stability lessons that follow later in this same module. A scenario that names "the learning rate" or "backpropagation" without further elaboration is testing whether you know precisely what that one term is responsible for, distinct from its four neighbors in the training loop.

The question tends to arrive in a small number of recognizable shapes.

1. Term-to-function matching. "Which term computes the gradient via the chain rule?" with the keyed answer naming backpropagation specifically, against distractors offering gradient descent or the optimizer.
2. Diverging-versus-stalling diagnosis. A scenario describes a loss curve behaving unusually, and asks what hyperparameter to check first — almost always the learning rate.
3. Why nonlinearity matters. "What happens if activation functions are removed from a deep network?" with the keyed answer naming the collapse into a single linear function.
4. SGD versus Adam recognition. A scenario names one of the two optimizers and asks for its defining trait — a fixed step size for SGD, an adaptive per-parameter step size for Adam.

What the distractors typically look like

The reliable distractor families: swapping backpropagation's "compute the gradient" job with gradient descent's "decide the update direction" job, since both terms sound similar and belong to the same pipeline; asserting that a higher learning rate is unconditionally better because it trains faster; and describing a linear stack of layers with no activation function as still gaining representational power from added depth.

The network's representational power collapses to that of a single linear layer, no matter how many layers you originally stacked. Because composing linear functions produces another linear function, a hundred-layer network with no nonlinear activation anywhere computes exactly the same class of function as a single-layer network — every extra layer of depth contributes nothing that could not already be expressed by one linear transformation. The nonlinear activation function is what lets each additional layer actually expand what the network can represent, which is why it is never optional in a genuinely deep architecture.

Adam adapts its effective step size for each individual parameter based on that parameter's recent gradient history, rather than applying one fixed step size uniformly to every weight the way plain SGD does. In practice this tends to make training converge faster and more reliably across a wide range of architectures and datasets, because parameters that need larger or smaller updates at a given point in training get them automatically, rather than all parameters being forced through the same fixed-size step. Plain SGD is simpler and sometimes still preferred for its own reasons — including cases where its simpler update behavior generalizes slightly better — but Adam's adaptive behavior is why it is the more commonly reached-for default in modern practice.

A batch is a group of training examples processed together in one forward-and-backward pass, producing one weight-update step; an epoch is one complete pass through the entire training dataset, made up of many batches in sequence. A batch answers "how many examples contributed to this one weight update," while an epoch answers "how many times has the model seen the whole dataset so far." A training run with a batch size of 100 on a 10,000-example dataset produces 100 weight-update steps per epoch, and typically runs for many epochs — repeated full passes over the same data — before the weights converge to a good solution, which is why both units are tracked separately in a training log rather than collapsed into one number.

• A neuron computes activation(Σ wᵢxᵢ + b) — a weighted sum of inputs plus a bias, through a nonlinear activation function.
• Without a nonlinear activation, stacked layers collapse into a single linear function, regardless of depth — the activation function is what makes depth actually matter.
• The training loop runs on five terms: the loss function measures error, backpropagation computes gradients via the chain rule, gradient descent decides the update direction, the learning rate scales the step size, and the optimizer (SGD, Adam) applies the actual update.
• A learning rate too high causes divergence or oscillation; too low causes impractically slow training — the single most sensitive hyperparameter in this whole loop.
• A falling loss does not automatically mean the model is improving at its real task — the loss function has to be the right one for the job.
• Adam adapts its step size per-parameter based on gradient history; plain SGD applies one fixed step size — which is why Adam is more commonly reached for in practice.
• A batch produces one weight update; an epoch is one full pass through the dataset made of many batches — the two answer different questions and are tracked separately.
• Multimodal training runs the same five-term loop, with the added complexity of watching multiple modality-specific branches' gradient behavior simultaneously, a theme this module returns to directly in M1-08 and M1-09.

This lesson covered a single neuron and the loop that trains it. What it did not cover is the specific architecture — convolutions — that lets a network process grid-shaped data like images and spectrograms efficiently, rather than treating every pixel as an unrelated, independent input. M1-06 covers convolutions and the building blocks of vision models next, the workhorse behind every image-processing component this course discusses, including the U-Net this exam's Domain 6 material builds on.