Multimodal Loss Functions: Cross-Entropy, Contrastive, Reconstruction, Adversarial, and Composite

Identity statement: multimodal training often optimizes more than one objective at once, and the specific loss function used depends entirely on what the model is being trained to do and what kind of target it is being compared against. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) frames this directly: "Multimodal training often optimizes more than one objective at once," and names five common patterns.

When it matters: any scenario that names a training setup by its data and its objective and asks which loss family applies, or any scenario describing more than one objective being trained simultaneously.

The organizing question behind every row in this table is the same: what is the model's output being compared against, and what relationship is the loss trying to enforce between them? Cross-entropy compares a predicted distribution against one correct answer. Contrastive loss compares many pairs against each other, relatively, rather than each prediction against one fixed target. Reconstruction loss compares an output against the very input that produced it. Adversarial loss compares an output against a second network's judgment of it, rather than against any fixed target at all. Composite loss does not introduce a new comparison — it combines several of the first four.

M1-05 and the earlier lessons in this module already named cross-entropy as the loss for classification and next-token prediction, and it recurs here as the first named multimodal loss family precisely because multimodal classification tasks — is this image-caption pair a match, which of several categories does this fused text-image input belong to — reduce to the exact same mechanism: −log(probability assigned to the correct class), averaged across a batch. Nothing about cross-entropy changes when the input feeding the classifier happens to be a fused, multimodal representation rather than a single-modality one; the loss only ever sees the final predicted distribution and the correct label, regardless of how many modalities contributed to producing that distribution.

Identity statement: a contrastive loss pulls the embeddings of matched pairs together and pushes the embeddings of mismatched pairs apart, within the same batch — CLIP's text-image alignment is the canonical multimodal instance of this mechanism. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this directly: "Contrastive (e.g. InfoNCE) — CLIP-style text↔image alignment — Pull matched pairs together, push mismatched apart in a shared embedding space."

L1 — Intuition

Cross-entropy judges a single prediction against one fixed correct answer, in isolation. Contrastive loss judges a prediction relative to every alternative available in the same batch — it does not ask "is this the correct answer" in a vacuum, it asks "does this pair score higher than every other possible pairing I could have compared it against." The name "contrastive" names exactly this: learning happens by contrast against alternatives, not by matching a fixed target alone.

L2 — Mechanism

Take a batch of N image-text pairs. Encode every image and every text with their respective encoders, producing N image embeddings and N text embeddings in a shared space. Form the full N-by-N grid of every possible image-text combination: N of those combinations are true matches (the diagonal, where image i is paired with its own caption i), and N² − N are mismatches. The contrastive loss pushes the similarity score of every true match up and the similarity score of every mismatch down, simultaneously, across the entire batch — not one pair at a time, but all of them together in a single training step. A larger batch supplies more mismatched pairs to contrast against per true match, which is part of why contrastive pretraining tends to use very large batch sizes: more negative examples per positive example generally sharpens the alignment the loss produces.

L3 — Why contrastive loss needs no human-written label

The exam-relevant depth here is recognizing why contrastive loss is the mechanism behind self-supervised multimodal pretraining, connecting directly back to M1-01's paradigm material. A contrastive loss's "correct answer" for a given image is simply whichever caption in the batch was already paired with it — information the data already carried, requiring no separate human annotation step. This is precisely the self-supervised pattern M1-01 introduced: a label the data supplies for itself, at essentially zero additional labeling cost, which is exactly how CLIP-style pretraining scales to hundreds of millions of pairs without a human ever writing a classification label for any single one of them.

Identity statement: a reconstruction loss (commonly MSE or L1) penalizes the difference between a model's output and its own original input, used when a model's job is to recreate or denoise something rather than classify or align it. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this pattern for "Autoencoders, diffusion denoising."

An autoencoder compresses an input down to a smaller representation and then reconstructs it back to its original form; the reconstruction loss measures how close the reconstruction is to the original input that was fed in. Diffusion denoising uses a structurally similar loss with a different target: rather than reconstructing the exact original input, a diffusion model's denoising network is trained to predict the noise that was added at a given step, and the reconstruction-style loss penalizes the gap between the predicted noise and the actual noise that was injected. Both cases share the defining feature of this loss family: the comparison target is derived from the model's own input (or a known transformation of it), not from a separately collected label the way cross-entropy's correct-class target is.

This is the loss family most directly responsible for the "what it does not prove" caveat worth internalizing early: a low reconstruction loss proves a model can recreate inputs similar to what it has seen, not that it has learned any semantically meaningful representation of what those inputs mean — an autoencoder can achieve excellent reconstruction loss while its compressed representation carries little of the structure a downstream task would actually need, which is a distinct question reconstruction loss alone cannot answer.

Identity statement: an adversarial loss trains two networks — a generator and a discriminator — against each other in a min-max game: the generator tries to produce outputs the discriminator cannot distinguish from real data, and the discriminator tries to get better at telling generated outputs apart from real ones. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this pattern directly for GANs.

Unlike every other loss family in this lesson, adversarial loss has no fixed target at all — there is no correct class, no matched pair, no original input to reconstruct. Instead, the generator's loss is defined relative to a second, simultaneously-training network's current judgment, which means the target the generator is optimizing against is itself changing throughout training, as the discriminator improves. This is the single structural fact that makes adversarial training notoriously harder to stabilize than the other four loss families: cross-entropy's correct-class target, contrastive loss's batch of pairs, and reconstruction loss's original input are all fixed, known quantities at every training step, while a GAN's generator is chasing a moving target that is itself being trained at the same time.

The generator and discriminator are optimizing opposing objectives — the generator wants the discriminator to be wrong, the discriminator wants to be right — which is exactly what "min-max" names: one network minimizing a quantity the other is simultaneously trying to maximize.

Why "the discriminator got better" can mean the generator's loss gets worse for a good reason

A subtlety worth flagging explicitly, because it is easy to misread as a training failure: if a GAN's generator loss rises partway through training, that is not automatically evidence that training has gone wrong. Because the discriminator is also improving throughout training, a generator that is producing genuinely better, more realistic outputs than it did earlier can still see its loss rise if the discriminator has improved faster at that point and become harder to fool, even with better generated images than before. This is a direct consequence of the moving-target property described above: unlike a fixed-target loss, where a rising number reliably means the model is getting worse at a stable task, an adversarial loss's number is confounded with how much the opposing network has changed, which is exactly why GAN training is typically monitored with additional signals — sample quality inspected visually, or a separate metric like FID (covered in a later module) computed on the generator's outputs — rather than trusting the raw adversarial loss curve alone the way a cross-entropy loss curve can more directly be trusted.

Identity statement: a composite (weighted-sum) loss combines two or more of the above loss families into a single number the optimizer minimizes, using weights (commonly written λ) to control how much each component contributes to the total. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this as the pattern for "Multi-task multimodal models."

total_loss = λ1 x loss_1 + λ2 x loss_2 + ... + λn x loss_n

A multimodal model might, for instance, combine a cross-entropy loss for a text classification sub-task with a reconstruction loss for an image-autoencoding sub-task, training both simultaneously inside one network, because both sub-tasks share some of the same underlying representation and jointly training them can produce a better shared representation than training either alone.

Why the weights are not optional details

The λ weights in a composite loss are not a cosmetic detail — they determine which component's gradient dominates the shared parameters both losses flow through. If one loss's raw numeric scale happens to be much larger than another's (a common, easy-to-overlook consequence of the two losses being computed differently — cross-entropy's typical range differs from a reconstruction loss's typical range, for instance), an unweighted sum can let the larger-scale loss dominate training entirely, effectively starving the smaller-scale loss's objective of any meaningful gradient signal, regardless of how important that objective actually is to the model's intended use. This is the exact mechanism M1-09's training-stability material addresses in depth — loss weighting exists specifically because an unweighted composite loss is not a neutral default, it is a specific, nameable failure mode that lets whichever modality's or sub-task's loss happens to be easier, or larger in scale, dominate the ones that are not.

⭐ THE EARNED INSIGHT A composite loss's weights are not a tuning knob you reach for only if something goes wrong — an unweighted composite loss is itself a choice, and it is the choice most likely to go wrong silently. Setting every λ to 1 looks like "no special treatment," but it is actually a specific bet that every loss component's raw numeric scale already happens to be comparable to every other's, which is rarely true by accident. Treating "no weighting" as the neutral default, rather than as an unexamined assumption, is exactly the gap this exam's training-stability material is built to close.

Where the λ values themselves come from

At the recognition depth this exam expects, it is enough to know that λ weights exist and what they do — but naming, briefly, where they typically come from helps distinguish this from an arbitrary or unprincipled choice. The most common approach is manual tuning against a held-out validation set: try a handful of candidate weightings, observe which one produces the best balance of performance across every sub-task the model is meant to serve, and select accordingly, which is itself an application of M1-01's validation-set discipline to a hyperparameter rather than to the model's ordinary weights. A second, more principled family of approaches learns the weights themselves during training, adjusting each λ based on signals like each loss component's own gradient magnitude or its recent rate of improvement, so that the balance can shift dynamically over the course of training rather than being fixed once in advance. Both approaches share the same underlying goal identified in this section: prevent one component's raw scale, rather than its genuine importance to the task, from deciding how much of the optimizer's effort it receives.

A second worked pattern: composite losses inside a single modality's own training

Composite losses are not exclusively a cross-modality phenomenon — the same weighted-sum pattern shows up whenever a single model is asked to satisfy more than one objective, even within one modality. A text-generation model might combine a cross-entropy loss for next-token accuracy with an auxiliary loss encouraging output diversity, to avoid the degenerate, repetitive outputs a pure cross-entropy objective can sometimes favor. The mechanism, the risk, and the fix are identical to the cross-modality case: without deliberate weighting, whichever component has the larger raw scale dominates, regardless of modality boundaries. The "multimodal" framing in this lesson's title names the setting this exam emphasizes most, but the underlying arithmetic of composite loss generalizes to any multi-objective training setup, single-modality ones included.

The five sections above described each family individually; this section collects the distinguishing questions into one sequence, because scenario questions rarely name a loss family directly — they describe a setup and expect you to derive the family from what the model is being asked to do.

Work through the questions roughly in this order for a genuinely ambiguous scenario, because several of them can look superficially similar. A setup describing "matched and mismatched pairs" could sound contrastive at first read, but if those pairs were pre-labeled by a human annotator rather than drawn relative to a batch's own contents, the first question's answer is "yes" and the correct classification is cross-entropy, not contrastive — this exact distinction is precisely what the next section's Setup A versus Setup B is built to test.

Supervised, self-supervised, and the loss families that implement each

M1-01 introduced the four learning paradigms — supervised, unsupervised, self-supervised, reinforcement — as a way of classifying where a training signal comes from. It is worth connecting that paradigm-level vocabulary to this lesson's loss-family vocabulary explicitly, because they answer related but distinct questions, and conflating them is a subtle, exam-relevant trap.

Cross-entropy is typically the loss used inside supervised learning, when the labels are pre-collected by a human annotator, or inside self-supervised learning, when the "label" is derived automatically from the data (next-token prediction being the canonical example — the next word is not a separately annotated label, but the loss computing that comparison is still ordinary cross-entropy). Contrastive loss is almost always the mechanism behind self-supervised multimodal pretraining specifically, because its batch-relative target requires no separate annotation step at all — this is why CLIP is self-supervised (per M1-01's framing) and uses a contrastive loss (per this lesson's framing): the paradigm names the training signal's origin, the loss family names the specific arithmetic that signal gets turned into. Adversarial loss does not map cleanly onto any single one of the four paradigms in the same way — a GAN's generator receives its training signal from a discriminator's judgment, which is itself being trained, making the adversarial setup structurally distinct from the clean supervised/self-supervised/unsupervised/reinforcement split, though it is most often discussed alongside self-supervised and unsupervised generative modeling in practice.

The generalizable point: "what paradigm is this" and "what loss family is this" are two separate, complementary questions a single scenario can ask, and answering one does not automatically answer the other — a self-supervised setup could in principle use cross-entropy (next-token prediction) or contrastive loss (CLIP), and knowing which loss family applies requires the section-1-through-6 mechanism-level detail this lesson has been building, not just the higher-level paradigm classification from M1-01.

A team describes four separate training setups. Identify the loss family each one uses, from the data and objective alone.

Setup A: "We train a model to predict whether a product review's
text and its attached photo describe the same product, using a
labeled dataset of matched and mismatched review/photo pairs, with
a fixed 'match' or 'no match' label for every pair."
  -> CROSS-ENTROPY. There is a single, fixed, human-labeled correct
     answer per example (match / no match) -- this is ordinary
     binary classification, not contrastive learning, because the
     "negative" examples are pre-labeled rather than drawn from
     within a batch by contrast.

Setup B: "We train an image encoder and a text encoder jointly on
scraped image-caption pairs with no separate match/mismatch labels,
using every other caption in the same batch as an implicit negative
for each image."
  -> CONTRASTIVE LOSS. No separate label exists per pair; matches
     and mismatches are drawn from the batch's own structure, exactly
     the CLIP-style pattern from section 3.

Setup C: "We train a network to compress a 512x512 image down to a
128-dimensional vector and then reconstruct the original image from
that vector, penalizing the pixel-wise difference between the
reconstruction and the original."
  -> RECONSTRUCTION LOSS. The comparison target is the model's own
     input, not a separately collected label -- the autoencoder
     pattern from section 4.

Setup D: "We train a network to generate realistic product photos,
and a second network to distinguish generated photos from real ones,
updating both networks against each other every training step."
  -> ADVERSARIAL LOSS. Two networks with opposing objectives, no
     fixed target for the generator -- the GAN pattern from section 5.

This is a constructed scenario with illustrative descriptions, not measurements from any real training pipeline. The diagnostic habit worth taking forward: read for whether a fixed, pre-existing label exists per example (cross-entropy), whether the "correct answer" is defined only relative to other examples in the same batch (contrastive), whether the target is the model's own input (reconstruction), or whether there is no fixed target at all because a second network's shifting judgment defines the objective (adversarial).

A multimodal model trains simultaneously on a text classification sub-task (cross-entropy) and an image reconstruction sub-task (MSE), sharing an underlying representation. Track the two raw loss values and a composite total across three training checkpoints, first with equal, unweighted weights.

Checkpoint 1: text_loss (cross-entropy) = 1.8    image_loss (MSE) = 42.0
  Unweighted total = 1.0 x 1.8 + 1.0 x 42.0 = 43.8

Checkpoint 2: text_loss = 1.7    image_loss = 35.0
  Unweighted total = 1.0 x 1.7 + 1.0 x 35.0 = 36.7

Checkpoint 3: text_loss = 1.65   image_loss = 20.0
  Unweighted total = 1.0 x 1.65 + 1.0 x 20.0 = 21.65

The unweighted total is falling steadily across checkpoints, which looks healthy at a glance. Look at each component separately, though: image_loss fell from 42.0 to 20.0 — more than half — while text_loss barely moved, from 1.8 to 1.65, a change of less than 10%. The optimizer, minimizing the unweighted sum, is spending almost all of its effective effort on the image reconstruction sub-task, because that component's much larger raw scale means it contributes far more to the total gradient than the text classification component does, regardless of how much either sub-task's relative performance actually needs to improve.

Same three checkpoints, reweighted with lambda_text = 10, lambda_image = 1,
chosen to bring both components' contribution to the total onto a
comparable scale:

Checkpoint 1: 10 x 1.8 + 1 x 42.0 = 18.0 + 42.0 = 60.0
Checkpoint 3: 10 x 1.65 + 1 x 20.0 = 16.5 + 20.0 = 36.5

Constructed scenario, illustrative numbers. With the reweighting, the text component now contributes a comparable share of the total loss to the image component, which in practice tends to push the optimizer to spend more effort improving the text classification sub-task than the unweighted version did — the reweighted total is not "more correct" in any absolute sense, but it changes which component the optimizer is implicitly prioritizing, which is the entire point of choosing the λ weights deliberately rather than defaulting to 1 for every term. This is the concrete arithmetic behind section 6's warning: an unweighted composite loss is a specific choice with a specific, predictable consequence, not a neutral absence of choice.

A scenario describes: "CLIP is trained by encoding a batch of images and their captions, then optimizing so that each image's embedding is closer to its own caption's embedding than to any other caption's embedding in the batch, and vice versa." Which loss family, and is any other family also involved?

Primary mechanism: pulling matched pairs together, pushing mismatched
pairs apart, using the batch's own structure as the source of
negatives -- this is CONTRASTIVE LOSS, exactly the section 3 mechanism.

Is cross-entropy also involved? In practice, yes, at the implementation
level: contrastive losses like InfoNCE are frequently implemented as a
cross-entropy loss applied to the similarity scores treated as logits
over "which caption in the batch is the right one" -- but this is an
IMPLEMENTATION DETAIL of how the contrastive objective's mechanics get
computed, not evidence that CLIP is "really" a classification task.
The defining property remains the CONTRASTIVE structure: the correct
answer is defined relative to the batch, not by a separately labeled
fixed target.

Constructed scenario, illustrative reasoning. This worked example names the trap most likely to cost a point: seeing cross-entropy arithmetic inside a contrastive loss's implementation and concluding the loss "is" cross-entropy. The defining, exam-relevant property of a contrastive loss is that its target is relative to other examples in the same batch, not a fixed external label — that property should drive the classification, regardless of which specific numerical formula happens to compute the final gradient.

The common-mistakes table below collects this trap alongside every other one this lesson has named.

Every row maps a specific symptom to a specific fix. Exam weight explains why this taxonomy is worth holding precisely.

Core Machine Learning and AI Knowledge carries 20% exam weight, and [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names all five loss families explicitly as the multimodal-specific stretch this domain adds on top of classic ML foundations — a direct signal that this exact taxonomy is testable material, not incidental background. The cross-domain weight is significant too: CLIP's contrastive mechanism recurs at full depth in Domain 4 (M4-03, 15% weight) and its loss feeds Domain 3's diffusion-conditioning material, so getting the taxonomy right here pays off in at least two later domains.

The question tends to arrive in a small number of recognizable shapes.

1. Loss-family identification from a described setup. A scenario names a training objective and its data, and asks which loss family applies — the keyed answer is read off what the target is computed against, per section 1's organizing table.
2. The contrastive-loss mechanism, directly. "How does CLIP's training loss work?" with the keyed answer naming the pull-together/push-apart batch mechanism, against distractors describing a fixed classification target.
3. Composite-loss imbalance. A scenario describes one sub-task's loss dominating a multi-task model's training, asking for the cause — the keyed answer names unweighted summing across differently-scaled components.
4. Adversarial-versus-other-loss distinction. A scenario asks why GAN training is harder to stabilize than a classifier's — the keyed answer names the shifting, discriminator-relative target.

What the distractors typically look like

The reliable distractor families: describing contrastive loss as requiring pre-labeled negative pairs; treating a low reconstruction loss as sufficient evidence of a good learned representation; and, for composite losses, describing unweighted summing as the safe or correct default rather than a specific, failure-prone choice.

Not in the defining sense, even though a contrastive loss like the one CLIP uses is frequently implemented using cross-entropy arithmetic internally, applied to similarity scores treated as classification logits over "which item in the batch is the correct match." What actually defines a loss as contrastive rather than as ordinary classification is what the correct answer is computed relative to: a contrastive loss's target is defined by the other examples present in the same training batch (pull the true pair together, push every other pairing in the batch apart), while an ordinary cross-entropy classification loss's target is a fixed, pre-existing label attached to that one example alone, independent of whatever else happens to be in the batch. CLIP's mechanism is contrastive by this defining property, regardless of which specific formula computes the gradient underneath it.

Because summing several loss components with a weight of 1 each is only a neutral choice if every component's raw numeric scale already happens to be comparable — an assumption that is rarely true by accident, since different loss families (cross-entropy, reconstruction MSE, adversarial) are computed differently and naturally land in different numeric ranges. When one component's scale is much larger than another's, the optimizer's gradient is dominated by whichever component contributes the larger raw numbers, regardless of which objective actually matters more or is further from being solved. This can silently starve a smaller-scale but important objective of meaningful training signal, which is exactly the failure mode M1-09's loss-weighting material exists to prevent by choosing λ weights deliberately rather than defaulting to an unexamined 1 for every term.

Because a GAN's generator has no fixed target to optimize against — its loss is defined relative to a discriminator that is itself being trained and changing at the same time, so the "correct" direction to move in at any given step depends on the discriminator's current, temporary state rather than on a stable, known quantity. Cross-entropy's correct-class label, contrastive loss's batch of pairs, and reconstruction loss's original input are all fixed at every training step regardless of how far along training is; a GAN's target shifts continuously as both networks improve, which can produce oscillating or non-converging training dynamics that a fixed-target loss does not have the same structural tendency toward. This is also why adversarial training is typically monitored using additional signals beyond the raw loss curve — inspecting generated samples directly, or a separate quality metric computed on the outputs — rather than trusting a rising or falling adversarial loss number alone to indicate whether training is going well.

• Five loss families cover multimodal training: cross-entropy (classification), contrastive (batch-relative alignment, CLIP's mechanism), reconstruction (comparison against the model's own input), adversarial (a shifting, discriminator-relative target), and composite (a weighted combination of the others).
• Contrastive loss's defining property is a batch-relative target — matches and mismatches drawn from the same batch, not a separately collected label, which is why it is self-supervised and scales to hundreds of millions of examples at near-zero labeling cost.
• A low reconstruction loss does not prove a meaningful learned representation — it only proves recreation fidelity against the model's own input.
• Adversarial training is inherently harder to stabilize because the generator's target — the discriminator's current judgment — is itself changing throughout training, unlike every other loss family's fixed target.
• An unweighted composite loss is a specific, failure-prone choice, not a neutral default — differently-scaled components can let one objective dominate training while another is starved of meaningful gradient signal.
• CLIP's contrastive mechanism may be implemented with cross-entropy arithmetic internally, but its defining structural property — a batch-relative target — is what makes it contrastive, not the formula computing its gradient.

This lesson named the five loss families and the specific risk an unweighted composite loss creates. What it did not cover is the fuller toolkit for keeping a multimodal training run stable once that risk is recognized. Next: M1-09 covers training stability in multimodal settings — normalization, learning-rate warmup, loss weighting in depth, and gradient clipping — the direct continuation of the composite-loss imbalance this lesson's worked example demonstrated.