Multimodal Prompt Engineering: Text-Plus-Image Prompts and Context Embeddings

Identity statement: prompt engineering steers a model's output with no training — no weight update, no gradient descent, no backpropagation. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states this directly: "Prompt engineering steers a model's output with no training."

When it matters: any scenario that asks how to change a model's behavior without retraining it, or that describes constructing an input to guide an output rather than adjusting the model itself.

This is worth contrasting explicitly against every mechanism this module has covered so far, because the contrast is the entire point of prompt engineering's existence. Full fine-tuning and parameter-efficient adaptation (M1-10) both change the model's weights, at real cost in data and compute, to produce a lasting change in behavior. Prompt engineering changes nothing about the model itself — it changes only what is handed to the model as input, for exactly one inference call at a time. The model's weights before and after a prompt-engineered call are identical; what changed is the specific instruction or context the already-fixed model was asked to respond to.

This is precisely why prompt engineering is the fastest, cheapest lever available: there is no training loop to run, no gradient to compute, no risk of catastrophic forgetting, and the effect of a change is visible on the very next inference call rather than after a training run measured in hours or days.

For a text-only model, a prompt is a string of text. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names the multimodal-specific extension directly: "For multimodal models, prompts can be text plus an image or audio input." A multimodal prompt is not limited to describing what an image contains in words — it can include the actual image (or audio clip) alongside text instructions, letting the model condition its response on the real visual or auditory content directly rather than on a textual description of that content.

This connects directly back to M1-11's fusion vocabulary: a multimodal prompt is, in effect, a small act of runtime modality combination — the model has to process the text and the image (or audio) together to produce a response, using whatever fusion mechanism its architecture implements internally. The prompt itself does not choose the fusion point; that is a property of the model's architecture, fixed at training time. What the prompt controls is what content enters each modality's input at inference time, not how the model combines those inputs internally.

A practical example makes the distinction concrete: asking a multimodal assistant "what is unusual about this photo?" while attaching an actual image is a genuinely multimodal prompt — the model receives real pixel data to reason over. Asking the same assistant "what would be unusual about a photo showing [a detailed text description of a scene]?" with no image attached at all is a text-only prompt, even though its subject matter is visual — no actual image data ever enters the model's input in the second case, only a textual description of one. The presence of visual subject matter in a prompt's wording does not make the prompt multimodal; only the presence of an actual image or audio input alongside the text does.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names four specific techniques: "clear/specific instructions, zero-/one-/few-shot examples (in-context learning), chain-of-thought for reasoning, and context embeddings to guide image generation."

Clear, specific instructions. The most basic technique: a vague or ambiguous prompt gives a model more room to produce an output that technically responds to the prompt but misses the user's actual intent, while a specific, unambiguous instruction narrows that room. This applies identically whether the prompt is text-only or multimodal — a specific instruction about what to identify or describe in an attached image is more likely to produce a useful response than a vague one.

Zero-, one-, and few-shot examples (in-context learning). Rather than only instructing a model in the abstract, a prompt can include worked examples of the desired input-output pattern directly inside the prompt itself, with no training involved: zero-shot provides no examples at all, relying purely on the instruction; one-shot provides exactly one worked example; few-shot provides several. The model is not updating any weight based on these examples — it is using them as additional context to infer the intended pattern for the current request, entirely within that single inference call. The name "in-context learning" is precise about what is and is not happening: the model appears to "learn" the demonstrated pattern within that one request, but nothing about that apparent learning persists past the current inference call — start a brand-new conversation with the same model and none of the previous prompt's examples carry over unless they are explicitly included again — a direct consequence of section 1's core fact that no weight update ever occurred in the first place.

Chain-of-thought. Prompting a model to work through its reasoning step by step, rather than jumping directly to a final answer, tends to improve performance on tasks that benefit from intermediate reasoning steps — arithmetic, multi-step logical problems, or tasks requiring the model to consider several pieces of information before concluding. This is a prompting technique, not a training technique: nothing about the model changes; what changes is the structure the prompt asks the model's output to follow.

Context embeddings, to guide image generation. This is the multimodal-specific technique with the most direct forward connection in this course. A context embedding is a vector — commonly produced by encoding a text prompt with a model like CLIP's text encoder — that steers what a generative model (a diffusion model, most directly) produces. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names this exact mechanism, and it connects directly to M1-08's contrastive-loss material: the same shared embedding space CLIP's contrastive pretraining produces is what supplies the context embedding a diffusion model conditions on.

Identity statement: a context embedding is a vector encoding a prompt's intended meaning, injected into a generative model's process to steer its output toward that meaning, without retraining the generative model itself for each new prompt. Testing and refining these embeddings — by revising the prompt whose encoding produces them — is how a user achieves a desired generated image, per the source material's own framing of this iterative practice.

L1 — Intuition

A diffusion model's denoising process (covered at full depth in M3-03) has no inherent sense of what to draw — left alone, it produces some plausible-looking image from its training distribution, with no particular content guaranteed. A context embedding acts as a steering signal injected into that process, nudging each step of denoising toward the region of possible images that the embedding's encoded meaning describes. Change the prompt, change the embedding, change what the denoising process is nudged toward — without changing anything about the denoising network's own trained weights.

L2 — Mechanism, at the depth this module expects

Encode a text prompt with a text encoder (commonly CLIP's, given the shared embedding space M1-08 introduced) to produce a vector. Inject that vector into the generative model's process at each relevant step, so the model's output is conditioned on it throughout generation rather than only once at the start. The generative model's own trained weights never change across different prompts — only the context embedding supplied at inference time changes, which is exactly why this is a prompting technique (an inference-time input) rather than a training technique (a weight update).

L3 — Why "testing and refining" is the expected practice, not a failure mode

Because a context embedding's steering is an inference-time input rather than a trained, fixed behavior, iterating on a prompt — trying a different phrasing, adding detail, removing an ambiguous term — and observing how the resulting embedding changes the generated output is the normal, expected way to reach a desired result, not evidence that the underlying model is broken or poorly trained. This directly foreshadows the fuller prompt-engineering-for-generative-systems treatment M6-05 gives this same practice once the U-Net and diffusion material is in scope, and the parallel structure is not accidental: iterating on a text prompt to shape a language model's output and iterating on a text prompt to shape a diffusion model's generated image are the same underlying discipline — testing and refining an inference-time input — applied to two different kinds of trained model.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) closes this domain's material by naming four trends "to be aware of," each of which gets its own full, dedicated module later in this course. This lesson's job is recognition only — name each one, place it correctly, and know where its full treatment lives.

Vision-language models (VLMs) is the umbrella term for models that process both visual and textual information jointly — CLIP, covered throughout this course from M1-08 onward, is the primary named instance, but the term names a broader category than CLIP alone. A VLM's defining property is joint reasoning over both modalities, whatever specific fusion point (per M1-11's taxonomy) or training objective (per M1-08's loss families) it happens to use internally — VLM is a category name describing what a model does, not a specific architecture or loss function in its own right, which is why it sits above, rather than alongside, the more specific mechanisms this module already covered.

Diffusion-based image/video generation learns to reverse a forward noising process, producing a coherent image or video from what started as random noise — this lesson's context-embedding material is the direct, minimal preview of the conditioning half of that mechanism. The word "video" in this trend's name is worth flagging specifically: the same forward-noise, learned-reverse-denoising principle that produces a single generated image extends to generating a sequence of frames, with additional mechanisms (not covered at this module's depth) handling consistency from frame to frame — recognizing that diffusion is not exclusively an image technique, even though this course's own diffusion treatment in M3-03 and M3-04 focuses on the image case, is itself a piece of the recognition-depth expected here.

Contrastive multimodal pretraining, named again here explicitly by its CLIP instance, is the self-supervised mechanism M1-01 and M1-08 already built up in detail — recognizing it here is confirmation that the earlier material connects to this closing trend list, not a new mechanism to learn. Its inclusion in an "emerging trends" list, despite this course having already covered its mechanism in depth two lessons ago, is itself worth noting: the source material treats CLIP's contrastive approach as an active, ongoing direction in multimodal AI rather than a settled, historical technique, which is consistent with M1-08's framing of contrastive loss as the mechanism behind a still-expanding range of cross-modal alignment applications beyond just text-image pairs.

Speech pipelines via NVIDIA Riva name a specific NVIDIA product ecosystem: automatic speech recognition (ASR) converts spoken audio to text, text-to-speech (TTS) converts text back to spoken audio, and a conversational system chains ASR, an NLP/LLM component, and TTS together in a fixed pipeline order — transcribe, understand and respond, speak the result back. This is the one trend among the four that introduces a genuinely new modality-handling detail not yet covered elsewhere in this module: audio as both an input (ASR) and an output (TTS) of the same conversational pipeline, distinct from every other example in this course so far, which has treated audio primarily as an input modality alongside text and image.

A user wants a diffusion model to generate a specific product photo and iterates on the prompt across three attempts.

Attempt 1: "A watch."
  -> Vague. The resulting context embedding encodes only the bare
     concept "watch," with no constraint on style, setting, angle,
     or material -- the generated image could reasonably be almost
     any watch, photographed almost any way.

Attempt 2: "A silver wristwatch on a white background, studio
lighting, product photography style."
  -> More specific. The context embedding now encodes constraints
     on color (silver), object framing (product photography), and
     lighting (studio) -- narrowing the region of image-space the
     denoising process is steered toward considerably.

Attempt 3: "A silver wristwatch with a black leather strap, on a
white background, studio lighting, shot from a 3/4 angle, product
photography style, no shadows."
  -> Most specific. Additional constraints (strap material and
     color, camera angle, absence of shadows) narrow the target
     region further still.

This is a constructed scenario with illustrative prompts, not measurements from a real image-generation system. Nothing about the diffusion model's own trained weights changed across the three attempts — the same frozen model processed three different context embeddings, each encoding progressively more constraint. This is precisely the "testing and refining" practice section 4's L3 subsection named as the expected, normal path to a desired result: each attempt is a new inference-time input to an unchanged model, not a new round of training.

The generalizable reading: a vague prompt does not fail because the diffusion model is weak — it fails because the context embedding it produces under-constrains the target region of image-space, leaving the denoising process free to land anywhere within a wide range of plausible images matching that vague description. Prompt refinement is the practice of tightening that region by adding the specific detail the desired output actually requires, and it is available entirely without retraining because the constraint lives in the embedding supplied at inference time, not in the model's weights.

A user submits four separate requests to a multimodal assistant. Identify which named technique each one demonstrates.

Request 1: "Summarize this document in exactly three bullet points,
each under 15 words."
  -> CLEAR, SPECIFIC INSTRUCTIONS. A precise, unambiguous constraint
     on the desired output format, with no examples or reasoning
     structure requested.

Request 2: "Here are two examples of a product description
rewritten in a formal tone: [example 1] [example 2]. Now rewrite
this new product description in the same tone: [new description]."
  -> FEW-SHOT (IN-CONTEXT LEARNING). Two worked examples are
     supplied directly in the prompt to demonstrate the desired
     pattern, with no training involved.

Request 3: "Before giving your final answer, walk through your
reasoning about which factors matter most for this decision, then
state your conclusion."
  -> CHAIN-OF-THOUGHT. The prompt asks for an explicit intermediate
     reasoning structure before the final answer, rather than
     asking for the answer directly.

Request 4: "Generate an image of 'a lighthouse at sunset, in the
style of a watercolor painting.'"
  -> CONTEXT EMBEDDING (implicitly). The text prompt will be encoded
     into a context embedding that steers a diffusion model's
     denoising process toward the described scene and style.

This is a constructed scenario with illustrative requests, not measurements from a real deployed system. Notice that all four requests share the defining property from section 1: none of them change the underlying model's weights. Each one supplies a different kind of inference-time input — an instruction, worked examples, a requested reasoning structure, or a prompt destined to become a context embedding — and the model's fixed, already-trained weights process each accordingly.

⭐ THE EARNED INSIGHT Every technique in this lesson, multimodal or not, is a variation on the same single fact: a prompt is the only lever available at inference time, and every named technique is a different way of packing more useful signal into that one lever, without ever touching the model's weights. Clear instructions pack signal into the request itself. Few-shot examples pack signal into demonstrated patterns. Chain-of-thought packs signal into the requested output structure. A context embedding packs signal into a vector a generative model conditions on. Recognizing all four as instances of shaping the input, never the model, is what keeps this list from feeling like four unrelated tricks to memorize separately.

The table below names the specific way each of these techniques gets misread when that single unifying fact is lost sight of.

Every row maps a specific symptom to a specific fix. Exam weight explains why this vocabulary is worth holding precisely.

Core Machine Learning and AI Knowledge carries 20% exam weight, and [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) closes this domain's material with exactly this content, framing it as the bridge from foundational mechanism to the generative and multimodal-application material the rest of this course covers. The four named emerging trends are each explicitly previewed here specifically so that later modules' material — diffusion, CLIP, Riva — arrives as a recognized continuation rather than as unfamiliar new territory.

The question tends to arrive in a small number of recognizable shapes.

1. Prompt engineering versus training, directly. "Does prompt engineering update a model's weights?" with the keyed answer naming no training involved at all.
2. Technique identification from a described prompt. A scenario describes a specific prompt structure and asks which named technique it demonstrates — clear instructions, few-shot, chain-of-thought, or a context embedding.
3. Context-embedding mechanism. "What steers a diffusion model's output toward a specific described scene?" with the keyed answer naming context embeddings, against distractors offering the noise schedule or the number of denoising steps — real diffusion parameters, but not the one controlling what rather than how well.
4. Trend-to-module mapping. A scenario names one of the four emerging trends and asks which later module or mechanism it connects to.

What the distractors typically look like

The reliable distractor families: describing prompt engineering as a form of fine-tuning or weight adjustment; confusing chain-of-thought (a prompting structure) with a training objective; and offering a diffusion model's mechanical parameters (noise schedule, step count) as the answer to a question specifically about what content is generated, which context embeddings control rather than those parameters.

A fifth distractor family draws on the emerging-trends list specifically: an item names one of the four trends and offers a description belonging to a different one of the four as a plausible-sounding wrong answer — describing CLIP's contrastive mechanism as "diffusion-based," for instance, or describing a VLM as inherently requiring the Riva speech pipeline. Each of the four trends is a genuinely separate concept, and the defense against this family is exactly the recognition-depth this lesson asks for: know which of the four a given description actually belongs to, not just that all four exist.

VLM is a broad category naming any model that jointly processes and reasons over both images and text; CLIP is one specific, named technique within that broader category, defined by its particular contrastive pretraining objective and its resulting shared embedding space. Not every VLM uses CLIP's specific contrastive-pretraining approach, and not every model that uses a contrastive image-text objective is necessarily described using the VLM label in every context — the two terms operate at different levels of specificity, with VLM naming the general capability (joint image-text reasoning) and CLIP naming one particular, well-known mechanism for achieving a piece of that capability (a shared embedding space via contrastive pretraining).

No. Prompt engineering steers a model's output entirely through what is supplied as input at inference time — instructions, examples, a requested reasoning structure, or a context embedding — with no gradient computed and no weight updated anywhere in the model. This is the defining property that distinguishes it from every other technique in this module: transfer learning, full fine-tuning, and parameter-efficient adaptation all change the model itself, while prompt engineering changes only what a fixed, already-trained model is asked to respond to for one specific inference call.

A context embedding is a vector — commonly produced by encoding a text prompt with a text encoder such as CLIP's — that is injected into a diffusion model's denoising process at each relevant step, steering the process toward the region of possible images the embedding's encoded meaning describes. The diffusion model's own trained weights do not change between different prompts; what changes is the context embedding supplied at inference time, which is why revising and testing different prompt phrasings — and observing how the resulting embedding changes the generated output — is the expected, normal way to reach a desired image, rather than a sign that the underlying generative model needs retraining.

• Prompt engineering steers output with zero training — no weight update, no gradient, no risk of catastrophic forgetting, effective on the very next inference call.
• A multimodal prompt can include an actual image or audio input alongside text, not merely a textual description of that content.
• Four named techniques: clear/specific instructions, zero-/one-/few-shot in-context learning, chain-of-thought, and context embeddings for guiding image generation.
• A context embedding is an inference-time input a frozen generative model conditions on — testing and refining prompts, and thereby the resulting embedding, is the normal path to a desired generated image.
• Four emerging trends close this module: VLMs, diffusion-based generation, contrastive multimodal pretraining (CLIP), and Riva speech pipelines — each previewed here, each with its own full treatment in a later module.
• Every prompting technique packs more useful signal into the same single lever — the input supplied at inference time — never into the model's weights.
• VLM is a broad category naming any model that jointly reasons over image and text; CLIP is one specific, named technique within that category — the two terms operate at different levels of specificity and are never interchangeable.
• Diffusion-based generation extends past still images to video, using the same forward-noise, learned-reverse-denoising principle with additional frame-consistency mechanisms layered on top.
• A vague prompt under-constrains a context embedding rather than exposing a weak model — refining the prompt, never retraining, is the expected, normal response to an unsatisfying generated result.

This closes Module 1's foundational pass through the ML mechanics every multimodal model builds on: paradigms, fitting, metrics, frameworks, the training loop, convolutions, residual connections, multimodal losses, training stability, transfer learning, fusion, and now prompting — steering an already-trained model's output being the one mechanism in this entire module that touches no weight at all. Every mechanism named here as a preview — diffusion, CLIP, Riva — gets its full, dedicated treatment in a later module. Next: M2-01 opens Module 2, Data Analysis, with data cleaning — handling missing values, outliers, scaling, and categorical encoding — the first step any of this module's training mechanics actually depend on having done correctly.