Application Patterns: RAG, Chatbots, and Summarizers Over Multimodal Data

An application pattern, in this context, is a repeatable pipeline shape that composes a generative model with additional infrastructure to solve a task the model alone cannot solve reliably by itself. A raw LLM, asked a question about a document it was never trained on, will either say it does not know or — worse — confidently generate a plausible-sounding but fabricated answer. A raw LLM, asked a follow-up question in a fresh API call with no memory of the prior turn, has genuinely no idea a prior turn happened. A raw LLM, asked to summarize a 40-page report, can do so, but with no guarantee the summary tracks the report's actual structure rather than the model's own priors about what a report "usually" contains. RAG, chatbots, and summarizers are the three named answers to these three specific gaps, and Domain 4's own framing groups them together because all three recur constantly in multimodal deployments specifically, not because they are the only application patterns that exist anywhere in generative AI.

L1 — Intuition: look it up before you answer

RAG — retrieval-augmented generation — is the pattern of retrieving relevant external information and injecting it into the model's context before generation happens, rather than relying entirely on whatever the model memorized during training. The intuition is exactly the same as a person answering a question by first looking something up rather than answering purely from memory: the lookup step grounds the eventual answer in a specific, checkable source, and reduces the chance of confidently stating something that is simply wrong.

L2 — Mechanism: chunk, embed, store, retrieve, generate

[GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names the pipeline in this exact order: "ingest → chunk → embed → store in a vector database → retrieve top-k by similarity → inject into the prompt → generate." Ingestion brings raw source documents into the pipeline. Chunking breaks each document into smaller pieces — a paragraph, a section, a fixed token window — because embedding an entire long document as a single vector loses the granularity needed to retrieve just the relevant part later, and because most embedding models have a practical input-length limit chunking respects. Embedding converts each chunk into a dense vector using the same kind of representation-and-shared-space machinery M4-01 and M4-03 covered — a text chunk becomes a text embedding, and in a multimodal RAG system, an image becomes an image embedding in the same or a comparable space. Storage places every chunk's embedding into a vector database, indexed for fast similarity search rather than a linear scan. Retrieval, at query time, embeds the user's query the same way, then searches the vector database for the top-k stored chunks whose embeddings are closest by similarity — the same cosine-similarity comparison CLIP's zero-shot classification performs, applied here to rank candidate chunks rather than candidate class labels. Injection places the retrieved chunks' text directly into the prompt sent to the generative model, alongside the user's original query. Generation is the model producing its answer, now grounded in the retrieved context rather than relying solely on its training-time knowledge.

L3 — The exam-relevant edge case: RAG reduces hallucination, it does not eliminate it

[GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) states RAG's benefit precisely: it "grounds answers in external, up-to-date knowledge and reduces hallucination." "Reduces" is a deliberately weaker claim than "eliminates," and the exam's foundational framing expects you to hold that distinction. A RAG system can still hallucinate in at least two ways worth naming specifically: the retrieval step can fail to surface the actually-relevant chunk (a retrieval failure, where the generation step then works correctly from irrelevant context and produces a plausible but ungrounded answer), or the generation step can ignore or misread correctly-retrieved context and produce an answer that contradicts what was actually retrieved (a generation failure, where retrieval worked but the model did not faithfully use what it retrieved). Judging a RAG system only on whether its final answer sounds right, without separately checking whether retrieval surfaced the right material, is exactly the shortcut the source material's evaluation-methodology material elsewhere warns against — the same discipline that section 6 below applies specifically to the multimodal case.

⭐ THE EARNED INSIGHT: > RAG's grounding only works if every stage in the chunk-embed-store-retrieve-generate chain does its job correctly — a failure at any single stage (bad chunking that splits a fact across two pieces, an embedding that does not capture the query's real intent, a retrieval step that returns the wrong top-k, or a generation step that ignores good context) produces the same visible symptom, a wrong or ungrounded-sounding answer, which is precisely why diagnosing a RAG failure means checking each stage separately rather than treating "the answer was wrong" as evidence about which stage actually broke.

[GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) states the mechanism directly: "the LLM is stateless; the app resends conversation history so context carries across turns." An LLM API call is, mechanically, a single request that receives some input text and returns some output text, with nothing preserved on the model-serving side between one call and the next — the model has no session, no memory buffer, no persistent state tied to a particular user's conversation. What makes a multi-turn conversation feel continuous is that the application layer sitting in front of the model keeps its own record of every prior turn and prepends that entire record to the prompt on every new call. Ask a follow-up question, and the app sends the model the full transcript so far plus the new question, every single time — the model is, in effect, re-reading the entire conversation from scratch on every turn and has no way to distinguish "a turn I remember" from "a turn the app just told me about a moment ago," because to the model those are the exact same thing: text that arrived in this call's input.

This has a direct, practical consequence worth naming: as a conversation grows longer, the amount of text the app has to resend on every turn grows with it, eventually running into the model's context-window limit — a constraint entirely explained by the resend mechanism, not by any notion of the model's memory "filling up," since there was never a persistent memory to fill in the first place. Production chatbot systems handle this by summarizing or truncating older turns before resending, a direct application of the summarization patterns covered in section 4, layered on top of the chatbot pattern rather than a separate concern.

RAG and the chatbot pattern combine naturally and frequently: a conversational assistant that also retrieves relevant documents per turn is simply the chatbot's resend-history mechanism and RAG's retrieve-then-generate mechanism running side by side, with retrieved context injected alongside the resent conversation history on each call — two independent patterns composed, not a fourth new pattern requiring its own separate mechanism.

[GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names the two approaches directly: "abstractive summarization (LLM-generated) vs. extractive (select existing sentences)." Extractive summarization selects a subset of the original document's own sentences (or, for a multimodal document, a subset of the original images or clips) and presents that subset as the summary, guaranteeing every word of the output was actually present in the source. Abstractive summarization generates new phrasing that was never literally present in the source, aiming to compress meaning more fluidly than picking whole sentences allows, at the cost of a real, structural risk: because the words are generated rather than selected, an abstractive summarizer can produce a summary that states something the source did not actually say, a specific flavor of hallucination distinct from RAG's retrieval-versus-generation failure but governed by the same underlying concern — a generated output is only as trustworthy as the process that constrains it to stay faithful to its source.

Extractive summarization's guarantee (every output sentence is verifiably from the source) trades away fluency and compression quality — stitching together selected sentences verbatim often reads less smoothly than a purpose-generated summary, and a summary limited to whole existing sentences cannot compress as aggressively as one free to rephrase. This is a real, named tradeoff rather than one option being strictly better: a legal or medical summarization tool, where every claim needs to be traceable to an exact source sentence, has a strong reason to prefer extractive summarization's traceability even at a fluency cost; a consumer-facing news-digest feature, where reading experience matters more than sentence-level traceability, has a correspondingly strong reason to prefer abstractive summarization's fluency even at a faithfulness-risk cost.

A field-service tool needs to answer a technician's question ("why won't the compressor start after a power cycle") by retrieving from a repair manual that mixes text procedures with diagrams and photos of the relevant components. Walk the pipeline exactly as section 2 lays it out, now with a second modality in play at every stage.

Stage 1 — Ingest: the repair manual PDF, containing text sections and embedded diagrams.

Stage 2 — Chunk: split text into paragraph-level chunks; treat each diagram
  and its associated caption as its own separate chunk (an image-plus-caption
  pair, not merged into surrounding text chunks, so a diagram can be
  retrieved on its own visual relevance).

Stage 3 — Embed: text chunks -> text encoder -> text embeddings
                  diagram chunks -> image encoder (the same shared-space
                  encoder CLIP trains, per M4-03) -> image embeddings
  Both land in the SAME shared embedding space, so a text query can be
  compared directly against both text-chunk and diagram-chunk embeddings.

Stage 4 — Store: all embeddings (text and image) in one vector database,
  tagged with their modality and source page number.

Stage 5 — Retrieve: technician's query "why won't the compressor start
  after a power cycle" -> embedded with the text encoder -> top-k=5
  nearest chunks by similarity, which in this query's case returns
  3 text chunks (a troubleshooting procedure, a power-cycle sequence
  note, a compressor-relay description) AND 2 diagram chunks (a wiring
  diagram, a relay-location photo) -- because both modalities' embeddings
  live in the same comparable space, retrieval ranks across both at once.

Stage 6 — Inject: the 3 text chunks are inserted into the prompt as text;
  the 2 diagrams are passed to a vision-capable model alongside the text,
  or described via their captions if the generation model is text-only.

Stage 7 — Generate: the model produces an answer citing the specific
  relay-check procedure the retrieved text described, with the retrieved
  wiring diagram available for the technician to view alongside the answer.

Every stage is the identical five-stage-plus-injection-and-generation pipeline from section 2; the only change is that stages 2 and 3 now branch by modality before converging back into one shared vector space at stage 4, and stage 5's retrieval ranks candidates from both modalities in one unified similarity search rather than running two separate searches. This is a constructed scenario — the specific retrieved-chunk contents are illustrative — but the mechanism it demonstrates is the direct multimodal extension [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names: "Multimodal RAG can retrieve across text and images."

Multimodal RAG's stage-4/stage-5 shared vector space is not a new mechanism this lesson invents — it is a direct application of the shared-embedding-space idea M4-03 covers for CLIP specifically, applied here to whole document chunks rather than to a single image-caption pair. That dependency is worth stating explicitly because it means multimodal RAG inherits CLIP's own named limitation: M4-03's material notes that contrastive training solves a coarse, whole-item alignment (whole image to whole caption) without solving fine-grained correspondence (which pixel matches which word). A multimodal RAG system built on the same kind of shared space inherits that same coarseness — it can retrieve a diagram whose overall content is relevant to a query, but it has no built-in mechanism for confirming that a specific detail within that diagram is the one the query actually needed, which is a real, source-traceable limitation rather than a flaw specific to this lesson's worked example.

A second dependency worth naming: if a chunk in the ingested manual is missing its expected companion modality — a diagram with no caption, or a procedure step that references "see Figure 4" with no Figure 4 actually embedded in the ingested document — the missing-modality strategies from M4-04 apply directly to the ingestion stage itself, before retrieval ever runs. Treating "the document has a gap" as a RAG-specific problem, rather than recognizing it as the same missing-modality problem this module already gave four named strategies for, would mean re-solving a problem this course already solved two lessons earlier.

A customer-support chatbot has been in a single conversation with one user for 45 turns, and the team notices the model's responses have started ignoring instructions the user gave in the first few turns — a stated preference for concise answers, say — even though the app's logs confirm every turn was resent as documented in section 3.

The diagnosis is not "the model forgot," because a stateless model never remembered in the model-level sense to begin with — it only ever had access to whatever was in each call's input. The actual mechanism is almost certainly context-window pressure interacting with how the app assembles the resent history: if the app is silently truncating or summarizing the oldest turns once the running history approaches the model's context limit — a common, necessary mitigation given section 3's own point that resent history grows without bound as a conversation lengthens — then turn 2's "please keep answers concise" instruction may have already fallen outside whatever window of history the app is still including by turn 45. The fix is not to search for a memory bug in the model; it is to check the app's own history-management logic — whether truncation is dropping the earliest turns indiscriminately (losing early instructions along with early small talk) or whether a smarter approach (a running summary that explicitly preserves stated user preferences, rather than blind truncation) is needed instead. This is a constructed scenario, illustrative rather than a measured production incident, but the diagnostic habit it demonstrates — checking what the app actually resent, not what the model "should" remember — generalizes to any chatbot behavior that looks like forgetting.

Application Patterns is named as its own Domain 4 subsection [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md), positioned directly after autoencoders/anomaly detection and directly before the Python tooling material — a placement reflecting that RAG in particular depends on the vector-database tooling the closing lesson names. The domain's foundational scope note applies here specifically as "recognize the pattern and its pipeline order," not "implement a production RAG system from scratch."

The question tends to arrive in three recognizable shapes. A sequencing question asks for the correct order of RAG's pipeline stages, keyed to chunk-embed-store-retrieve-generate, with a distractor scrambling that order (retrieve before embed, or generate before retrieve) to test whether the sequence is genuinely memorized rather than vaguely gestured at. A mechanism question asks why a chatbot appears to remember earlier turns, keyed to the stateless-model-plus-resent-history mechanism, with a distractor claiming the model itself stores conversation state. A distinction question asks which summarization approach guarantees output fidelity to the source, keyed to extractive, against abstractive offered as the (plausible but wrong, for this specific property) alternative.

What the distractors typically look like

The most reliable distractor for RAG scrambles the pipeline order, banking on the fact that "chunk, embed, retrieve, generate" sounds plausible in almost any order to someone who has not actually internalized the sequence and the dependency it encodes — you cannot retrieve from a vector database that has not yet stored anything, and you cannot store an embedding that has not yet been computed. For chatbots, the reliable distractor attributes memory to the model itself ("the LLM retains context between calls") rather than to the resend mechanism, testing the stateless-versus-stateful distinction directly. For summarization, the reliable distractor swaps which approach is "guaranteed faithful" — describing abstractive summarization as inherently more accurate because it is "smarter," when the actual faithfulness guarantee belongs to extractive summarization specifically because of what it structurally cannot do (invent new phrasing).

Because the retrieval step's core operation — finding the top-k stored items whose embeddings are most similar to a query embedding — is a similarity search over dense vectors, an operation a regular relational database is not built to perform efficiently at scale. A conventional database excels at exact matches and range queries over structured fields (find every row where a date falls in this range), but "find the k vectors closest to this vector, out of millions, by cosine similarity" requires index structures purpose-built for approximate nearest-neighbor search. A vector database provides exactly those index structures, which is why RAG's storage stage is specifically a vector database rather than any other kind of data store — the closing lesson of this module names the specific tools (FAISS, Milvus, Pinecone, Chroma, pgvector) that implement this.

Check whether every element of the output — every sentence, every selected image — can be matched, verbatim or as an exact selection, back to something present in the source, and check that mechanism separately for each modality, since a multimodal summarizer often does not apply the same approach uniformly across modalities. A summary that selects three sentences verbatim from a report's text but generates a new caption describing a selected chart, rather than reusing the chart's original caption unchanged, is extractive for its text component and abstractive for its image-adjacent component — a legitimate, common hybrid rather than a contradiction, and one a scenario question can specifically test by asking you to classify each component of a described summary separately rather than assigning one label to the whole system. The practical implication carries the same weight section 4 already established: the generated (abstractive) component carries the faithfulness risk the selected (extractive) component does not, even within one otherwise-mixed summarizer.

Yes — the two patterns are independent and frequently combined rather than mutually exclusive. A chatbot is defined by its resend-history mechanism for sustaining multi-turn context; RAG is defined by its retrieve-then-generate mechanism for grounding a single response. A conversational assistant that retrieves relevant documents on every turn, while also resending the growing conversation history on every turn, is running both mechanisms side by side within the same application, and describing it as "a RAG chatbot" is accurate without implying either mechanism has replaced or absorbed the other — each still does exactly what section 2 and section 3 describe independently.

• RAG's pipeline is fixed and sequential: chunk, embed, store, retrieve, inject, generate — each stage depends on the previous stage's output, and a failure at any single stage can produce the identical symptom (a wrong-sounding answer).
• A chatbot's apparent memory is an application-layer illusion; the underlying LLM is stateless, and the app resends conversation history on every call, which is also why long conversations eventually hit the context-window limit.
• Extractive summarization guarantees source-traceability at a fluency cost; abstractive summarization gains fluency and compression at a faithfulness-risk cost — neither is universally superior.
• Multimodal RAG extends the identical pipeline across text and images by embedding both into the same shared space CLIP's contrastive training builds, ranking retrieval candidates across modalities jointly.
• RAG reduces hallucination; it does not eliminate it, and retrieval failures and generation failures are separate, independently-diagnosable problems.
• A missing modality inside a RAG-ingested document is the same problem M4-04 already named, not a new RAG-specific gap requiring its own fix.

Next: M4-07 closes the module with the concrete Python tooling — NumPy, spaCy, Keras, and the named vector databases (FAISS, Milvus, Pinecone, Chroma, pgvector) — that actually implement the RAG pipeline's embed-and-store stages this lesson described in the abstract.