Prompt Chains, Dynamic Branching, and Refining Agent Decision-Making

[GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): "A prompt chain sequences prompts so the output of one step feeds the next, letting an agent break a task into reliable, inspectable stages." A prompt chain is a sequence of prompts in which the output of one step becomes part of the input to the next. Instead of asking a model to do an entire multi-part task in a single call — extract the entities, summarize the document, then answer a question about it, all in one prompt — a chain splits the task into stages, each with its own narrower prompt, and passes the result of stage N forward into stage N+1's context. The immediate benefit is reliability: a single sprawling prompt asking a model to do five things at once gives the model five chances to drop or blend one of them together, while five separate, focused prompts each give the model one job to do well, and each stage's output can be checked before it becomes another stage's input.

The second benefit is inspectability, and it matters more in production than the reliability gain does. When a single giant prompt produces a wrong final answer, there is no way to tell which part of the reasoning went wrong — the failure is opaque, buried inside one model call. When a chain produces a wrong final answer, each stage's output is a concrete, loggable artifact: you can look at what stage 2 actually returned, see that stage 2 misclassified the input, and know exactly where the chain diverged from correct. This is the same reason a pipeline of small, testable functions is preferred to one large function in ordinary software engineering, applied to prompting — decomposition buys both correctness and diagnosability, at the cost of more calls and more latency per task.

The fixed chain: a script, not a decision-maker

The simplest prompt chain is a fixed sequence: step 1 always runs, its output always feeds step 2, step 2's output always feeds step 3, and so on, regardless of what any step actually returned. A fixed chain is appropriate when a task genuinely always follows the same shape — summarize, then translate, then format, for instance, where every input needs all three stages in that order and no input ever needs something different. The problem is that most agent tasks are not that uniform. A ticket that turns out to be a billing question needs a different next step than one that turns out to be a bug report, and a fixed chain that routes both through the same three stages either wastes a stage on the wrong path or, worse, forces an answer out of a stage that was never designed to handle that kind of input.

A dynamic prompt chain looks at an intermediate step's output and uses it to choose which step runs next, rather than following one predetermined sequence. "Dynamic" here has a precise meaning worth pinning down, because the exam's most common way to test this objective is to describe a multi-step process and ask whether it counts: dynamic means the branching decision itself happens at runtime, based on the actual content of an intermediate result, not that the chain merely has multiple steps or multiple possible outputs. A chain with five fixed stages that always run in the same order is not dynamic just because it is long; a chain with two stages where the second stage's prompt is chosen based on what the first stage classified the input as is dynamic, even though it is short.

L1 — Intuition

Think of a fixed chain as a single hallway with doors in a row — you walk through door one, then door two, then door three, and the layout never changes no matter what happens inside any room. A dynamic chain is a hallway with a fork: after room one, a sign based on what happened in room one tells you which of two or three next doors to take, and different visitors end up taking different paths through the building depending on what they were carrying when they walked in. The building is designed once; the path through it is decided per visit.

L2 — Mechanism

Mechanically, a dynamic chain's first stage (or an early stage) produces some structured signal — a classification label, a confidence score, an extracted field, a boolean flag — and the orchestration logic around the chain reads that signal and dispatches to one of several possible next prompts. A support-ticket agent's first stage might output a category (billing, bug, feature-request, unclear), and the orchestration code branches: billing routes to a prompt that has account-lookup context baked in, bug routes to a prompt with a reproduction-steps template, unclear routes to a clarifying-question prompt instead of guessing. The branching logic itself does not have to live inside a prompt — it is ordinary control flow (an if/switch, or a small router function) sitting between chain stages, reading the previous stage's structured output and selecting the next stage's prompt template and inputs accordingly.

L3 — The exam-relevant edge case: a chain that branches is not automatically a decision-refinement system

A chain that branches correctly today is not the same claim as a chain whose branching decisions are actually good, and conflating the two is the specific trap this domain's framing is warning about when it treats prompt chaining (objective 2.1's first half) and decision refinement (its second half, and objective 2.6) as a paired but distinct pair of skills. A chain can branch reliably — the routing logic is bug-free, every category maps to a real next stage, nothing crashes — while still branching on a bad signal, for instance a classification stage that is only 70% accurate at telling bug apart from feature-request. Building the dynamic branching mechanism is objective 2.1's job; measuring whether the branches it takes are the right ones, and iterating on the prompts, tools, or control flow that produce that classification, is a separate, later job — the one the next section covers. A chain that has never been evaluated for decision quality, no matter how cleanly it branches, has only done half of what this pairing of objectives asks for.

[GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): objective 2.6 is named directly as "evaluate and refine agent decision-making," described as closing the loop by measuring "how the agent chooses actions and iterate on the prompts, tools, and control flow that drive those choices." "Evaluate and refine agent decision-making" is its own objective precisely because building a chain that branches and building a chain that branches well are different amounts of work, and the second one is where most of the actual engineering effort in a production agent goes after the first working version ships. Refinement starts with measurement: you need visibility into what decision the agent actually made at each branch point, not just what the final output was. A ticket-routing agent that gets the wrong final answer might have failed at classification (chose bug when it was billing), at retrieval (chose the right category but pulled the wrong account record), or at generation (had the right category and the right record but wrote an unhelpful reply) — and without logging the decision at each branch, all three failure modes look identical from the outside: one bad final answer.

Once decisions are visible, refinement is an iterative loop: identify where the chain branched wrong, form a hypothesis about why (an ambiguous prompt, a missing example, an under-specified category boundary), change one thing — the classification prompt's wording, the set of categories offered, a tool the branch has access to — and re-run against the same set of cases to see whether the change actually improved the branch's accuracy rather than just shifting which cases fail. This loop deliberately mirrors the "change one thing and re-evaluate on a fixed set" discipline this cert's Module 3 (Evaluation and Tuning) treats as its own domain — decision refinement is where an agent-development chain first needs that evaluation discipline, well before a full evaluation pipeline exists to formalize it.

Consider an agent whose job is to take an uploaded document and produce a structured summary appropriate to what kind of document it is — a contract, an invoice, or a resume — each of which needs a genuinely different extraction template.

Stage 1 (classification): "Read this document's first 200 words and classify it as one
of: contract, invoice, resume, unclear. Return only the label."
  Input: [first 200 words of an uploaded PDF]
  Output: "contract"

Branching logic (ordinary code, not a prompt):
  if label == "contract": next_prompt = CONTRACT_EXTRACTION_TEMPLATE
  elif label == "invoice": next_prompt = INVOICE_EXTRACTION_TEMPLATE
  elif label == "resume":  next_prompt = RESUME_EXTRACTION_TEMPLATE
  else:                    next_prompt = CLARIFYING_QUESTION_TEMPLATE

Stage 2 (extraction, contract branch): "Extract: parties, effective date, term length,
termination clause, and governing law from this contract text."
  Input: [full document text] + [stage 1's label, for logging]
  Output: {parties: [...], effective_date: "...", term: "...", termination: "...",
           governing_law: "..."}

Constructed scenario — the document content and extracted fields are illustrative, not drawn from a real case. The chain has two stages, but it is dynamic because stage 2's prompt template is selected at runtime from four possibilities based on stage 1's output. Note what a fixed version of this same chain would have to do instead: either run all four extraction templates against every document regardless of type (wasteful, and the invoice template asking a resume for a "termination clause" will confidently hallucinate one), or pick one extraction template in advance and apply it to every uploaded document type (wrong for three out of four document types). The branch is what lets one chain correctly serve all three document types without either problem.

Now trace the refinement loop against this same chain. Suppose an audit of 50 processed documents finds that 6 resumes were misclassified as "unclear," triggering the clarifying-question branch instead of the resume-extraction branch — a decision-quality failure, not a mechanism failure, since the chain branched exactly as designed on the (wrong) label it received. The fix is not to touch the branching logic, which worked correctly; it is to refine stage 1's classification prompt — perhaps by adding a short list of resume-identifying signals (a "Skills" or "Experience" header, a chronological work history) that the original prompt did not point to — and then re-running the same 50-document audit set to confirm the misclassification rate actually dropped rather than shifting to a different document type.

Dynamic branching is not automatically the better design, and a scenario that always needs the same treatment is exactly where a fixed chain is the correct choice, not a missed opportunity to add a branch. Consider an agent whose job is to take a raw customer review and prepare it for a weekly analytics report: every review, without exception, needs to be (1) translated to English if it is not already, (2) scored for sentiment on a fixed five-point scale, and (3) tagged with one or more product-feature categories from a fixed list.

Stage 1 (translation): "If this text is not already in English, translate it to
English. If it is already in English, return it unchanged."
  Input: "Le service client était excellent, mais la livraison a pris trop de temps."
  Output: "Customer service was excellent, but delivery took too long."

Stage 2 (sentiment scoring): "Score this review's sentiment from 1 (very negative)
to 5 (very positive)."
  Input: [stage 1's output]
  Output: 3

Stage 3 (feature tagging): "Tag this review with any of: customer-service, delivery,
pricing, product-quality, packaging. Return all that apply."
  Input: [stage 1's output]
  Output: ["customer-service", "delivery"]

Constructed scenario — the review text and scores are illustrative. Every single review that enters this pipeline needs all three stages, in this exact order, with no exceptions: there is no version of "prepare a review for the analytics report" that skips translation, skips scoring, or reorders scoring before translation. Adding a branch here — for instance, routing reviews differently based on their detected language — would add complexity without adding any actual capability, because stage 2 and stage 3 need the English text regardless of what language the review started in. This is the case the earlier comparison table's "best fit" row describes directly: a task where every input genuinely needs the same stages in the same order is exactly the case a fixed chain handles with no loss, and forcing a branch into a design like this does not make the chain more capable, only more code to maintain for no behavioral gain.

Contrast this directly with the document-triage example: there, three different document types needed three genuinely different extraction templates, so a fixed chain would have had to either run every template against every document or silently apply the wrong one. Here, there is no such divergence — translation, then scoring, then tagging apply uniformly, so the branch the document-triage agent needed has no equivalent job to do in this pipeline. Recognizing when a task's stages genuinely do not vary by input is as much a part of correct chain design as recognizing when they do; treating every chain as a candidate for dynamic branching, on principle, adds unnecessary decision points to a design that a fixed script already serves correctly.

⭐ THE EARNED INSIGHT "Dynamic" is a property of the decision, not a property of the chain's shape. A ten-stage chain that always runs the same ten stages in the same order is not dynamic no matter how long or sophisticated it looks, and a two-stage chain is dynamic the instant one real runtime choice exists between them. This is exactly why the exam's most durable trap here is not "can you define a dynamic chain" — it is "can you tell, from a description of a process, whether a genuine branch point exists at all," and that same question, one level up, is also what separates a mechanism bug from a decision-quality failure: a router that dispatches flawlessly on a bad signal is not broken, it is faithfully executing the wrong instruction, and no amount of rewriting the router fixes a problem that lives one layer upstream, in the signal itself.

The row worth internalizing for scenario questions is the second-to-last one: a chain being long or having many stages is not what makes it dynamic, and a chain being short is not what makes it fixed. The only question that decides the label is whether a runtime decision — one that could have gone a different way given different input — determines which stage runs next.

This lesson's chain-and-branch pattern is the substrate that the rest of Module 2 assumes exists. The tool calls M2-03 covers happen inside a chain's stages — a branch might dispatch to a stage that calls a database lookup tool rather than one that does not. The Retry pattern M2-04 and the Retry-vs-Circuit-Breaker pairing already authored protect an individual stage's tool call against transient failure; they say nothing about which stage runs next, which is exactly the decision this lesson's branching logic makes. And the decision-quality loop this lesson describes for a single branch point is a small-scale preview of the evaluation discipline this cert's Module 3 formalizes for an entire agent — the same "change one thing, re-check against a fixed set" habit, applied first here at the scale of one classification prompt.

Agent Development carries 15% of the NCP-AAI blueprint, tied with Agent Architecture for the heaviest weight of any domain in the exam, and prompt chaining together with decision refinement is objective 2.1's and objective 2.6's paired territory within it. [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): the domain's own framing states this as engineering "prompts and dynamic prompt chains for reliable performance," with objective 2.6 specifically named as evaluating and refining "how the agent chooses actions" and iterating on "the prompts, tools, and control flow that drive those choices." Expect scenario items that describe a multi-step process and ask whether it qualifies as a dynamic chain, with distractors built around the trap covered above — a long fixed sequence dressed up to sound adaptive, or a genuinely dynamic two-step chain that a test-taker dismisses as "too simple" to count.

Expect a second question shape built around refinement specifically: a scenario describes an agent's branching decision going wrong in production, and the options split between "fix the branching mechanism" and "fix the signal the branching mechanism reads" — the correct answer is almost always the latter, since a working router acting on a bad classification is a decision-quality problem, not a control-flow bug, and conflating the two is the standing trap this section flagged directly in the worked example's resume-misclassification case.

A fixed prompt chain always executes its stages in the same predetermined order regardless of what any stage returns — it is a script. A dynamic prompt chain uses an intermediate stage's actual output (a classification label, a confidence score, an extracted field) to decide, at runtime, which stage runs next, so different inputs can take different paths through the same chain design. The distinguishing test is not how many stages exist but whether a genuine runtime decision determines the next step; a long chain with a fixed order is still not dynamic, and a short two-stage chain with one real branch point already qualifies.

Because a chain can branch correctly — the routing mechanism works, every category dispatches to a real next stage, nothing crashes — while still branching on a low-quality signal, such as an inaccurate classification step, which produces wrong final outputs even though the branching logic itself has no bug. Building the branching mechanism is an engineering task; evaluating whether the branches taken are actually the right ones, and then iterating on the prompts, tools, or control flow that produce that decision, is a measurement-and-iteration task that only starts once the mechanism exists and needs its own fixed evaluation set to be done rigorously rather than by feel.

Log each stage's output individually, not just the chain's final result, because a wrong final answer can trace back to any stage — a wrong classification that sent the input down the wrong branch, a correct branch that retrieved the wrong supporting data, or a correct branch and correct data that a later generation stage still wrote poorly. Comparing the logged intermediate outputs against what a human reviewer would have produced at each stage isolates exactly where the chain diverged from correct, which is the diagnostic step that a single opaque end-to-end prompt would never have made possible in the first place.

Work through each item before checking the answer key. Every option names a real chain-design or refinement concept — the task is matching it to the scenario described.

1. A five-stage chain always runs stages 1 through 5 in the same order for every input, regardless of what any stage returns. Is this chain dynamic?
   • A. Yes, because it has five stages.
   • B. No — no stage's output determines which stage runs next.
   • C. Yes, because each stage's prompt is different.
   • D. It depends only on how long the chain takes to run.
2. A two-stage chain classifies an input, then dispatches to one of three different second-stage prompts based on that classification. Is this chain dynamic?
   • A. No, because it only has two stages.
   • B. Yes — the classification result determines which second-stage prompt runs.
   • C. No, unless it has at least four stages.
   • D. Only if the classification stage uses a different model than the second stage.
3. An agent's routing logic dispatches correctly to one of four branches every time, but an audit finds the classification stage is only 70% accurate. What kind of problem is this?
   • A. A control-flow bug in the router.
   • B. A decision-quality problem in the branch signal, not a mechanism failure.
   • C. Proof the chain should not be dynamic at all.
   • D. An idempotency violation.
4. What is the correct fix for the problem described in question 3?
   • A. Rewrite the branching logic (the if/switch code).
   • B. Refine the classification prompt or its inputs, then re-check against a fixed audit set.
   • C. Remove the branch and make the chain fixed.
   • D. Add a fourth branch option.
5. A review-processing pipeline translates, then scores sentiment, then tags features, for every single review with no exceptions. What kind of chain is this?
   • A. Dynamic, because it has three stages.
   • B. Fixed — every input needs the same stages in the same order.
   • C. Dynamic, because sentiment scores vary.
   • D. Neither a chain nor a script.
6. Why is refining agent decision-making treated as a separate objective from building a dynamic chain?
   • A. They are actually the same objective, just described twice.
   • B. A chain can branch mechanically correctly while still branching on a low-quality signal, which is a separate problem to measure and fix.
   • C. Refinement only applies to fixed chains.
   • D. Decision refinement replaces the need for a router entirely.
7. What does a fixed evaluation set provide that ad hoc spot-checking does not?
   • A. Faster model inference.
   • B. A repeatable basis for confirming a prompt or branch change actually improved accuracy, rather than just shifting which cases fail.
   • C. Automatic idempotency.
   • D. A guarantee the chain is dynamic.
8. Which single fact best distinguishes a fixed chain from a dynamic one?
   • A. The number of stages.
   • B. Whether any stage's choice of the next stage depends on a prior stage's actual output.
   • C. Whether the chain uses more than one model.
   • D. The total latency of the chain.

Answers

1. B. No runtime decision determines the next stage; the order is fixed regardless of any output, which is exactly what makes it a script rather than a dynamic chain.
2. B. A genuine runtime branching decision exists here even though the chain is short — stage count does not determine the label, the presence of a real branch point does.
3. B. The router dispatched exactly as designed; the classification signal it dispatched on on was inaccurate, which is a decision-quality issue, not a mechanism bug.
4. B. Fixing the signal, not the router, and confirming the fix against a fixed set is what separates a measured improvement from a guess.
5. B. Every review needs all three stages in the same order with no exceptions, which is the defining case for a fixed chain, not a missed branching opportunity.
6. B. Mechanism correctness and decision quality are independent properties; a chain can have one without the other, which is why refining decisions is its own objective.
7. B. Without a fixed set to re-run, there is no way to distinguish an actual accuracy improvement from a change that merely shifted which inputs fail.
8. B. This is the single test that resolves the fixed-versus-dynamic question regardless of stage count, latency, or model count.

• A prompt chain sequences prompts so one step's output feeds the next; this buys reliability (one narrow job per stage) and inspectability (a loggable intermediate result per stage) that one giant prompt does not.
• A chain is dynamic only if a genuine runtime decision, based on an intermediate stage's actual output, determines which stage runs next — stage count and length do not make a chain dynamic.
• Decision refinement is a distinct, later step from building the branching mechanism: a chain can branch correctly while still branching on a bad signal, and fixing the signal is a different job than fixing the router.
• Refinement requires visibility into the actual decision made at each branch point, plus a fixed set of cases to re-check after any prompt or control-flow change, so an improvement claim is measured rather than assumed.
• On the exam, expect a scenario to describe a multi-step process and ask whether it counts as dynamic, and a separate scenario to describe a bad branching outcome and ask whether the fix belongs in the router or in the signal the router reads.

This lesson assumed an agent making decisions with only its own reasoning and a single model behind each stage. The next lesson in this module drops that assumption: a real agent often needs to route a task to more than one kind of model — a vision model for an image, a speech model for audio, a language model for reasoning — and fuse what comes back. M2-02 covers integrating multimodal and generative models across text, vision, and audio, including how multimodal retrieval pairs a vector index with a toolkit to ground an agent in more than plain text.