Dynamic Batching vs. Sequence Batching in Dynamo-Triton

A GPU is fastest when it processes many requests' worth of arithmetic in one matrix multiplication rather than many small ones back to back — this is simply how modern accelerator hardware is built, and it is true independent of any particular model or framework. The gap between "many small requests arriving over time" and "one large batched matrix multiplication" is exactly what an inference server's scheduler is for: it collects individual requests and decides how to group them into batches before handing that batch to the model for a forward pass. Dynamo-Triton is NVIDIA's production inference server built to make that decision well, across frameworks and hardware, and it exposes more than one scheduling strategy because "group requests together" means something different depending on whether those requests are independent of each other or bound together as one ongoing sequence.

That last distinction — independent versus bound-together — is the entire subject of this lesson. Two requests from two different users asking two unrelated one-shot questions are independent: nothing about processing them together or apart changes the correctness of either answer. Two requests that are turn three and turn four of the same multi-turn conversation are not independent: turn four's correct answer depends on state accumulated from turn three (and one and two before it), and if that state is lost or attached to the wrong instance, turn four's answer is simply wrong, not just slower.

It helps to notice that this is not a question the model architecture answers by itself, in the sense of one architecture always being stateless and another always being stateful. A single decoder-only model can be deployed statelessly — score this one prompt, return this one completion, forget everything — or statefully, if the serving layer is asked to hold a running conversation's context between calls so the client does not have to resend the whole history every time. What determines statelessness is the deployment contract between client and server, not some fixed property baked into the model file. That is precisely why Dynamo-Triton exposes the choice as a per-model scheduler configuration rather than inferring it automatically: nothing in a model's weights announces whether the serving team intends to keep cross-request state, so the team has to declare it.

Identity statement: dynamic batching is Dynamo-Triton's scheduler for stateless models — it collects multiple independent inference requests that arrive close together in time and combines them into a single batch, formed at runtime rather than fixed in advance, before running one forward pass over the whole batch. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states this directly: dynamic batching is described as combining "multiple incoming inference requests into a single batch formed at runtime to raise throughput," and the source material adds explicitly that it is "intended for stateless models."

L1 — Intuition

Picture a classification model that scores one image at a time: is this a cat, a dog, or neither. Four users submit four unrelated images within the same few milliseconds. None of these four requests has anything to do with any other — no user's image depends on another user's result, and there is no notion of "session" connecting them. Dynamic batching is what lets the server notice that four requests just arrived, wait a small configurable window for a few more to show up, then stack all four images into one batch tensor and run the model once instead of four separate times. Throughput goes up because the GPU spends its cycles on one efficient batched operation rather than four small ones with per-request overhead repeated four times.

L2 — Mechanism

Mechanically, the scheduler maintains a queue of incoming requests and a configurable batching window — a small amount of time (and/or a target batch size) it is willing to wait before it stops accepting more requests into the current batch and dispatches what it has. Once the window closes or the batch fills, whichever comes first, the collected requests are concatenated along the batch dimension and sent through the model together in a single forward pass. The result tensor is then split back apart so each original requester gets only their own output. Crucially, nothing about this process requires any request to know about, or be routed consistently with, any other request — batches can be formed from an arbitrary mix of callers each time, because the model has no notion of a caller's history to preserve between calls.

L3 — Why this specific mechanism is unsafe for stateful models

The property that makes dynamic batching correct for stateless models — that any request can be grouped with any other request, in any batch, on any model instance, because nothing carries over between calls — is exactly the property a stateful model violates. If a model instance maintains some internal state across a sequence of calls (a decoder's growing KV cache in an ongoing generation, or a streaming ASR model's accumulated audio context), then routing turn four of a conversation to a different physical model instance than turn three used, or interleaving turn four with an unrelated caller's turn one inside the same batch without regard for which instance is tracking which state, either loses the state outright or corrupts it by attaching it to the wrong sequence. Dynamic batching's runtime, order-agnostic grouping is a feature for stateless work and a correctness bug for stateful work — the same mechanism, evaluated against a different requirement, flips from "raises throughput" to "silently returns wrong answers."

Identity statement: sequence batching is Dynamo-Triton's scheduler for stateful models — every request that belongs to the same ongoing sequence is routed to the same model instance for the sequence's entire lifetime, so that whatever state the model accumulates call to call stays attached to the correct conversation. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) names this directly: "sequence batching — the scheduler for stateful models, where requests belong to a sequence and must route to the same model instance."

Where dynamic batching's whole value proposition is "any request, any instance, any grouping, because nothing persists," sequence batching's whole value proposition is the opposite: "this request must go where its predecessor went, because something does persist." A sequence in this sense is not a single request — it is a labeled, ordered stream of related requests (a conversation id, a session id, a stream id) that the client marks as belonging together, typically with an explicit start flag on the first request and an end flag on the last. The scheduler uses that sequence identity to pin the sequence to one model instance and keep it there for every subsequent request in that sequence, regardless of what other unrelated traffic the server is handling at the same moment.

This pinning is not a batching optimization in the throughput sense that dynamic batching is — it is a correctness requirement first, and any throughput benefit sequence batching offers (and it can still combine multiple different sequences' current-turn requests into a batch, as long as each sequence's turn goes to its own correctly-pinned instance) is secondary to the guarantee that state never gets misrouted. A server handling both stateless and stateful models at once needs both schedulers configured correctly, model by model, because using one server does not mean every model on it behaves the same way.

It is worth being precise about what "the same model instance" means operationally, because it is easy to picture a single physical GPU when what actually matters is a specific loaded copy of the model. If a stateful model is deployed with several instances spread across several GPUs to serve more concurrent sequences than one instance could handle alone, sequence batching's guarantee is still per-sequence: sequence 77 is pinned to whichever one of those instances first accepted its opening request, and every later request in sequence 77 must reach that same instance specifically, not merely "some instance of this model somewhere on the server." Two different sequences can be, and typically are, pinned to two different instances simultaneously, each instance tracking only the sequences assigned to it. This is what allows a stateful model to scale to many concurrent conversations at all — without multiple instances, a single stateful model could only ever track one sequence's state at a time.

Take a constructed scenario: a Dynamo-Triton deployment serves two models — Model A, a stateless sentiment classifier, and Model B, a stateful multi-turn chat model that maintains conversation state across turns. In one short window, four requests arrive:

Request 1: Model A, review text "great product" (no sequence id — independent)
Request 2: Model A, review text "terrible service" (no sequence id — independent)
Request 3: Model B, sequence_id=77, turn=3, "and what about pricing?" (start_flag=false)
Request 4: Model B, sequence_id=77, turn=4, "compare that to last year" (start_flag=false)

Correct routing:
  Requests 1 and 2 -> dynamic batcher for Model A
    -> combined into one batch, one forward pass, results split back to each caller
  Requests 3 and 4 -> sequence batcher for Model B, sequence_id=77
    -> BOTH must land on whichever model instance already holds sequence 77's state
    -> turn 4 depends on turn 3's (and turns 1-2's) accumulated context

The scenario's numbers are illustrative — this is a constructed scenario, not a measured benchmark — but the routing logic is not: requests 1 and 2 have no sequence id and no shared state to protect, so dynamic batching's runtime grouping is not just permitted but is exactly the throughput win it exists to provide. Requests 3 and 4 carry the same sequence_id, meaning they are turns in one ongoing conversation, and the only correct behavior is to pin both to the same instance that has been accumulating Model B's state for sequence 77 since turn 1. If the server instead treated requests 3 and 4 as eligible for arbitrary dynamic batching — grouping turn 4 with some unrelated caller's fresh, turn-1 request purely because they arrived close together in time — turn 4 would either lose sequence 77's accumulated context or, worse, silently combine with a different sequence's state, producing a coherent-looking but wrong answer with no error thrown.

A second constructed scenario makes the failure mode concrete from the operations side. A team deploys a streaming speech-recognition model — inherently stateful, since each new audio chunk's correct transcription depends on acoustic and language-model state built up from every prior chunk in the same utterance — but configures it under Dynamo-Triton's dynamic batcher instead of the sequence batcher, because dynamic batching is the default most teams reach for first and the model's stateful nature was not flagged during the deployment review.

Symptom observed in production:
  - Short utterances (1-2 chunks): transcriptions look mostly correct
  - Long utterances (10+ chunks): transcription quality degrades partway through,
    sometimes resetting mid-sentence as if the model "forgot" the earlier audio

Root cause, worked backward from the symptom:
  1. Dynamic batching groups whichever requests arrive close together, without
     regard for which physical instance processed a chunk's predecessor.
  2. Under load, chunk N of a long utterance can be routed to a different
     instance than chunk N-1 was, because dynamic batching carries no
     "same instance" guarantee.
  3. That different instance has no accumulated state for this utterance --
     it starts effectively fresh, explaining the mid-utterance "reset."
  4. Short utterances mask the bug because there is less opportunity for a
     chunk to land on a different instance before the utterance ends.

Fix: reconfigure the model under the sequence batcher with sequence-id-based
routing, pinning every chunk of one utterance to the instance handling that
utterance's state for its full duration.

The diagnostic habit worth keeping from this example is that a stateful-model bug caused by the wrong batching mode does not usually announce itself as an error — it shows up as a quality regression that gets worse the longer or more concurrent the workload is, which is a subtler and more expensive failure to trace than a crash would be.

Dynamic and sequence batching answer "how do I group requests correctly," but Dynamo-Triton offers other capabilities layered around that scheduling decision that this module covers next. Concurrent model execution, which runs multiple instances of the same model (or multiple different models) in parallel on one system, is a separate mechanism from either batching mode — it is about how many copies of a model are available to receive work, not about how requests are grouped once they arrive at an instance. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) is explicit that these are complementary, not the same mechanism: batching decides how requests going to one instance are grouped, while concurrent execution decides how many instances exist to receive them in the first place. A deployment commonly uses both at once — several concurrent instances of a stateful chat model, each one internally using sequence batching to keep its own set of pinned conversations straight.

It is also worth being precise about what Dynamo-Triton is not, since this domain's source material calls the confusion out directly: Dynamo-Triton is the server, and a separate NVIDIA tool, TensorRT, is the compiler that optimizes a model's kernels for a target GPU before that model is ever loaded into a server at all. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states the boundary as a named exam trap: "TensorRT optimizes; Triton serves." A model can be TensorRT-compiled and still served through either of Dynamo-Triton's batching modes — compilation and scheduling are two different stages of the same model's path to production, not competing choices.

Model Deployment is objectives 8.1 through 8.3 and carries 9% of the NCP-GENL blueprint — a mid-sized domain, and the outline's own weight note frames the entire module around exactly this distinction: Dynamo-Triton as the general inference server, with the batching-mode choice as one of its most concretely testable behaviors. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states the scope directly: "Professional-level questions test the roles of Triton vs NIM and the right batching mode for stateless vs stateful models." That framing puts this lesson's exact distinction — right batching mode for the right statefulness — at the center of what a professional-level question is checking.

How the question tends to be phrased

Expect a scenario that describes a model's behavior (does it maintain context across calls, or does every call stand alone) and asks which batching mode Dynamo-Triton should use, or a scenario that describes a symptom (a multi-turn conversation losing context under load, exactly as in section 5's worked example) and asks you to identify the misconfiguration. A more direct phrasing simply asks, "dynamic batching in Dynamo-Triton is intended primarily for:" with stateless models as the keyed answer against stateful sequence models, training jobs, and tokenizer training as distractors — real infrastructure concepts attached to the wrong question, in this domain's consistent distractor style. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) contains this exact self-check item.

What the distractors typically look like

The standard trap is offering dynamic batching as the correct scheduler for a described stateful scenario, or describing sequence batching's sequence-id routing and asking whether it applies to an described independent, one-shot workload (it does not, though nothing breaks if you use it there — sequence batching is simply unnecessary overhead for traffic with no state to protect). A second recurring distractor conflates batching with concurrent execution, framing "running more instances" as if it were a batching mode; that mix-up is corrected directly in section 7 and taken up in full in the module's next lesson.

Dynamic batching combines independent, stateless requests into a single batch formed at runtime purely to raise throughput, with no requirement about which physical model instance handles any particular request. Sequence batching instead exists for stateful models, where every request belonging to one identified sequence — a multi-turn conversation, a streaming session — must be routed to the same model instance for the sequence's whole lifetime, because that instance is the only one holding the sequence's accumulated state. The two solve different problems: one raises throughput for work with no memory requirement, the other preserves correctness for work that has one.

Because dynamic batching's defining feature — that any request can be grouped with any other, on any available instance, since nothing carries over between calls — is precisely the assumption a stateful model violates. A stateful model's correctness depends on every request in a sequence reaching the same instance that has been accumulating that sequence's state; dynamic batching offers no such guarantee, since its whole point is flexible, order-agnostic grouping for throughput. Using it on a stateful model does not just under-perform, it produces wrong answers by losing or misattributing state, typically without raising any error.

Work through each item before checking the answer key. Every option names a real serving concept — the task is matching it to the scenario described, not spotting an obviously invented distractor.

1. A model scores individual product reviews for sentiment, with no relationship between one review and the next. Which Dynamo-Triton scheduler fits?
   • A. Sequence batching, to guarantee correctness.
   • B. Dynamic batching, since no state needs to be preserved between requests.
   • C. Neither — this workload cannot be batched.
   • D. Concurrent model execution, not a batching mode at all.
2. A multi-turn chatbot's answers occasionally seem to "forget" earlier turns once traffic increases. What is the most likely misconfiguration?
   • A. The model was quantized too aggressively.
   • B. Dynamic batching is being applied to what should be a sequence-batched, stateful model.
   • C. The GPU ran out of memory.
   • D. TensorRT failed to compile the model.
3. What does sequence batching guarantee that dynamic batching does not?
   • A. Lower latency for every request.
   • B. That every request in one sequence reaches the same model instance.
   • C. A larger maximum batch size.
   • D. Automatic use of TensorRT compilation.
4. Which of the following is true about dynamic batching's request grouping?
   • A. It requires a sequence id on every request.
   • B. It is order-agnostic — any independent request can join any batch.
   • C. It only works with encoder-only models.
   • D. It replaces the need for concurrent model execution.
5. A streaming ASR model's transcription quality degrades mid-utterance only under heavy concurrent load. What diagnosis does this pattern best support?
   • A. The acoustic model itself is undertrained.
   • B. A chunk of the utterance was likely routed to a different instance than its predecessor, losing accumulated state.
   • C. The audio codec is lossy.
   • D. Dynamic batching's batching window was set too small.
6. Why doesn't applying dynamic batching to a stateful model typically throw a visible error?
   • A. Dynamo-Triton silently rejects stateful models from the dynamic batcher.
   • B. The scheduler has no way to know the model is stateful; it groups requests exactly as designed, and the model produces a plausible-looking but wrong answer.
   • C. Stateful models cannot be loaded into Dynamo-Triton at all.
   • D. Errors are always thrown but are commonly suppressed by default logging settings.
7. Which statement correctly separates batching mode from concurrent model execution?
   • A. They are two names for the same mechanism.
   • B. Batching decides how requests to one instance are grouped; concurrent execution decides how many instances exist to receive work.
   • C. Concurrent execution replaces the need to choose a batching mode.
   • D. Batching mode only applies when concurrent execution is disabled.
8. A team deploys a model that is TensorRT-compiled and asks whether that changes which batching mode it should use. What is the correct answer?
   • A. TensorRT-compiled models always require sequence batching.
   • B. No — TensorRT compiles and optimizes the model's kernels; the batching-mode choice still depends solely on whether the model is stateless or stateful.
   • C. TensorRT-compiled models cannot use dynamic batching.
   • D. Compilation determines batching mode automatically.

Answers

1. B. No request depends on any other, so dynamic batching's flexible, order-agnostic grouping is exactly the throughput win it is designed to provide.
2. B. Load-dependent, intermittent "forgetting" in a stateful model is the signature symptom of a batching-mode misconfiguration, not a model-quality issue.
3. B. Sequence batching's defining guarantee is same-instance routing for every request in a sequence; dynamic batching makes no such promise because it does not need to.
4. B. Independence is the default assumption for dynamic batching — no sequence id or other membership signal is required.
5. B. The pattern described (works fine at low load, degrades under concurrency, mid-utterance) is the textbook signature of a chunk landing on an instance that does not hold the utterance's prior state.
6. B. The scheduler is doing exactly what dynamic batching is designed to do; nothing in the mechanism itself detects that the model was stateful, so the failure surfaces as a wrong answer, not an exception.
7. B. These are complementary, separate mechanisms: one is about request grouping, the other about how many model copies exist to be scheduled against.
8. B. Compilation (TensorRT) and scheduling (Dynamo-Triton's batching mode) are two independent stages of a model's deployment path; one does not determine the other.

• Dynamic batching is for stateless models; sequence batching is for stateful models. This is the domain's single most tested distinction, and it is stated as a named trap in the source material.
• Dynamic batching's grouping is order-agnostic by design — any independent request can join any batch, because nothing persists between calls for a stateless model.
• Sequence batching's whole job is a routing guarantee, not a throughput trick: every request in one sequence must land on the same instance that holds that sequence's state.
• Applying dynamic batching to a stateful model does not error loudly — it corrupts or loses state silently, typically surfacing as a load-dependent quality regression rather than a crash.
• Batching mode is a per-model configuration choice, driven by whether that specific model is stateless or stateful, not a server-wide default.
• Batching and concurrent model execution are separate mechanisms that combine in practice: several concurrent instances of a model, each internally scheduled by the correct batching mode for that model's statefulness.
• Model Deployment is 9% of the NCP-GENL blueprint, and this lesson's distinction anchors the module's own guiding question about which serving technology does which job.

Dynamic and sequence batching answer how requests get grouped once they arrive at a model instance. They say nothing about how many instances of a model exist to receive that traffic in the first place, or what happens when a team wants several copies of the same model — or several different models entirely — running in parallel on one system, including a single GPU. That is the subject M8-03 picks up next: concurrent model execution and instance groups, the mechanism this lesson has already flagged as complementary to, but distinct from, everything covered here.

Next: M8-03 covers concurrent model execution and instance groups — how Dynamo-Triton runs multiple model copies in parallel, and why that mechanism is not a substitute for correct batching-mode selection.