Concurrent Model Execution and Instance Groups in Dynamo-Triton

Batching raises throughput by grouping multiple requests into one forward pass, but even a perfectly batched model instance is still just one instance: it processes one batch at a time, sequentially, and while it is computing that batch, any further incoming requests simply wait in the scheduler's queue until the current batch finishes (or until the batching window closes and a new batch begins, depending on the scheduler's exact timing). If demand is high enough, that one instance becomes the bottleneck regardless of how well it batches, because there is only one execution unit doing the work.

Modern GPUs, meanwhile, frequently have more compute and memory capacity available than a single small-to-medium model instance can fully occupy on its own. A model that only uses a fraction of a GPU's available resources per forward pass, run as a single instance, leaves the rest of that GPU idle — capacity paid for and physically present but not doing useful work. Concurrent model execution exists to close that gap: instead of one instance sitting on a GPU using part of its capacity while the rest sits unused, Dynamo-Triton can load multiple instances onto the same GPU (or spread instances across multiple GPUs) so that more of the available capacity is doing useful work at any given moment.

Identity statement: an instance group is Dynamo-Triton's configuration mechanism for specifying how many copies of a given model should be loaded and made available to receive work, and on which device(s) those copies should run. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states the underlying capability directly: "concurrent model execution — run multiple models, or multiple instances of the same model, in parallel on the same system (including on a single GPU) via instance groups, boosting utilization."

L1 — Intuition

Picture a small classification model that, run as a single instance, uses only a modest slice of a GPU's compute and memory. Configuring an instance group of size four for that model tells Dynamo-Triton: load four independent copies of this model, all resident in GPU memory at the same time, all capable of independently accepting and processing requests. When four requests (or four batches, if dynamic batching is also configured for this model) arrive close together, instead of all four queueing behind one instance, Dynamo-Triton can dispatch them to four different instances simultaneously, each running its own forward pass in parallel with the others.

L2 — Mechanism

Mechanically, each instance in an instance group is a separate, independent load of the model's weights and its own execution context — not a shared model being time-sliced by some other mechanism, but genuinely multiple resident copies. Dynamo-Triton's scheduler routes incoming work across the available instances in a group, and because the instances are independent, they can execute truly in parallel on a GPU that has spare capacity, rather than one instance's work finishing before the next instance's work can start. This is what "boosting utilization" concretely means: the same GPU that could only keep one instance's worth of the model's compute busy at a time now has multiple instances' worth of compute in flight simultaneously, filling in the capacity a single instance left idle.

L3 — Same model, several instances vs. several different models, each with instances

[GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) is explicit that concurrent execution covers two related but distinct configurations: "run multiple models, or multiple instances of the same model, in parallel on the same system." The first configuration — several instances of one model — is the utilization scenario described above: more copies of the same thing, to absorb more concurrent demand for that one model. The second configuration — several different models, running concurrently — solves a different but related problem: a serving system that must offer more than one model (a classifier and an embedding model, say) does not need to dedicate an entire separate GPU to each model if the GPU has spare capacity; instance groups let both models' instances coexist on the same hardware, each independently schedulable, without either model needing the whole device to itself. Both configurations are instance-group mechanics; the difference is simply whether the instances loaded are copies of one model or instances drawn from several different models.

[GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states the boundary as a named trap: "concurrent execution ≠ dynamic batching. Instance groups run models in parallel; batching merges requests. They complement each other." This is the single most important distinction this lesson exists to sharpen, and it is worth stating from more than one angle, because the exam tests it from more than one angle.

What each one actually changes. Dynamic batching (covered in M8-01) changes how many requests are combined into one forward pass sent to one instance. Concurrent execution, via instance groups, changes how many instances exist to receive work in the first place. Batching operates within an instance's queue, grouping requests before they are processed; instance groups operate across instances, determining how many independent copies exist to process work at all. A model can have exactly one instance and still batch requests efficiently within that instance's queue — batching does not require more than one instance to function. A model can also have many instances and batch nothing at all within each instance — running one request at a time per instance while still benefiting from having several instances available in parallel.

Why they complement rather than compete. Because the two mechanisms operate at different levels, a real deployment typically uses both together: several concurrent instances of a stateless model, each instance internally using dynamic batching to group whatever requests land in its own queue. The two together raise throughput along two independent axes — more instances means more simultaneous forward passes in flight; better batching within each instance means each of those forward passes does more useful work per GPU cycle. Treating them as substitutes for each other — assuming that adding instances makes batching unnecessary, or that better batching removes the need for more instances — misses that each addresses a different limit on throughput.

The distractor shape to recognize. A scenario describing a symptom that is actually a lack-of-instances problem (requests queueing up behind a single busy instance while the GPU has spare headroom) should not be answered with "configure better batching" — batching only helps requests that are already queued at the same instance, and does not create more instances. Conversely, a symptom that is actually a batching problem (an instance processing requests one at a time when it could combine several into one forward pass) should not be answered with "add more instances" — more instances add parallel capacity, but they do not make any single instance's own request handling more efficient.

Take a constructed scenario in the exam's preferred style: a stateless image-classification model is deployed on Dynamo-Triton with a single instance and dynamic batching enabled. Under moderate load, latency is acceptable. Under heavier load, requests begin queueing and end-to-end latency rises sharply, even though GPU utilization metrics show the GPU itself is only lightly used during the slow period.

Observed symptom: requests queueing, latency rising, GPU utilization LOW during the slowdown

Candidate explanation A: "batching is misconfigured; a larger batching window
  would fix this."
  -> Does not fit: dynamic batching already groups whatever requests are
     queued to the single instance; a bigger window still routes through
     the SAME one instance, and low GPU utilization suggests the GPU has
     spare capacity that a single instance simply isn't using.

Candidate explanation B: "there is only one instance; the GPU has spare
  headroom that a second (or third) instance could use concurrently."
  -> Fits the evidence: low GPU utilization during a period of queueing
     is the signature of spare capacity sitting idle because too few
     instances exist to use it, not a signature of poor batching.

Fix: increase the instance-group size for this model (e.g., from 1 to 3),
so multiple instances can process work in parallel on the same GPU,
each still using dynamic batching internally for its own queued requests.

The diagnostic habit worth keeping is to check GPU utilization alongside queueing before deciding which mechanism to reach for. Queueing with high GPU utilization suggests the GPU itself is saturated and more instances would not help (there is no spare capacity for them to use) — that symptom instead points toward the model-type or compute-tradeoff questions this module closes with in M8-05, or toward a different GPU. Queueing with low GPU utilization, as in this example, is the more specific signature of too few instances relative to demand, exactly the problem instance groups are built to solve.

A second constructed scenario tests the "several different models" configuration directly. A team serves both a lightweight sentiment classifier and a larger embedding model from the same Dynamo-Triton deployment. Both models individually use only a modest share of a single GPU's memory and compute when run as one instance each, and the team wants to avoid dedicating a separate GPU to each model given that combined they still leave headroom on one device.

Model A (sentiment classifier): small memory footprint, low compute per request
Model B (embedding model): moderate memory footprint, moderate compute per request

Configuration: both models' instance groups target the SAME GPU
  - Model A: instance group size 2 (handles bursty, high-volume, cheap requests)
  - Model B: instance group size 1 (handles steadier, moderate-volume requests)

Result: three total instances (2 of Model A + 1 of Model B) coexist on one
GPU, each independently schedulable, sharing the device's compute and
memory rather than requiring three separate GPUs.

This is the "run multiple models... in parallel on the same system" half of the source material's definition made concrete: instance groups are not restricted to copies of one model, and a serving team's real-world constraint is often GPU headcount, not model count — fitting several models' worth of instances onto fewer GPUs is a direct, practical value of the mechanism, as long as the combined memory and compute demand of every instance loaded still fits within the device's actual capacity.

It is worth being explicit about the constraint that makes this configuration work at all: every instance loaded onto a given GPU, whether it is a second copy of Model A or the first copy of Model B, consumes some share of that GPU's finite memory for its weights and activation workspace, and some share of its compute when it is actively processing a forward pass. Instance groups do not create additional physical GPU capacity out of nothing — they let existing spare capacity be claimed by additional, independently schedulable instances instead of sitting unused behind a single instance that never fully occupies the device. If the combined memory footprint of every instance a team wants to load exceeds what the GPU actually has, the correct response is not to force the configuration through — it is to either reduce the instance count, move some instances to a different GPU, or investigate a memory-reduction technique (quantization, distillation) from Model Optimization before returning to the instance-group question. Reading the resource ceiling correctly, rather than assuming instance groups are an unlimited lever, is itself part of what a professional-level configuration question is checking.

Instance groups sit alongside, not inside, the batching-mode decision from M8-01: a model's batching mode (dynamic or sequence, chosen based on statelessness) and its instance-group size (chosen based on desired concurrency and available GPU headroom) are two independent configuration axes on the same deployed model. A stateful model using sequence batching still benefits from multiple instances if it needs to serve more concurrent sequences than one instance could track state for simultaneously — each instance tracks its own pinned set of sequences, and more instances means more sequences can be served concurrently, exactly as noted when this module first introduced sequence batching.

Instance groups also sit underneath NIM: a NIM container, as M8-02 covered, packages a model with an already-tuned serving configuration, and that packaging can itself include instance-group-style parallelism internally, even though the operator interacting with a NIM container does not typically configure instance groups by hand the way a direct Dynamo-Triton deployment would. The mechanism is the same category of idea — more execution capacity in parallel — whether it is exposed for direct configuration (Dynamo-Triton) or handled inside a prepackaged product (NIM).

This also connects forward to a mechanism this module has not yet named directly: Multi-Instance GPU (MIG), covered fully in M8-04. It is worth flagging the boundary early, because the two are easy to conflate on a first pass. Instance groups are a software-level scheduling concept — several model copies loaded and scheduled by Dynamo-Triton, sharing the GPU's resources opportunistically, with no hardware guarantee that one instance's memory or compute cannot be affected by another instance's workload spiking. MIG is a hardware-level partitioning feature of the GPU itself, carving a single physical device into isolated slices with dedicated memory and compute, each behaving like a smaller independent GPU. A team can use instance groups without MIG (several instances sharing one undivided GPU's capacity), MIG without instance groups (dedicating a single MIG slice to a single model instance for guaranteed isolation), or both together (instance groups running within one or more MIG slices). Knowing that one is a scheduling decision and the other is a hardware-partitioning decision prevents treating them as the same lever.

Model Deployment is objectives 8.1 through 8.3 and carries 9% of the NCP-GENL blueprint. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) lists concurrent model execution as one of Dynamo-Triton's headline capabilities alongside dynamic batching, and names the batching-versus-concurrency mix-up as one of the domain's three explicitly called-out common traps, alongside the dynamic-versus-sequence-batching trap M8-01 covers and the TensorRT-versus-Triton trap covered there and in M8-02. That pattern — three named traps, each pairing two real, individually correct concepts that get swapped for each other — is this domain's consistent house style, and concurrent execution's pairing with dynamic batching is the second of the three.

How the question tends to be phrased

Expect a direct identification question ("running several instances of the same model in parallel on one GPU is achieved via:" with instance groups / concurrent model execution as the keyed answer against gradient accumulation, masked language modeling, and beam search as distractors — unrelated techniques borrowed from other domains). [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) contains this exact self-check item. Expect also a scenario question describing a queueing symptom (as in section 4) and asking whether the fix is a batching change or an instance-count change, testing whether the GPU-utilization signal is read correctly.

What the distractors typically look like

The standard trap offers dynamic batching as the fix for a problem that is actually a too-few-instances problem, or offers "add more instances" as the fix for a problem that is actually inefficient per-instance batching — both are real, correct mechanisms, misapplied to the other mechanism's problem, in this domain's consistent distractor style of a real technique attached to the wrong symptom.

Work through each item before checking the answer key. Every option names a real serving mechanism — the task is matching it to the scenario, not spotting an invented distractor.

1. A single model instance is fully occupied, requests are queueing, and GPU utilization metrics show the GPU is only lightly used. What is the most likely fix?
   • A. Increase the dynamic batching window.
   • B. Increase the instance-group size for this model.
   • C. Switch to sequence batching.
   • D. Recompile the model with TensorRT.
2. What does an instance group actually configure?
   • A. How many requests get grouped into one forward pass.
   • B. How many copies of a model (or which models) are loaded and available to receive work.
   • C. Which GPU architecture the model targets.
   • D. The model's precision (FP16 vs INT8).
3. Can instance groups host instances of more than one distinct model on the same GPU?
   • A. No — one GPU can only ever run one model.
   • B. Yes — as long as the combined resource demand of every loaded instance fits the device.
   • C. Only if the models share identical architectures.
   • D. Only through Kubernetes, never through Dynamo-Triton directly.
4. Why doesn't a larger dynamic batching window fix a too-few-instances problem?
   • A. Batching windows only apply to encoder models.
   • B. A larger window still routes every request through the same limited number of instances; it does not add execution capacity.
   • C. Batching windows cannot be resized without redeploying.
   • D. Dynamic batching is incompatible with instance groups.
5. Do instances within the same instance group share state with each other?
   • A. Yes, always, by default.
   • B. No — each instance is an independently loaded copy with its own execution context.
   • C. Only for stateless models.
   • D. Only if explicitly synchronized through Kubernetes.
6. Queueing is observed alongside HIGH GPU utilization. What does this suggest about the fix?
   • A. Adding more instances will definitely resolve it, since the GPU has room.
   • B. The GPU itself may be saturated, so more instances may not help — this points elsewhere, such as model-type compute tradeoffs.
   • C. This always means dynamic batching is misconfigured.
   • D. This is unrelated to concurrency and points to a data-loading bug.
7. Which pairing correctly matches a mechanism to the level it operates at?
   • A. Dynamic batching operates across instances; instance groups operate within one instance's queue.
   • B. Instance groups operate across instances; dynamic batching operates within one instance's queue.
   • C. Both operate at exactly the same level and are interchangeable.
   • D. Neither operates at the instance level.
8. A stateful model using sequence batching needs to serve more concurrent conversations than one instance can track. What is the appropriate response?
   • A. Switch the model to dynamic batching instead.
   • B. Increase the instance-group size, so more instances exist to each track their own pinned set of sequences.
   • C. Reduce the batching window to zero.
   • D. Move the model off Dynamo-Triton entirely.

Answers

1. B. Queueing with low GPU utilization is the signature of too few instances relative to demand, not a batching problem.
2. B. Instance groups configure how many model instances (of one model, or of several distinct models) are loaded and available to receive work.
3. B. Instance groups are not restricted to one model; several distinct models' instances can coexist on one GPU as long as resource demand fits.
4. B. A bigger window still funnels requests through the same number of instances — it does not create additional execution capacity.
5. B. Each instance is an independent load of the model with its own execution context; nothing is shared between instances by default.
6. B. High utilization alongside queueing suggests the GPU itself may already be saturated, which points toward a compute-tradeoff or hardware question rather than an instance-count fix.
7. B. Instance groups add parallel execution units (across instances); batching groups requests within one instance's own queue.
8. B. More instances mean more independently pinned sets of tracked sequences, directly relieving the per-instance sequence-tracking ceiling.

Concurrent model execution, configured through instance groups, controls how many copies of a model (or how many different models) are loaded and available to process work in parallel on a system, including a single GPU. Dynamic batching controls how many independent requests get grouped into one forward pass sent to a single instance's queue. The two operate at different levels — across instances versus within one instance's queue — and they complement rather than replace each other: a real deployment commonly runs several concurrent instances, each internally using dynamic batching for its own queued requests.

Yes — instance groups let Dynamo-Triton load several independent copies of the same model's weights onto one GPU simultaneously, as long as the GPU's memory and compute capacity can accommodate all of them. Each instance is a separate execution unit capable of independently accepting and processing requests, and running several of them in parallel is exactly how a model that only uses a fraction of a GPU's capacity per instance can still make use of the GPU's remaining headroom, raising overall utilization rather than leaving that capacity idle.

• Instance groups run multiple copies of a model, or multiple different models, in parallel on one system, including a single GPU, raising utilization by keeping more available capacity busy at once.
• Concurrent execution and dynamic batching are not the same mechanism. Instance groups add parallel execution units; batching groups requests within one unit's queue — they complement, not substitute for, each other.
• The diagnostic signal that distinguishes the two problems is GPU utilization. Queueing with low utilization points to too few instances; queueing with high utilization points elsewhere.
• Instance groups can host instances of several different models on one GPU, not only copies of a single model, as long as combined resource demand fits the device.
• Each instance is independently loaded and does not share state with other instances — relevant when combining instance groups with a stateful, sequence-batched model.
• This is the second of Model Deployment's three named trap pairs, alongside dynamic-vs-sequence batching (M8-01) and TensorRT-vs-Triton (touched in M8-02) — all three pair two real, individually correct mechanisms that get swapped for each other.
• Model Deployment is 9% of the NCP-GENL blueprint, and instance groups are one of Dynamo-Triton's headline capabilities alongside dynamic batching.

Instance groups answer how many execution units exist to receive work on a given piece of hardware. They say nothing about how that hardware gets packaged for reproducible deployment in the first place, or how a GPU itself can be partitioned to guarantee isolation between different tenants' workloads rather than merely sharing capacity opportunistically. That is the subject M8-04 picks up next: containerization, Kubernetes, and Multi-Instance GPU, moving from "how many instances run in parallel" to "how the infrastructure around those instances is packaged, scaled, and isolated."

Next: M8-04 covers containerization, Kubernetes, and Multi-Instance GPU — how Docker packages a model and its runtime, how Kubernetes scales and health-checks that package, and how MIG partitions a single GPU into hardware-isolated instances for multi-tenant serving.