Containerization, Kubernetes, and Multi-Instance GPU for LLM Serving

Identity statement: containerization, using Docker, packages a model together with its runtime — the exact libraries, framework version, system dependencies, and configuration the model needs to run correctly — into a single, portable unit that behaves identically wherever it is deployed. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states this directly: "containerized pipelines (Docker) package model + runtime for reproducible deployment."

L1 — Intuition

The problem containerization solves is one every team that has ever moved code from a laptop to a server has hit: "it works on my machine" is not the same claim as "it works in production," because a laptop and a production server rarely have identical library versions, system configurations, or installed dependencies. A model that depends on a specific version of a deep learning framework, a specific CUDA toolkit version, and a handful of other packages can behave differently — or fail outright — if any of those versions differ between where it was developed and where it is deployed. A container eliminates that gap by bundling the model together with the exact environment it needs, so the same container image runs identically on a developer's workstation, a cloud instance, or an on-premises server.

L2 — Mechanism

A Docker container is built from an image — a layered, versioned specification of a filesystem and the processes that should run inside it — that includes the model's weights (or a path to load them), the inference engine or framework, and every system-level dependency the two require. Running that image produces a container: an isolated process (or set of processes) that behaves as if it has its own filesystem and environment, separate from whatever else is running on the host machine, while still sharing the host's kernel and, for GPU workloads, being granted access to the host's GPU hardware through a GPU-aware container runtime. Because the image is a fixed, versioned artifact, deploying "this image" to any machine that can run Docker and has compatible GPU drivers produces the same behavior every time — reproducibility is the direct consequence of packaging the dependencies alongside the code rather than assuming they already exist correctly on the target machine.

L3 — Why reproducibility matters more, not less, for GPU-accelerated inference

Reproducibility matters for any software deployment, but it carries extra weight for GPU-accelerated model serving specifically, because the dependency chain is deeper and more version-sensitive than typical application code: a GPU driver version, a CUDA toolkit version, a specific build of an inference engine, and the model's own expected input/output format all have to align correctly, and a mismatch anywhere in that chain can silently produce wrong numerical results rather than a clean crash — a subtly miscompiled kernel or an incompatible precision setting can still run, just incorrectly. Packaging the entire validated combination into one container image, rather than trusting that a target machine's separately installed dependencies happen to match, is what makes "the same model behaves the same way everywhere it is deployed" an achievable guarantee rather than a hope.

There is one dependency a container cannot fully absorb, and it is worth naming so the reproducibility claim is not overstated: the host machine's own GPU driver still has to be present and compatible with whatever the container expects, because a container shares the host's kernel and, through a GPU-aware runtime, is granted access to the host's physical GPU rather than bringing its own virtualized one. A container that bundles a CUDA toolkit version requiring a newer driver than the host machine actually has installed will still fail on that host, no matter how correctly everything else inside the container is packaged. This is why "containerized" and "guaranteed to run everywhere with zero host-side prerequisites" are not quite the same claim — the container removes almost every source of environment drift, but the host's own GPU driver compatibility is the one piece of the puzzle that still has to be verified independently, typically as a fleet-wide baseline requirement rather than something each container image can enforce on its own.

Identity statement: Kubernetes orchestrates containerized AI workloads at the fleet level — handling horizontal and vertical scaling, health checks, and rolling updates — so that a containerized model runs reliably across many machines rather than being manually managed on each one individually. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states this directly: "Kubernetes orchestrates AI workloads: horizontal/vertical scaling, health checks, rolling updates."

A single container running on a single machine answers "how do I package this model reproducibly," but it does not answer "what happens when demand exceeds what one container can serve," "what happens when that container crashes," or "how do I deploy a new version without an outage." Kubernetes exists to answer exactly those operational questions, for containers in general and for containerized model-serving workloads specifically.

Horizontal scaling means running more copies (replicas) of the same containerized model across more machines as demand rises, and fewer as demand falls — conceptually related to, but operating at a coarser, cluster-wide level than, the instance-group scaling M8-03 covered inside a single Dynamo-Triton deployment on one system. Where an instance group adds more model copies within one running Dynamo-Triton process on one machine (or a small set of GPUs on that machine), horizontal scaling in Kubernetes adds more whole replicas of the entire containerized deployment, potentially spread across many different physical machines in a cluster — the two mechanisms nest, with instance groups operating inside each individual replica Kubernetes manages. Vertical scaling means giving an existing replica more resources (more CPU, more memory, or a different GPU allocation) rather than adding more replicas. Health checks mean Kubernetes periodically verifies that a running container is actually responsive and functioning — not merely that the process has not crashed, but that it is genuinely able to serve requests — and automatically restarts or replaces a container that fails those checks, without requiring a human to notice and intervene. Rolling updates mean deploying a new container image version gradually, replacing old replicas with new ones a few at a time while keeping enough old replicas running to continue serving traffic, so a version upgrade does not require taking the whole service offline.

It is worth being precise about what a rolling update actually guards against, because the value is easy to understate. Deploying a new model version — a retrained checkpoint, an updated quantization, a bug fix in the serving code — by simply stopping every old replica and starting every new one at once creates a window, however brief, where no replica is available to serve traffic at all. A rolling update instead brings up a small number of new-version replicas, waits for their health checks to pass, then retires an equal number of old-version replicas, repeating in small increments until every replica is running the new version, with the total serving capacity never dropping to zero at any point in the process. If a health check on a new replica fails partway through, Kubernetes can halt the rollout before more old replicas are retired, limiting the blast radius of a bad deployment to a small fraction of total capacity rather than the whole fleet.

[GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) names one more capability worth placing correctly in this stack: "ensembles chain multiple models/steps (e.g., pre-process → model → post-process) into one served pipeline." An ensemble in this deployment sense is not the machine learning meaning of combining several models' predictions to improve accuracy — it is a serving-pipeline concept, where a request passes through a defined sequence of stages (a preprocessing step, then a model's forward pass, then a postprocessing step, or several models chained in sequence) and the serving infrastructure treats that whole sequence as a single served unit, handling the hand-off between stages internally rather than requiring the calling client to make several separate round-trip calls. This is a Dynamo-Triton-level capability that sits alongside batching and instance groups, and it is worth keeping distinct from both Kubernetes (which orchestrates containers, not intra-pipeline stage hand-offs) and MIG (which partitions hardware, not pipeline logic).

Identity statement: Multi-Instance GPU (MIG) partitions a single physical GPU into multiple, hardware-isolated instances, each with its own dedicated slice of memory and compute, so that several tenants or workloads can share one GPU with guaranteed isolation rather than opportunistic sharing. [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) states this directly: "MIG (Multi-Instance GPU) partitions a GPU into isolated instances for multi-tenant/multi-task serving."

The word "isolated" is the load-bearing difference between MIG and the instance-group sharing M8-03 described. Instance groups let several model instances share one undivided GPU's resources, scheduled by software (Dynamo-Triton's scheduler) — a busy instance can, in principle, consume more of the GPU's shared resources at a given moment, at the expense of another instance's available capacity in that moment, because nothing hardware-level enforces a strict boundary between them. MIG instead divides the physical GPU itself at the hardware level into fixed-size slices, each with its own dedicated memory and compute allocation that another MIG instance cannot use even if it is idle. A workload running in one MIG instance cannot starve or be starved by a workload running in a different MIG instance on the same physical card, because the hardware itself, not a scheduler's cooperation, enforces the boundary.

This distinction is exactly why MIG is named for "multi-tenant" serving specifically: when different instances on the same GPU belong to different customers, teams, or workloads with strict isolation or predictable-performance requirements, software-level sharing's "no strict boundary" property is a liability, not just an efficiency question — one tenant's traffic spike should never be able to degrade another tenant's guaranteed performance. MIG's hardware partitioning provides that guarantee at the cost of some flexibility: a MIG slice's resources are fixed once configured, so a model instance in one slice cannot temporarily borrow another slice's idle capacity the way two instance-group instances sharing an undivided GPU implicitly can.

A useful way to hold the two mechanisms apart for exam purposes is to ask what kind of question the deployment is actually answering. "How do I let this one model use more of the GPU when it needs it, and less when it doesn't, without wasting capacity" is an instance-group question — the answer benefits from flexible, software-scheduled sharing. "How do I guarantee that tenant A's workload can never be slowed down by tenant B's workload on the same physical card, contractually or for compliance reasons" is a MIG question — the answer requires the hardware boundary that only a physical partition provides, because a promise enforced by a scheduler's cooperation is not the same strength of promise as one enforced by the silicon itself. A scenario that states an isolation requirement explicitly, in the language of tenants, contracts, or guaranteed performance, is signaling MIG; a scenario that states a utilization goal without any isolation language is signaling instance groups instead.

Take a constructed scenario in this domain's preferred style: a platform team must serve three different internal customers' models on shared GPU hardware, with a contractual requirement that no customer's traffic spike can degrade another customer's latency, while also needing the ability to deploy new model versions without downtime and to automatically recover from a crashed container.

Requirement 1: no customer's traffic spike can degrade another's latency
  -> points to MIG: hardware-level isolation is the only mechanism among
     the four that provides a guaranteed, not-just-probable, boundary

Requirement 2: deploy new model versions without downtime
  -> points to Kubernetes rolling updates: gradual replacement of old
     replicas with new ones while traffic keeps flowing

Requirement 3: automatically recover from a crashed container
  -> points to Kubernetes health checks: automatic detection and
     restart/replacement of an unresponsive container

Requirement 4 (implicit): the model itself must run identically across
  however many machines this deployment spans
  -> points to Docker containerization as the baseline packaging choice
     underneath both the MIG-partitioned hardware and Kubernetes'
     orchestration layer

The predicted infrastructure, reasoning from the stated constraints, layers all four together rather than picking one: a Docker container packages each customer's model reproducibly; each customer's container(s) run inside a dedicated MIG slice, guaranteeing hardware isolation between customers; Kubernetes orchestrates the containers across the cluster, providing the rolling-update and health-check behavior the requirements ask for. Instance groups can still apply inside a given MIG slice if a single customer's own model needs several concurrent instances within its guaranteed slice of the GPU — the four mechanisms are not mutually exclusive alternatives, they are layers that commonly stack.

The exam's style also tests the boundary from the other direction. A team serves one internal model to one internal audience, on a GPU it does not share with any other tenant, and simply wants that one model's instances to make full, flexible use of the entire GPU's capacity as demand fluctuates throughout the day.

No multi-tenant isolation requirement stated -- only one workload, one
audience, one GPU

MIG would fix that GPU's capacity into rigid, pre-defined slices, which
would prevent the single workload from flexibly claiming more or less
of the GPU's full capacity as its own demand rises and falls throughout
the day.

Better fit: instance groups sharing the UNDIVIDED GPU, letting Dynamo-
Triton's scheduler flexibly allocate available capacity across the
model's own instances as demand changes, with no need for a hardware
isolation guarantee since there is no other tenant to isolate from.

Reading these two examples together makes the decision rule concrete: MIG's hardware guarantee is valuable precisely when isolation between independent workloads matters more than flexible, dynamic sharing of one workload's own capacity; when there is only one workload and no isolation requirement, MIG's rigidity is a cost with no corresponding benefit, and software-level instance-group sharing is the better fit.

Model Deployment is objectives 8.1 through 8.3, carrying 9% of the NCP-GENL blueprint, and this subsection of the domain's source material groups containerization, Kubernetes, ensembles, and MIG together as the infrastructure layer underneath the serving technologies the rest of the module covers. A professional-level question in this territory is less likely to ask "define Kubernetes" in isolation and more likely to present an infrastructure requirement (isolation, uptime during upgrades, reproducibility across environments) and ask which of these tools actually addresses it, in the pattern this lesson's two worked examples model directly.

How the question tends to be phrased

Expect a direct identification question ("partitioning a single GPU into isolated instances for multi-tenant serving uses:" with Multi-Instance GPU as the keyed answer against data parallelism, beam search, and sequence batching as distractors — real techniques from adjacent domains, attached to the wrong problem). [GROUND TRUTH] (Sources/ncp-genl/domain-8-model-deployment.md) contains this exact self-check item. Expect also a scenario question naming a specific operational requirement (zero-downtime upgrades, automatic crash recovery, guaranteed multi-tenant isolation) and asking which of Docker, Kubernetes, or MIG addresses it, in the style of sections 5 and 6.

What the distractors typically look like

The house style here offers a real mechanism from the wrong layer: proposing Kubernetes as the fix for a hardware-isolation requirement (Kubernetes orchestrates containers, but does not itself provide the hardware-level guarantee MIG does, without MIG or an equivalent resource-partitioning feature underneath it), or proposing MIG as the fix for a rolling-update requirement (MIG partitions hardware statically; it has no concept of gradually replacing a running deployment's containers). Each distractor is a real, correctly-described capability, simply answering a different question than the one asked.

Work through each item before checking the answer key. Every option names a real infrastructure mechanism — the task is matching it to the requirement described, not spotting an invented distractor.

1. A team needs the same model to behave identically whether deployed on a developer's workstation or a production server. What does that requirement point to?
   • A. Kubernetes rolling updates.
   • B. Docker containerization.
   • C. Multi-Instance GPU.
   • D. Dynamic batching.
2. A containerized model crashes intermittently under load, and no one is available to notice and restart it manually. What addresses this?
   • A. Kubernetes health checks, which detect and restart unresponsive containers automatically.
   • B. MIG, which prevents crashes by isolating hardware.
   • C. Docker alone, since the image already specifies the correct dependencies.
   • D. Instance groups, since more instances always prevent crashes.
3. Which requirement is the strongest signal that MIG, specifically, is the right tool?
   • A. Wanting to deploy a new model version without downtime.
   • B. A contractual requirement that no tenant's traffic spike can degrade another tenant's latency.
   • C. Wanting the model to run identically across different machines.
   • D. Wanting more copies of a model to absorb higher demand.
4. What is the key difference between MIG and instance groups sharing an undivided GPU?
   • A. They are the same mechanism under two names.
   • B. MIG provides a hardware-enforced isolation guarantee with fixed slices; instance groups share an undivided GPU flexibly with no such guarantee.
   • C. Instance groups always require Kubernetes; MIG never does.
   • D. MIG only works with stateless models.
5. In this domain's deployment sense, what does "ensemble" refer to?
   • A. Averaging several models' predictions to improve accuracy.
   • B. A chained sequence of stages (e.g., pre-process, model, post-process) served as one pipeline.
   • C. A group of GPUs working together via tensor parallelism.
   • D. A synonym for an instance group.
6. Does containerizing a model improve its inference latency?
   • A. Yes, containers are inherently faster than uncontainerized deployments.
   • B. No — containerization addresses reproducible deployment, not compute or memory performance.
   • C. Yes, but only for encoder-only models.
   • D. Only if the container also includes TensorRT.
7. A single-tenant workload wants flexible, dynamic use of an entire GPU's capacity as its own demand fluctuates. Why would MIG be a poor fit here?
   • A. MIG cannot run any model at all.
   • B. MIG's slices are fixed once configured, which would prevent the single workload from flexibly claiming more or less of the GPU's full capacity.
   • C. MIG requires Kubernetes to function.
   • D. MIG only supports one instance per GPU.
8. Which statement correctly separates Kubernetes from MIG?
   • A. Kubernetes orchestrates containers across a cluster; MIG partitions the physical resources of one GPU.
   • B. Kubernetes and MIG both operate only at the single-GPU level.
   • C. MIG orchestrates containers; Kubernetes partitions GPU hardware.
   • D. They are interchangeable terms for the same orchestration layer.

Answers

1. B. Reproducible behavior across different machines is exactly the problem Docker containerization is built to solve.
2. A. Automatic detection and restart of an unresponsive container is a Kubernetes health-check job, not something Docker or MIG provide on their own.
3. B. A guaranteed, hardware-enforced isolation boundary between tenants is the specific requirement MIG addresses that software-level sharing cannot.
4. B. MIG trades flexibility for a hardware guarantee; instance groups trade a strict boundary for flexible sharing of an undivided GPU.
5. B. The deployment sense of "ensemble" is a served pipeline of chained stages, distinct from the machine-learning sense of combining predictions.
6. B. Containerization is a reproducibility mechanism; it does not itself change a model's compute or memory behavior.
7. B. MIG's fixed slices are the wrong tool when a single workload needs to flexibly claim more or less of a GPU's full capacity as its own demand changes.
8. A. Kubernetes operates at the cluster/container level; MIG operates at the single-physical-GPU hardware level — different layers entirely.

Kubernetes is a software orchestration layer that manages containers across a cluster of machines — scaling the number of running replicas, checking that each one is healthy, and rolling out new versions without downtime. Multi-Instance GPU (MIG) is a hardware feature of the GPU itself, partitioning one physical device into isolated slices with dedicated memory and compute, so multiple tenants or workloads can share that one GPU with a guaranteed performance boundary between them. Kubernetes manages containers across many machines; MIG partitions the physical capacity of one machine's GPU. The two are commonly used together — Kubernetes orchestrating containers that each run inside a dedicated MIG slice — but they solve different problems at different layers of the stack.

No — containerization's purpose is reproducible deployment, not performance. Packaging a model with its exact runtime dependencies ensures the same behavior across different machines, which prevents deployment failures and subtle numerical inconsistencies caused by mismatched library or driver versions, but it does not itself change the model's compute, memory usage, or latency. Performance-oriented decisions — batching mode, instance-group sizing, GPU partitioning, or compiling the model with an optimizer such as TensorRT — are separate choices layered on top of, or independent of, whatever container packages the model.

• Docker packages a model plus its runtime into one reproducible unit — solving the "works on my machine" gap, not a performance problem.
• Kubernetes orchestrates that containerized unit at the fleet level: horizontal/vertical scaling, health checks, and rolling updates, none of which Docker alone provides.
• MIG partitions a single physical GPU into hardware-isolated instances, providing a guaranteed performance boundary that software-level sharing (instance groups) does not.
• MIG and instance groups are not the same lever: MIG trades flexibility for a hardware isolation guarantee; instance groups trade a strict boundary for flexible, opportunistic sharing of an undivided GPU.
• An "ensemble" in this domain means a served pipeline of chained stages, not the machine-learning sense of combining multiple models' predictions.
• These four mechanisms — Docker, Kubernetes, ensembles, MIG — commonly stack rather than compete: a container running in a Kubernetes-managed, MIG-isolated slice, potentially also using instance groups within that slice.
• Model Deployment is 9% of the NCP-GENL blueprint, and this infrastructure layer sits underneath every serving technology the rest of the module covers.

Containerization, Kubernetes, and MIG describe how a model gets packaged, scaled, and hardware-isolated once a serving technology (Dynamo-Triton or NIM) has already been chosen. None of it yet addresses a question specific to the model itself: whether the model's own architecture — encoder, decoder, or encoder-decoder — creates a latency or memory profile that no amount of infrastructure alone can fix. That is the module's closing question, and the subject of M8-05.

Next: M8-05 closes Module 8 with model-type compute tradeoffs — why decoder-only generation latency scales with output length in a way encoder-only models never encounter, and why KV caching, not bigger batches or more instances, is the direct mitigation.