ONNX vs TensorRT vs TensorRT-LLM: Compiling a Model for Inference

Take them in the order a model actually travels.

ONNX — the exchange format. A trained model in PyTorch is a Python object graph plus a tensor file, meaningful only to PyTorch. ONNX defines an open, framework-neutral representation of the same thing: a computation graph of standardized operators, with the weights attached. Export a model to ONNX and any ONNX-compatible runtime — ONNX Runtime, TensorRT, a mobile inference engine, a hardware vendor's toolchain — can consume it. ONNX is a format, not an accelerator. Exporting to ONNX by itself makes nothing faster; it makes the model portable, which is what then lets an optimizing compiler get hold of it. The course source material names ONNX as the framework-neutral exchange format and flags opset and version pitfalls as the thing that actually goes wrong in practice.

TensorRT — the general-purpose inference compiler and runtime. TensorRT takes a model graph (from ONNX, or built through its own APIs) and compiles it into an engine: a serialized, hardware-specific artifact optimized for one GPU architecture and, historically, one set of input shapes. Its named optimizations are:

TensorRT is general purpose: it optimizes CNNs, transformers used as encoders, recommender models, speech models, anything expressible as a graph. What it does not have is any notion of autoregressive generation.

TensorRT-LLM — the LLM-specific layer. Autoregressive generation is not a graph-optimization problem; it is a stateful, iterative, memory-management problem. A generic compiler has nothing to say about a cache that grows one token at a time, about scheduling sequences that finish at unpredictable times, or about verifying speculatively drafted tokens. TensorRT-LLM is the product that adds exactly those capabilities on top of TensorRT's compilation machinery. The course source material's feature list, which is the list to memorize:

• KV cache management
• Paged attention
• In-flight / continuous batching
• Speculative decoding

Plus the LLM-relevant structural work: optimized attention and transformer kernels, and support for tensor and pipeline parallelism so a model too large for one GPU can be compiled across several.

The relationship in one line: ONNX is how a model travels, TensorRT is how a model is compiled, TensorRT-LLM is how an LLM is compiled and served. And the sentence that answers the exam question: TensorRT-LLM is built on TensorRT and is a different product from it, adding LLM-specific runtime features that TensorRT does not have.

L1 — Intuition: a translated recipe, a rehearsed kitchen, and a kitchen that knows it is making one dish repeatedly

ONNX is translating the recipe into a language every kitchen reads. Nothing cooks faster; the recipe is simply no longer trapped in one kitchen's private notation.

TensorRT is a chef who reads the recipe, notices that three steps can be done in one pan, times four different knives to find the fastest for this specific vegetable on this specific counter, and writes down a single optimized procedure for this kitchen. Move to a different kitchen and the procedure has to be rewritten — which is exactly why a TensorRT engine is hardware-specific.

TensorRT-LLM is the same chef, plus the knowledge that this kitchen serves one dish, over and over, one bite at a time, to a queue of customers who arrive and leave unpredictably. That knowledge produces a completely different set of decisions: how to keep the partially-eaten plates (the KV cache), how to seat arriving customers without making everyone else wait (in-flight batching), how to prepare several likely next bites in advance (speculative decoding).

L2 — Mechanism: the export-and-compile pipeline, and where each stage fails

The canonical path for a non-LLM model:

PyTorch / TensorFlow model
        │  torch.onnx.export(...) — trace the graph, map ops to ONNX operators
        ▼
model.onnx  (graph + weights, opset version N)
        │  TensorRT builder: parse → optimize → autotune → calibrate
        ▼
engine.plan (serialized, architecture-specific, precision-specific)
        │  TensorRT runtime, or Triton's TensorRT backend
        ▼
inference

Each arrow has a characteristic failure:

Export → ONNX. The opset version is the version of the ONNX operator specification the file targets. Operators are added and revised across opsets, so an exported file whose opset is newer than the consumer supports will not load, and one whose opset is older may lack an operator the model needs. Two further export hazards matter: unsupported operators, where a custom or exotic layer has no ONNX equivalent and export fails or silently substitutes something; and tracing versus scripting, where export by tracing records the operations executed on one example input and therefore bakes in data-dependent control flow — an if on tensor values, a loop whose length depends on the input — producing a graph that is correct for the traced shape and wrong for others. Dynamic axes must be declared explicitly at export time if batch size or sequence length will vary; otherwise the graph is fixed to the shapes it was traced with.

ONNX → engine. The TensorRT build step is where the real optimization happens, and it has three properties that surprise people:

1. It is slow. Autotuning means actually benchmarking candidate kernels, so building an engine takes minutes and sometimes much longer. It is a build step, not a load step, and belongs in CI rather than in a container's startup path.
2. The engine is hardware-specific. Because kernels were selected by benchmarking on a particular GPU architecture, an engine built for one architecture is not portable to another. You build per target. This is the single most operationally consequential fact about TensorRT.
3. It is version-sensitive. Engines are generally tied to the TensorRT version that built them. Treat cross-version engine reuse as unsupported unless the documentation for your versions says otherwise.

Engine → serving. The engine is loaded by the TensorRT runtime, or served through Triton Inference Server's TensorRT backend, which is the usual production arrangement — Triton handles the HTTP/gRPC endpoint, model repository, versioning, and dynamic batching while TensorRT provides the optimized execution (12-13).

For an LLM the path differs at the middle stage. Rather than exporting to ONNX and parsing that, the typical TensorRT-LLM flow converts a model checkpoint into TensorRT-LLM's own definition of the network, applies quantization if requested, and builds an engine with the LLM runtime features enabled — batching mode, paged KV cache, parallelism configuration, maximum batch size and sequence length. The specific commands and file layouts are version-dependent and not worth memorizing for this exam; the shape of the flow is.

L3 — Why fusion and autotuning work, and what compilation does not fix

Why layer fusion is the biggest single win. Consider a convolution followed by a bias add followed by a ReLU. Executed as three kernels, the intermediate tensor is written to HBM after the convolution, read back for the bias, written again, read again for the ReLU, written a third time. Fused into one kernel, the intermediate values stay in registers or shared memory and HBM is touched twice — once in, once out — instead of six times. Since these operations are memory-bound, eliminating that traffic is close to a proportional speedup. It is the same principle that makes decode memory-bound in 12-05: on modern GPUs, moving bytes is the expensive part, and fusion moves fewer bytes.

Why autotuning needs the real hardware. The fastest kernel for a given operation depends on tensor shapes, data types, cache sizes, the number of streaming multiprocessors, and the memory hierarchy — all of which vary by GPU architecture. There is no analytical model reliable enough to pick a winner, so TensorRT measures. Measuring requires the target device, which is why "build the engine on the machine you will serve on, or at least on the same architecture" is the operational rule.

What precision calibration adds. TensorRT can build an engine at FP16, at INT8 with a calibration pass over representative data, or at FP8 where the hardware supports it. This is the same procedure as 12-02's post-training quantization, exposed inside the compiler: you supply a calibration dataset, TensorRT observes activation ranges, and the engine bakes in the scale factors. And it carries the same obligation — an engine built at a lower precision is a behaviour change and requires the eval set to be re-run. This lesson is flagged evalRerun for precisely that reason. Compilation is usually described as a pure performance step, and the moment precision reduction enters the build, that is no longer true.

What compilation does not fix. Compilation cannot make a model more accurate, cannot reduce the number of parameters, cannot shorten a prompt, and cannot lower the memory-bandwidth floor of reading weights during decode beyond what precision reduction achieves. It also cannot fix a bad retrieval pipeline or a bad prompt — a slow RAG system whose latency is dominated by a reranker will not be rescued by an optimized LLM engine. Compilation is one term in the latency budget, and 12-10 insists on measuring which term dominates before optimizing any of them.

The portability-versus-performance trade, stated plainly. ONNX buys you portability at the cost of leaving performance on the table; TensorRT buys you performance at the cost of portability. ONNX Runtime sits in the middle — it will execute an ONNX model on many backends, including through a TensorRT execution provider. There is no configuration that gives you both maximum portability and maximum performance, and knowing which you are buying is the actual engineering decision.

This is the highest-value asset in the lesson. Learn it in both directions.

The four one-liners to be able to produce under time pressure:

• ONNX = the format that makes a model portable.
• TensorRT = the general-purpose inference compiler: fusion, autotuning, precision.
• TensorRT-LLM = the LLM-specific layer on top: KV cache, paged attention, in-flight batching, speculative decoding.
• They are different products. TensorRT-LLM is not "TensorRT with a longer name."

And the adjacent products this trio is confused with, kept straight:

A constructed scenario, illustrative and not measured. Two models must go to production: a BERT-based text classifier used for routing support tickets, and a 7B decoder-only LLM used for drafting replies.

Model A — the classifier. The right path is ONNX → TensorRT.

Step 1  Export to ONNX
        - declare dynamic axes for batch and sequence length
        - pin opset to a version the TensorRT you deploy supports
        - validate: run the ONNX file and the PyTorch model on the same 200 inputs,
          compare logits, assert max absolute difference below a tolerance
Step 2  Build a TensorRT engine
        - FP16 first; measure
        - INT8 with a calibration set of ~1,000 in-domain tickets; measure again
        - build on the same GPU architecture as production
Step 3  Validate the engine
        - re-run the frozen eval set on the engine, not on the PyTorch model
        - report per-class precision and recall, not just aggregate accuracy
Step 4  Serve
        - Triton with the TensorRT backend, dynamic batching enabled with a queue
          window sized from the latency budget

Why this path is right: a classifier is a single forward pass per request with roughly uniform service time. Graph optimization is the whole game, there is no autoregressive state, and Triton's dynamic batching is a natural fit (12-06). TensorRT-LLM would be the wrong tool — nothing in its feature list applies.

The two validation steps deserve emphasis because both are routinely skipped. Step 1's numeric comparison catches export bugs — a traced conditional, a substituted operator, a silently wrong dynamic axis — while they are still cheap to fix. Step 3's eval re-run catches precision regressions from the INT8 build. Skipping step 1 means you debug a wrong model at serving time; skipping step 3 means your users find the regression.

Model B — the 7B LLM. The right path is TensorRT-LLM, not ONNX → TensorRT.

Step 1  Convert the checkpoint into TensorRT-LLM's network definition
Step 2  Choose the build configuration, which encodes your capacity plan:
        - weight precision (BF16 / INT8 / FP8) → from the memory arithmetic in 12-01
        - maximum batch size and maximum sequence length → from the KV-cache
          arithmetic in 12-05, because these two multiply into the cache budget
        - paged KV cache enabled
        - in-flight batching enabled
        - tensor/pipeline parallelism if the model spans GPUs
Step 3  Build the engine on the target architecture
Step 4  Validate: re-run the frozen eval set against the engine, with a
        long-context slice and a structured-output slice
Step 5  Serve via Triton's TensorRT-LLM backend, or deploy a NIM microservice
        that ships a prebuilt engine

Notice what step 2 really is: the build configuration is the capacity plan, written in a different notation. Maximum batch size and maximum sequence length are compiled into the engine and together determine the KV-cache memory the engine will demand. Choosing them without having done the 12-05 arithmetic produces an engine that either does not fit or wastes capacity. This is the point where the module's earlier arithmetic stops being academic.

The arithmetic that decides step 2, worked. Using 12-05's model — 32 layers, hidden size 4096, BF16 — at 512 KB of cache per token, on a device with 24 GB usable:

weights at BF16                          14.0 GB
runtime + engine workspace (assumed)      1.5 GB
                                        --------
available for KV cache                    8.5 GB
cache budget in tokens = 8.5e9 / 524,288 ≈ 16,212 tokens

so: max_batch_size × max_seq_len ≲ 16,212
    → 8 concurrent × 2,000 tokens = 16,000  ✓ fits
    → 16 concurrent × 2,000 tokens = 32,000 ✗ does not fit
    → 32 concurrent ×   500 tokens = 16,000 ✓ fits

Three viable configurations, three different products. Now add the INT8 weight option:

weights at INT8                           7.0 GB
runtime + workspace                       1.5 GB
available for KV cache                   15.5 GB
cache budget                        ≈ 29,563 tokens
    → 14 concurrent × 2,000 tokens = 28,000 ✓ fits

Quantizing the weights nearly doubled the concurrency the engine can be built for — and it also obliges the step-4 eval re-run, because it changed the model's numerics. The compile step is where precision, capacity, and quality all get decided at once, which is why treating engine building as a pure ops task is a mistake.

The failure this example is designed to prevent. A team exports the 7B LLM to ONNX, builds a plain TensorRT engine, and finds that serving is slow and memory usage is strange. The diagnosis: the engine has no KV cache management, so either generation is recomputing history each step or the state handling has been hand-rolled around a graph that knows nothing about it; there is no in-flight batching, so throughput collapses under variable output lengths; there is no paged cache, so memory fragments. Nothing is broken — the wrong tool was used. The general-purpose compiler has no LLM runtime, and that absence is the whole content of the confusable.

Three separate reasons converge on this lesson, which is why it carries more exam weight than its 25 designed minutes suggest.

First, the objectives. Objective 4.1 covers assisting with deployment and evaluation of model scalability, performance, and reliability, and 4.4 covers identifying the system, hardware, and software components required to meet user needs. Choosing a compilation and serving toolchain is exactly a 4.4 activity.

Second, the suggested readings. The official study guide's reading list names ONNX explicitly, and names TensorRT together with INT8 quantization-aware training. Items on that list are the guide's own statement of what a candidate should know.

Third, and most importantly, the confusable. The course index identifies TensorRT vs TensorRT-LLM as one of the top reported confusables and states that they are different products, with TensorRT-LLM's distinguishing features being KV cache, paged attention, in-flight/continuous batching, and speculative decoding. It also notes that where two options are technically defensible, the NVIDIA-branded answer tends to be keyed. The combination means: know the feature list, and prefer the specific NVIDIA product that matches the described need.

Question phrasings:

• "What is ONNX used for?" — a framework-neutral format for exchanging trained models between frameworks and runtimes.
• "Which of the following is an inference optimizer that performs layer fusion and precision calibration?" — TensorRT.
• "Which product adds KV cache management and in-flight batching for large language models?" — TensorRT-LLM.
• "A team has a TensorRT engine built for one GPU architecture and is migrating to another. What must they do?" — rebuild the engine for the new architecture.
• "Which of these does TensorRT-LLM provide that TensorRT does not?" — paged attention / in-flight batching / speculative decoding / KV cache management.
• "What is the main risk when exporting a model to ONNX?" — opset/version incompatibility and unsupported operators; also traced control flow and undeclared dynamic axes.
• "After compiling a model to INT8 with TensorRT, what must the team do before release?" — re-run the evaluation set, because the numerics changed.
• "Which component optimizes a model and which serves it?" — TensorRT/TensorRT-LLM optimize; Triton serves; NIM packages.

Distractor families:

They are different products, and this is one of the two most-reported confusables on the NCA-GENL exam. TensorRT is a general-purpose inference compiler and runtime: it takes a model graph and produces a GPU-specific optimized engine using layer and kernel fusion, kernel autotuning against the actual target hardware, precision calibration for FP16/INT8/FP8, and graph-level simplification. It works on any model expressible as a graph — vision, speech, recommenders, transformer encoders — and it has no concept of autoregressive text generation. TensorRT-LLM is built on top of TensorRT and adds exactly what generation requires: KV cache management, paged attention, in-flight (continuous) batching, and speculative decoding, along with optimized transformer kernels and tensor/pipeline parallelism for models that span GPUs. The practical consequence: use TensorRT for single-pass models, and TensorRT-LLM for generative LLMs. Using plain TensorRT for an LLM leaves you without any of the runtime machinery that makes LLM serving efficient.

ONNX — Open Neural Network Exchange — is an open, vendor-neutral specification for representing a trained neural network as a graph of standardized operators together with its weights. Its purpose is interoperability: a model trained in one framework can be exported to a .onnx file and then loaded by any compatible runtime, including ONNX Runtime, TensorRT, mobile inference engines, and various hardware vendors' toolchains. You use it when the training framework and the serving runtime are different, when you want to decouple your deployment from a specific framework's release cycle, or when you need one artifact to target several runtimes. What ONNX does not do is make inference faster on its own — it is a container, and any speedup comes from the compiler or runtime that consumes it. The recurring practical hazards are opset version mismatches, operators with no ONNX equivalent, control flow baked in by tracing, and dynamic axes that were never declared.

TensorRT's compilation machinery underpins LLM optimization, but plain TensorRT is not the right tool for serving a generative LLM, because it lacks every runtime feature autoregressive generation depends on. There is no KV cache management, so the token-by-token state that makes generation linear rather than quadratic has no home; no paged attention, so cache memory fragments; no in-flight batching, so throughput collapses when output lengths vary; and no speculative decoding. TensorRT-LLM is the product that supplies those. The reason the distinction is not merely pedantic is that a team can technically build an LLM graph into a TensorRT engine and then discover that serving it is slow and memory-hungry for reasons nothing in the compiler can address — the missing pieces are runtime architecture, not graph optimization. For an encoder-only transformer used as a classifier or embedder, by contrast, plain TensorRT is exactly right.

Because a substantial part of TensorRT's optimization is empirical. Kernel autotuning works by actually benchmarking multiple candidate implementations of each operation on the target device and selecting the fastest, and the winner depends on the GPU's architecture — its streaming-multiprocessor count, cache sizes, memory hierarchy, and tensor-core capabilities — as well as on tensor shapes and data types. The engine therefore encodes decisions valid for the hardware it was built on. It is also generally tied to the TensorRT version that produced it. The operational consequences are three: build engines on the same architecture you will serve on, build one engine per architecture if your fleet is heterogeneous, and treat engine building as a CI artifact step rather than something that happens at container startup, since builds take minutes and a build inside a cold start is a production incident waiting to happen.

They sit at three different layers and each solves a different problem. TensorRT and TensorRT-LLM optimize: they compile a model into a fast engine for NVIDIA GPUs. Triton Inference Server serves: it exposes inference endpoints, hosts many models across multiple framework backends including TensorRT and TensorRT-LLM, and provides dynamic batching, concurrent model execution, a model repository with versioning, and ensembles. NVIDIA NIM packages: it is a pre-optimized inference microservice with stable, secure APIs, shipping prebuilt optimized engines including TensorRT-LLM, and it is part of NVIDIA AI Enterprise. So a typical production arrangement is a TensorRT-LLM engine served through Triton, or a NIM microservice that has already made both of those choices for you. Being able to place a described need at the right layer — optimize, serve, or package — is precisely what the stack-map questions test, and 12-13 builds the full map.

If the compilation changed the numerics, yes, unconditionally. A build at reduced precision — FP16, INT8, FP8 — changes the computed values and therefore the model's outputs, which makes it a quality intervention rather than an infrastructure change. Even a same-precision build can shift results slightly, because layer fusion and different kernel selections alter the order of floating-point reductions. The right practice is a two-stage validation: first, a numeric comparison between the source model and the compiled engine on a few hundred inputs, asserting a tolerance, which catches export and build defects; and second, a full run of the frozen eval set against the engine, reported per slice rather than as an aggregate, because precision-related damage concentrates in specific capabilities such as structured output and long-context reasoning. This lesson is flagged as an eval re-run in the course design for exactly this reason.

• ONNX = format. Framework-neutral portability. It makes nothing faster by itself.
• TensorRT = general-purpose inference compiler. Layer/kernel fusion, kernel autotuning, precision calibration, graph simplification.
• TensorRT-LLM = LLM-specific layer on TensorRT. KV cache, paged attention, in-flight/continuous batching, speculative decoding. These are different products — the top reported confusable.
• TensorRT has no notion of autoregressive generation. That absence is the entire content of the distinction.
• A TensorRT engine is architecture- and version-specific. Build per target, in CI, on matching hardware.
• ONNX export fails in four characteristic ways: opset mismatch, unsupported operators, traced control flow, undeclared dynamic axes. Validate numerically.
• For an LLM, the build configuration is the capacity plan: weight precision, maximum batch size, and maximum sequence length come from the 12-05 cache arithmetic.
• Fusion is the biggest single win because it removes HBM traffic, and moving bytes is the expensive part on modern GPUs.
• Any precision-changing build is a behaviour change. Re-run the frozen eval set, per slice.
• Layers, not synonyms: TensorRT/TensorRT-LLM optimize · Triton serves · NIM packages · NeMo builds.

Next: 12-09 converts everything the module has computed so far into the currency your manager actually asks about. Weights, KV cache, batch size, precision, and tokens per second all resolve into a single number — cost per million tokens — and that number decides architecture more often than any quality metric does.