Tensor Parallelism vs Pipeline Parallelism: Splitting Within vs Across Layers

Both techniques exist to solve the same problem: a model too large to fit, or run fast enough, on one GPU. Both cut it into pieces and hand the pieces to different GPUs. That surface similarity is exactly why they are confused, and the confusion is costly, because the two techniques split along genuinely orthogonal axes and answer different questions. Tensor parallelism asks "how do I divide the arithmetic inside one layer so several GPUs share the work of computing it together?" Pipeline parallelism asks "how do I divide the stack of layers so different GPUs each own a different stage of the model?" One shrinks a single computation by spreading it sideways across GPUs; the other shrinks the model by spreading its depth across GPUs. Getting this backward is not a cosmetic error — it changes which communication pattern you should expect, which interconnect the setup needs, and which failure mode (constant chatty synchronization versus idle "bubble" time) you should be watching for.

This lesson exists because the exam's own domain material calls this exact pairing out by name as the most common distractor in the parallelism-taxonomy section, and that section sits inside Domain 7, GPU Acceleration and Optimization, which carries 14% of the blueprint — the second-largest domain after Model Optimization at 17%. Together those two domains are 31% of the exam, which is reason enough to make sure the intra-layer/inter-layer distinction is automatic rather than something you reconstruct under time pressure. Everything below builds toward that automaticity: what each technique actually does to the math, what it costs in communication, how the two compare side by side, how they combine with each other and with data parallelism, and the shapes the exam's distractors reliably take.

1. What tensor parallelism splits, mechanically

Identity statement: tensor parallelism (TP) partitions the weight tensors of a single layer across GPUs, so that each GPU holds and computes on only a slice of that layer, and the GPUs must communicate to reassemble a correct result before the next layer can proceed.

Picture one feed-forward matrix multiplication inside a transformer block: an activation vector multiplied by a large weight matrix. Tensor parallelism does not give this matrix to one GPU and the next layer's matrix to another GPU — that would be pipeline parallelism, covered next. Instead it slices this one matrix — say, by column — so GPU 0 holds the left half of the columns and GPU 1 holds the right half. Each GPU multiplies the full input activation by its own column slice and produces a partial output. Neither GPU's partial output is the answer; the correct output is a combination of both partial outputs. That combination step is a collective communication operation — commonly an all-reduce — that every GPU participating in this layer's tensor-parallel group must perform before the result can move on to the next operation in the layer, and often again before it can move on to the next layer.

The load-bearing phrase is intra-layer: the split happens inside one layer's math. Every GPU in a tensor-parallel group is, at every instant, working on the same layer, the same forward pass, the same token — just a different slice of the same matrix. This is why tensor parallelism reduces model-state and activation memory per GPU (each GPU now stores only its slice of the weights and the activations proportional to that slice) without changing which layer is "in flight." From the perspective of the model's depth, nothing changed: there are still the same number of layers, executed in the same order. What changed is that each layer's arithmetic is now a joint computation performed by several GPUs at once.

Why the split has to be undone before the next step

Because the layer's true output depends on combining every GPU's partial slice, tensor parallelism cannot simply let each GPU proceed independently to the next layer with its own partial result — that would silently compute the wrong function. The combination (the all-reduce, or in some schemes an all-gather of shards) is not optional overhead you could skip for speed; it is the step that makes the sliced computation equal to the un-sliced one. That is the seed of the communication-cost story in the next section: tensor parallelism is correct only because it pays for a synchronization point inside the layer, and it pays for that synchronization on every layer, every forward pass, and every backward pass.

2. The all-reduce cost and why tensor parallelism wants NVLink

Tensor parallelism's defining cost is not compute — the total multiply-add work is roughly the same whether one GPU does it or four GPUs split it — its defining cost is communication frequency and latency sensitivity. Because the all-reduce (or equivalent) has to happen inside the forward pass of every tensor-parallel layer, and again inside the backward pass, a model with dozens of transformer blocks triggers dozens of these synchronization points on every single training step, and the same pattern recurs on every inference forward pass too.

That frequency is what makes the interconnect matter so much. A collective operation that fires once per layer, dozens of times per step, needs to complete in microseconds, not milliseconds, or the GPUs spend most of their time waiting on each other instead of computing. This is precisely why tensor-parallel groups are conventionally kept to GPUs connected by the fastest, lowest-latency link available — NVLink within a single server node — rather than spread across nodes connected by ordinary networking. The traffic is frequent, latency-sensitive, and synchronous: every GPU in the group must wait for the collective to finish before any of them can proceed, so a slow link does not just add latency once, it adds latency multiplied by the number of layers, on every step.

Read that as an engineer, not an arithmetician: tensor parallelism's practical GPU-count ceiling on a given machine is set less by memory and more by how many GPUs share a fast enough interconnect. Scale tensor parallelism past the GPUs on one NVLink-connected node and the all-reduce traffic crosses a slower fabric, and the technique's cost profile degrades sharply. That is the operational reason tensor-parallel group sizes are conventionally kept modest (commonly matching the GPU count on a single node) even when a model could in principle be sliced far more ways.

3. What pipeline parallelism splits, mechanically

Identity statement: pipeline parallelism (PP) partitions the model by depth, assigning consecutive layers (a contiguous stage) to each GPU, so that a single input's forward pass visits GPU 0's layers, then GPU 1's layers, then GPU 2's layers, and so on, with activations physically handed off between GPUs at each stage boundary.

Where tensor parallelism keeps every participating GPU working on the same layer at the same moment, pipeline parallelism does the opposite: at any instant, different GPUs in a pipeline-parallel group are working on different layers — and, in an efficiently scheduled pipeline, on different micro-batches too. GPU 0 might own layers 1 through 8, GPU 1 layers 9 through 16, GPU 2 layers 17 through 24, and so on. A token's forward pass starts on GPU 0, and once GPU 0 has finished its eight layers, it sends the resulting activations — not weights, not gradients, just the intermediate output tensor — across to GPU 1, which resumes the computation from layer 9. This is the inter-layer axis: the split is across the model's depth, not inside any one layer's arithmetic.

This has an immediate memory benefit that mirrors tensor parallelism's but for a different reason: each GPU only ever needs to store the weights for its own stage of the model, not the whole stack. A 96-layer model split eight ways by pipeline stage means each GPU holds roughly a twelfth of the parameters. Communication between stages is comparatively light in volume — one activation tensor handed off at each stage boundary, rather than a full collective across every GPU on every layer — which is why pipeline parallelism tolerates a less exotic interconnect than tensor parallelism does; stage-to-stage hand-offs can cross node boundaries, over ordinary high-speed networking, far more comfortably than a per-layer all-reduce can.

The trade you make for that lighter communication

The lighter communication does not come free. Because each GPU owns a different depth-slice of the model, a single input has to physically travel through every stage in sequence before its forward pass is complete, and the backward pass has to travel back through every stage in the reverse order. If you only ever ran one input at a time, most GPUs in the pipeline would sit idle waiting for the input to arrive from the previous stage — an obviously wasteful arrangement. The next section is about the idle time this creates and what shrinks it.

4. The pipeline bubble and how scheduling shrinks it

The idle time inherent to a naive pipeline is called the bubble. It arises because filling the pipeline (getting the first micro-batch through every stage) and draining it (letting the last micro-batch finish every stage after no new work is being fed in) both take time during which some GPUs have nothing useful to do. Picture four pipeline stages and a single micro-batch: GPU 0 works while GPUs 1, 2, and 3 wait for the activation to arrive; then GPU 1 works while GPU 0 could in principle be idle (or, better, already starting the next micro-batch) and GPUs 2 and 3 still wait. The fewer micro-batches in flight relative to the number of stages, the larger the fraction of total wall-clock time that is bubble rather than useful compute.

The standard mitigation is to keep multiple micro-batches moving through the pipeline concurrently — as soon as GPU 0 hands off micro-batch 1 to GPU 1, GPU 0 immediately starts on micro-batch 2, rather than waiting idle. With enough micro-batches in flight, every stage stays busy nearly all the time except at the very start and very end of a pipeline schedule, and the bubble shrinks to a small fraction of the total. More refined schedules go further: interleaved or virtual-pipeline scheduling breaks each GPU's contiguous block of layers into smaller, non-contiguous chunks distributed through the pipeline order, which further shrinks the unavoidable fill/drain bubble at the cost of more frequent (but still comparatively light) activation hand-offs.

The mistake worth naming early, because it is a distractor in its own right, is assuming pipeline parallelism eliminates idle time by definition. It does not — the bubble is inherent to the technique's structure, and it is scheduling choices layered on top (enough micro-batches in flight, interleaving) that shrink it toward zero. A poorly configured pipeline, with too few micro-batches for its stage count, can lose a large fraction of its theoretical throughput to bubble time even though the parallelism itself is "working correctly."

5. Tensor parallelism vs pipeline parallelism: the comparison table

The two techniques answer different questions, and the table below is the fast way to resolve a scenario question the moment you see which axis it is describing.

Property	Tensor Parallelism (TP)	Pipeline Parallelism (PP)
What is split	A single layer's weight tensors (e.g., a matrix's rows or columns)	Consecutive layers / contiguous stages of the model
Axis	Intra-layer — inside one layer's computation	Inter-layer — across the model's depth
What GPUs in the group are doing at any instant	All working on the same layer, the same forward/backward pass	Each working on a different layer, ideally a different micro-batch
Primary communication pattern	Collective all-reduce (or all-gather) to recombine partial results	Point-to-point activation hand-off between adjacent stages
Communication frequency	Very high — inside every layer's forward and backward pass	Low — only at stage boundaries
Latency sensitivity	High — synchronous, blocking, needs fast interconnect	Lower — hand-off tolerates a slower link between stages
Preferred interconnect	NVLink within a node	Can cross nodes over standard high-speed networking
Memory saved, and how	Per-GPU slice of one layer's weights and activations	Per-GPU whole layers for only its own stage
Signature cost	Frequent, latency-sensitive synchronization overhead	The pipeline bubble — idle time filling/draining the pipeline
Typical GPU-count scaling	Kept modest, usually bounded by GPUs sharing one node's fast fabric	Scales with the number of stages you are willing to define, more forgiving across nodes
Best problem fit	A single layer too wide (too many parameters or too much activation memory) to fit or compute fast enough on one GPU	A model too deep (too many layers in total) for its full weight set to fit on one GPU

Read the two rightmost cells of the "communication" rows together and the whole distinction compresses to one sentence: tensor parallelism trades frequent, latency-sensitive chatter for per-layer memory relief; pipeline parallelism trades a coarser, occasional hand-off for per-stage memory relief, at the cost of idle bubble time it must actively schedule away.

6. Where sequence, context, data, and expert parallelism fit around this pair

Tensor and pipeline parallelism are the two axes the exam calls out as the most confusable, but the taxonomy has several neighbors worth placing correctly, because a distractor sometimes borrows a real technique's name from just outside the TP/PP pair.

Data parallelism (DP), and its common PyTorch implementation Distributed Data Parallel (DDP), replicates the entire model on every GPU and splits the batch — different GPUs process different examples through an identical, fully replicated model, then synchronize gradients via all-reduce before the shared optimizer step. This is a different axis entirely from both TP and PP: it does not split any layer or any part of the model's depth at all. It splits the data.

Sequence parallelism (SP) is not an independent axis so much as an extension of tensor parallelism: it splits activations along the sequence dimension in the regions adjacent to a tensor-parallel layer, and it is only meaningful when the tensor-parallel size is already greater than one — SP has nothing to attach to otherwise.

Context parallelism (CP) is broader than SP: rather than being confined to TP-adjacent regions, it splits the input along the sequence dimension across all layers of the model, using ring-style key/value communication in the backward pass. Do not treat SP and CP as synonyms — SP requires TP > 1 and lives in specific regions; CP is layer-wide and does not have that prerequisite.

Expert parallelism (EP) applies specifically to Mixture-of-Experts architectures, distributing different experts across different GPUs; the expert count must be evenly divisible by the EP size. EP is meaningless for a dense transformer that has no expert layers at all — a distractor that offers EP as the fix for a dense model's memory problem is offering a technique that does not apply.

None of DP, SP, CP, or EP collapses the TP/PP distinction — if anything, holding them at arm's length makes the TP/PP boundary sharper, because you can see that "splits the batch" (DP), "splits within a layer" (TP), "splits across layers" (PP), "splits the sequence near TP" (SP), "splits the sequence everywhere" (CP), and "splits MoE experts" (EP) are six genuinely different answers to "what got split," and a scenario question is usually testing whether you can name the right one.

7. Worked example: sharding a model across a stated GPU topology

Take a concrete scenario of the kind the exam favors: a team has a large transformer with 96 layers and unusually wide feed-forward matrices — call it a model whose per-layer weight tensors are large enough that a single layer's forward and backward activations do not comfortably fit in one GPU's memory, even before accounting for the other 95 layers. They have a server with 8 GPUs connected by NVLink, and they can also add a second identical 8-GPU server connected to the first over a standard (much slower) network fabric.

Start from the symptom, not the tool. The complaint here is "one layer is too wide" — too much weight and activation memory inside a single layer's math. That symptom points squarely at tensor parallelism: splitting the wide matrices of each layer across several GPUs shrinks the per-GPU footprint of exactly the thing that is too big. Given the 8-GPU, NVLink-connected node, a sensible choice is a tensor-parallel group of size 4 or 8 within that node — small enough to keep every all-reduce inside the fast NVLink fabric, large enough to shrink each layer's per-GPU slice to a manageable size.

Now add the second server. Its GPUs are not on the same NVLink fabric as the first, so extending the same tensor-parallel group across both servers would push that latency-sensitive, every-layer all-reduce over the slower link — exactly the arrangement the interconnect discussion in section 2 says to avoid. The better move is to use the second server's GPUs for a different axis: split the 96 layers into two pipeline stages, stage one running on server one's tensor-parallel group and stage two running on server two's tensor-parallel group, with only the coarse, infrequent activation hand-off crossing the slower inter-server link at the stage boundary. That hand-off is exactly the kind of point-to-point, comparatively light traffic pipeline parallelism tolerates well across a slower fabric.

The predicted configuration, then, is a combination: tensor parallelism sized to fit inside each NVLink-connected node (solving the too-wide-layer problem with fast, frequent communication kept local), and pipeline parallelism used to span the two nodes (solving the too-many-layers-for-one-node problem with infrequent, tolerant communication crossing the slower link). Predicting that combination from the topology alone — without running anything — is exactly the reasoning the exam's scenario items reward, and it is the reasoning the next section generalizes.

8. Combining tensor, pipeline, and data parallelism

The parallelism families are explicitly designed to be combined — none of them is a complete answer on its own once a model is large enough, and the exam's own framing is that all of these families can be combined to scale from billions to trillions of parameters. The combination most commonly named alongside tensor and pipeline parallelism is data parallelism, and stacking all three at once is often called 3D parallelism: tensor parallelism handles the intra-layer split within a node's fast fabric, pipeline parallelism handles the inter-layer split across nodes or groups of nodes, and data parallelism replicates that entire tensor-plus-pipeline arrangement across additional groups of nodes to process more of the batch at once, synchronizing gradients across the data-parallel replicas via all-reduce before each optimizer step.

Layered on top of any of this, a memory-sharding technique such as ZeRO or FSDP (Fully Sharded Data Parallel) shards optimizer state — and optionally parameters and gradients too — across the data-parallel ranks specifically. This is worth stating precisely because it is a frequent trap: the distributed optimizer is not a fourth parallelism axis sitting alongside tensor, pipeline, and data parallelism. It is a memory-saving technique layered on top of data parallelism, typically doing a reduce-scatter of gradients followed by an all-gather of updated parameter shards. If a scenario's real problem is "optimizer state does not fit in memory," the answer is the distributed optimizer, not a wholesale switch to tensor or pipeline parallelism, both of which solve a different memory problem (a wide layer, or too many layers) rather than the optimizer-state problem specifically.

Put together, a fully scaled training job might look like this: within each 8-GPU node, tensor parallelism of size 8 splits every layer's matrices across the node's NVLink fabric; pipeline parallelism assigns different contiguous blocks of layers to different groups of nodes, with activations crossing the slower inter-node network only at stage boundaries; data parallelism replicates that whole tensor-plus-pipeline unit across still more groups of nodes to chew through more of the batch simultaneously; and a distributed optimizer shards the resulting optimizer state across the data-parallel replicas so it does not have to be fully duplicated everywhere. Four axes, four different questions each one answers, and none of them substitutable for another.

9. Why this is the exam's number-one distractor pair

Domain 7, GPU Acceleration and Optimization, is 14% of the NCP-GENL blueprint — the second-largest domain behind Model Optimization's 17%, and together the two account for 31% of the exam, which the domain's own framing calls out as where to invest study time. Inside Domain 7, the parallelism taxonomy is explicitly the section whose own scope note says to know which parallelism splits what, and its own list of common exam traps names tensor-versus-pipeline parallelism as the single most common distractor in the entire domain — not one distractor among many, but the one flagged first.

That framing tells you the shape of what to expect. Objectives in this domain ask you to configure multi-GPU training and fix bottlenecks, which means the exam is not testing whether you can recite a definition in isolation — it is testing whether, given a scenario describing a symptom (a layer too wide to fit, a model too deep to fit, GPUs on a fast local fabric versus GPUs spread across a slower network, idle time observed in a profiler), you can name the correct axis and predict the correct interconnect and communication consequence. Because tensor and pipeline parallelism are the two techniques most likely to be offered as a false pair — one correct, one plausible-sounding but wrong for the stated symptom — the fastest way to bank points in this section is to have the intra-layer/inter-layer test ready before you read the four answer options, rather than reconstructing it from the scenario's details under time pressure.

How the question tends to be phrased

Expect a scenario naming a symptom and a topology, then four techniques as options, exactly as in section 7's worked example. The phrasings to recognize: "a single layer's weights do not fit on one GPU" points to tensor parallelism; "the model has too many layers to fit its full weight set on one GPU" points to pipeline parallelism; "GPUs are connected by NVLink within a node" is a cue favoring tensor parallelism's placement; "GPUs are spread across multiple nodes on standard networking" is a cue favoring pipeline (or data) parallelism's placement; "optimizer state does not fit" points to a distributed optimizer, not a new parallelism axis; and "the model uses Mixture-of-Experts layers" is the only situation where expert parallelism is even on the table.

10. Common mistakes with tensor and pipeline parallelism

Mistake	Symptom you would actually observe	Fix
Confusing intra-layer (TP) with inter-layer (PP) splitting	You propose tensor parallelism for a "too many layers" problem, or pipeline parallelism for a "one layer too wide" problem, and the configuration does not relieve the actual bottleneck	TP splits within one layer's tensors; PP splits across consecutive layers — match the axis to which dimension is actually too big
Assuming pipeline parallelism eliminates idle time by default	Measured throughput falls well short of theoretical, with GPUs visibly idle in a profiler	The bubble is inherent to pipelining; it must be actively shrunk with enough micro-batches in flight, or interleaved/virtual-pipeline scheduling
Running tensor parallelism across a slow, multi-node link	Training throughput collapses even though the model now "fits"; profiling shows GPUs waiting on collective communication	Keep tensor-parallel groups inside a single NVLink-connected node; use pipeline or data parallelism to span nodes
Treating the distributed optimizer (ZeRO/FSDP) as a fourth parallelism axis	You reach for TP or PP to fix an optimizer-state memory error, and it does not help because the problem was never the layer or depth axis	ZeRO/FSDP shards optimizer state (and optionally params/grads) across data-parallel ranks — it is a memory-sharding technique layered on data parallelism, not a new split
Treating sequence and context parallelism as synonyms	You enable SP with tensor-parallel size of 1 and see no benefit, or apply CP's layer-wide logic where SP's TP-adjacent scope was intended	SP requires TP > 1 and is scoped near TP; CP splits the sequence across all layers and has no TP prerequisite
Offering expert parallelism for a dense model's memory problem	EP configuration fails or is a no-op because there are no experts to distribute	EP applies only to Mixture-of-Experts layers; a dense transformer has nothing for it to shard
Believing gradient accumulation is a parallelism technique that reduces compute	You expect a training-time speedup from accumulation and do not get one	Gradient accumulation only lets a larger effective batch fit under a memory cap by skipping gradient sync between micro-steps; it does not reduce total FLOPs

11. Glossary recap: the terms this lesson introduced

Term	One-line definition
Tensor Parallelism (TP)	Splits a single layer's weight tensors across GPUs; intra-layer
Pipeline Parallelism (PP)	Assigns consecutive layers/stages to different GPUs; inter-layer
All-reduce	Collective operation that combines and synchronizes partial results across GPUs, central to TP and to DDP's gradient sync
Pipeline bubble	Idle GPU time while a pipeline fills or drains, inherent to naive pipelining
Interleaved / virtual-pipeline scheduling	Splits each stage's layers into smaller non-contiguous chunks to shrink the bubble further
Data Parallelism (DP) / DDP	Replicates the whole model, splits the batch, syncs gradients via all-reduce
Sequence Parallelism (SP)	Splits activations along the sequence dimension; only meaningful when TP size > 1
Context Parallelism (CP)	Splits inputs along the sequence dimension across all layers, using ring-style KV communication
Expert Parallelism (EP)	Distributes Mixture-of-Experts layers' experts across GPUs; MoE-only
ZeRO / FSDP	Shards optimizer state (and optionally params/grads) across data-parallel GPUs; a memory technique, not a parallelism axis
NVLink	High-bandwidth, low-latency intra-node GPU interconnect; the preferred fabric for tensor parallelism's frequent all-reduce traffic
3D parallelism	The combination of tensor, pipeline, and data parallelism layered together to scale to very large models
Micro-batch	A small slice of the batch moved through a pipeline stage at a time, used to keep pipeline stages busy and to enable gradient accumulation

12. Key takeaways on tensor parallelism vs pipeline parallelism

• Tensor parallelism splits within a single layer (intra-layer); pipeline parallelism splits across consecutive layers (inter-layer). This is the exam's flagged number-one distractor in the parallelism taxonomy.
• Tensor parallelism's defining cost is frequent, latency-sensitive collective communication (an all-reduce inside every layer's forward and backward pass), which is why it wants a fast intra-node fabric like NVLink.
• Pipeline parallelism's defining cost is the bubble — idle time filling and draining the pipeline — which scheduling (enough micro-batches, interleaving) shrinks but does not eliminate automatically.
• Data parallelism splits the batch, not the model, and is a different axis from both TP and PP; ZeRO/FSDP is a memory-sharding technique layered on data parallelism, not a fourth parallelism axis.
• Sequence parallelism needs tensor-parallel size greater than one; context parallelism is layer-wide and has no such prerequisite — they are not interchangeable, and expert parallelism applies only to Mixture-of-Experts models.
• All of these families are designed to combine: a topology's fast-fabric boundaries (NVLink within a node) are the natural place to put tensor parallelism, while slower inter-node links are the natural place to put pipeline or data parallelism.
• Domain 7 is 14% of the NCP-GENL blueprint, second only to Model Optimization's 17%; together they are 31% of the exam, and the parallelism taxonomy — with this exact TP/PP pairing — is its own scope note's first-named trap.

13. Next: finding a real serving bottleneck instead of guessing at one

Knowing which parallelism axis fits a given topology tells you how to configure multi-GPU training. It does not yet tell you how to find out, on a running system, where the actual bottleneck is — whether a kernel is memory-bound or compute-bound, whether occupancy is low, whether the slow part is even on the GPU at all rather than in data loading. Domain 7's own trap here is exactly that: guessing at a bottleneck wastes time, which is why profiling comes before rebalancing anything.

Next: Reading an Nsight profile to separate a memory-bound kernel from a compute-bound one — the diagnostic habit that turns "training is slow" from a guess about which parallelism knob to turn into an evidence-backed fix.