How Spiral avoids a 9× video pipeline slowdown

Aug 13, 2026 · by Nicholas Gates · 5 min read

Four innocent lines of PyTorch hide a video preprocessing footgun: the decoder and preprocessing each materialize an RGB tensor for the next stage.

In our H100 benchmark, the resulting physical plan made the model's first operator run 8.9× slower than a specialized native-YUV plan.

Python

decoder = VideoDecoder(path, device="cuda")
frames = decoder.get_frames_at(indices).data
frames = preprocess(frames)
activations = model.stem(frames)

Nothing here looks wasteful. The footgun is in the tensor boundaries.

Hardware decoders such as NVDEC produce frames in a native YUV format, commonly NV12. The TorchCodec CUDA decoder then uses a CUDA kernel to convert that output to RGB and returns a materialized uint8 tensor with shape [N, 3, H, W] in GPU memory.

TorchCodec already pushes resize and crop into FFmpeg on CPU, avoiding a full-resolution frame in Python. The current implementation restricts decoder transforms to CPU devices. On CUDA, the natural library composition is therefore to receive RGB, run ordinary PyTorch preprocessing, return another model-ready tensor, and pass it to the model.

NVDEC → NV12 → RGB uint8 tensor
      → resize + normalization
      → model-ready RGB tensor
      → first convolution

Each library has done exactly what its API promised. Their composition has still fixed the physical plan around an RGB tensor.

What if the tensor boundary were optional?

NVDEC → NV12
      → fused sampling + preprocessing + first convolution
      → first-layer activations

At 4K with eight output channels, the conventional composed plan took 0.499 ms. A specialized native-YUV plan took 0.056 ms: an 8.9× gap in the very first model operator.

Some of that win comes from skipping the RGB tensor. Some comes from giving the first convolution a kernel tailored to this particular shape.

The byte budget

NV12 uses approximately 1.5 bytes per pixel. Decoded RGB8 uses 3 bytes per pixel. After scaling to floating point and applying per-channel mean and standard-deviation normalization, model-ready RGB uses 6 bytes per pixel in FP16 or BF16 and 12 bytes per pixel in FP32.

For one 4K frame, an RGB FP16 tensor occupies 47.5 MiB. Materializing it means writing those bytes during preprocessing and reading them again in the first convolution: a logical 94.9 MiB boundary before considering caches.

Fusing the two operators removes this expanded RGB boundary. The fused kernel can sample Y and UV, perform color conversion and normalization, and accumulate the first convolution directly into the output activations.

This sounds like an obvious win. It isn't always.

So where does 9× come from?

We ran the same 4K, 3→8 first operator three ways on an NVIDIA H100 80GB HBM3 GPU. Each starts after decode with NV12 already in GPU memory.

Plan	What happens	Time
Composed	Preprocess → RGB FP16 → cuDNN 1×1	0.499 ms
Same conv, split	Preprocess → RGB FP16 → Triton 1×1	0.107 ms
Same conv, fused	Preprocess + Triton 1×1, no RGB tensor	0.056 ms

The middle row makes the 9× easy to unpack:

Replacing cuDNN with a kernel tailored to this odd three-input-channel shape is 4.7×.
Fusing preprocessing into that same kernel, removing the RGB tensor and an extra launch, is another 1.9×.

Together, they make 8.9×.

In other words, writing RGB is not 9× slower. The physical plan hidden behind a clean API boundary can be.

Fusion is not free either. At 64 output channels the matched win falls to 1.2×, because the fastest fused schedule repeats preprocessing across channel tiles. Writing RGB spends bandwidth, but gives the next operator somewhere to reuse the result. Neither plan wins for every shape.

🤓 This is a first-operator microbenchmark, not an end-to-end training result. It uses synthetic NV12, a simple 1×1 convolution, and excludes decode and the rest of the model.

Don't throw the choice away

Spiral can materialize a regular model-ready tensor when that is fastest. It can also keep the native Y and UV planes around, hand them to a custom Torch operator, or fuse sampling into the model itself.

The important part is not choosing too early. Users should not need to hand-write Triton, or spot that four clean lines of Python have quietly committed them to a bad physical plan.

But of course, with Spiral, you don't need to think about any of this. We've got you covered!