How Spiral avoids a 9× video pipeline slowdown
Nicholas Gates
Published Aug 13, 2026 · 5 min read
Four innocent lines of PyTorch hide a video preprocessing footgun: the decoder and preprocessing each materialize an RGB tensor for the next stage.
In our H100 benchmark, the resulting physical plan made the model's first operator run 8.9× slower than a specialized native-YUV plan.
decoder = VideoDecoder(path, device="cuda")
frames = decoder.get_frames_at(indices).data
frames = preprocess(frames)
activations = model.stem(frames)
Nothing here looks wasteful. The footgun is in the tensor boundaries.
Hardware decoders such as NVDEC produce frames in a native YUV format, commonly
NV12. The
TorchCodec CUDA decoder
then uses a CUDA kernel to convert that output to RGB and returns a materialized
uint8 tensor with shape [N, 3, H, W] in GPU memory.
TorchCodec already pushes resize and crop into FFmpeg on CPU, avoiding a full-resolution frame in Python. The current implementation restricts decoder transforms to CPU devices. On CUDA, the natural library composition is therefore to receive RGB, run ordinary PyTorch preprocessing, return another model-ready tensor, and pass it to the model.
NVDEC → NV12 → RGB uint8 tensor
→ resize + normalization
→ model-ready RGB tensor
→ first convolution
Each library has done exactly what its API promised. Their composition has still fixed the physical plan around an RGB tensor.
What if the tensor boundary were optional?
NVDEC → NV12
→ fused sampling + preprocessing + first convolution
→ first-layer activations
At 4K with eight output channels, the conventional composed plan took 0.499 ms. A specialized native-YUV plan took 0.056 ms: an 8.9× gap in the very first model operator.
Some of that win comes from skipping the RGB tensor. Some comes from giving the first convolution a kernel tailored to this particular shape.
The byte budget
NV12 uses approximately 1.5 bytes per pixel. Decoded RGB8 uses 3 bytes per pixel. After scaling to floating point and applying per-channel mean and standard-deviation normalization, model-ready RGB uses 6 bytes per pixel in FP16 or BF16 and 12 bytes per pixel in FP32.
For one 4K frame, an RGB FP16 tensor occupies 47.5 MiB. Materializing it means writing those bytes during preprocessing and reading them again in the first convolution: a logical 94.9 MiB boundary before considering caches.
Fusing the two operators removes this expanded RGB boundary. The fused kernel can sample Y and UV, perform color conversion and normalization, and accumulate the first convolution directly into the output activations.
This sounds like an obvious win. It isn't always.
So where does 9× come from?
We ran the same 4K, 3→8 first operator three ways on an NVIDIA H100 80GB HBM3 GPU. Each starts after decode with NV12 already in GPU memory.
| Plan | What happens | Time |
|---|---|---|
| Composed | Preprocess → RGB FP16 → cuDNN 1×1 | 0.499 ms |
| Same conv, split | Preprocess → RGB FP16 → Triton 1×1 | 0.107 ms |
| Same conv, fused | Preprocess + Triton 1×1, no RGB tensor | 0.056 ms |
The middle row makes the 9× easy to unpack:
-
Replacing cuDNN with a kernel tailored to this odd three-input-channel shape is 4.7×.
-
Fusing preprocessing into that same kernel, removing the RGB tensor and an extra launch, is another 1.9×.
Together, they make 8.9×.
In other words, writing RGB is not 9× slower. The physical plan hidden behind a clean API boundary can be.
Fusion is not free either. At 64 output channels the matched win falls to 1.2×, because the fastest fused schedule repeats preprocessing across channel tiles. Writing RGB spends bandwidth, but gives the next operator somewhere to reuse the result. Neither plan wins for every shape.
🤓 This is a first-operator microbenchmark, not an end-to-end training result. It uses synthetic NV12, a simple 1×1 convolution, and excludes decode and the rest of the model.
Don't throw the choice away
Spiral can materialize a regular model-ready tensor when that is fastest. It can also keep the native Y and UV planes around, hand them to a custom Torch operator, or fuse sampling into the model itself.
The important part is not choosing too early. Users should not need to hand-write Triton, or spot that four clean lines of Python have quietly committed them to a bad physical plan.
But of course, with Spiral, you don't need to think about any of this. We've got you covered!
Train on everything you record.
Spiral keeps video, sensor streams, agent traces, spatial data, and rows in one system. Analytics, search, and training read them straight from object storage.