Solutions / Video
Everything but data acquisition.
Frames, clips, samples, embeddings, and every access pattern in between. A guide to training on video data, and every problem you have to think about along the way. Unless you have Spiral.
- Stages
- 12
- between bucket and model checkpoint
- Full decode passes
- 5–7
- before your first training step
- Derived assets
- 6
- each expiring on its own schedule
The problem
A bucket of video is not a dataset.
Between the two sit twelve stages, and your corpus gets fully decoded five to seven times before a single training step runs. Once to filter it. Once to find shot boundaries. Once to caption. Once to embed. Once to encode latents. Then once per epoch, forever.
Every one of those passes re-derives the same thing from scratch: which bytes hold which frame, and what has to be decoded before it. That mapping is the cheapest thing in the system to compute once and the most expensive to keep rediscovering.
And none of it is a pipeline. It is a set of materialized views over one source of truth, each with a different expiry date. Captions die when the captioning model improves. Latents die when the VAE changes. Shards die when you pick a new resolution. The video itself never changes at all. Teams who model this as a pipeline rebuild everything whenever anything moves, because rebuilding everything is the only operation they can trust.
We run all twelve, and we rebuild only what actually expired.
The work
The twelve stages.
Each one produces something you asked for, and hides something you should never have to think about. The note under each stage is the part we would rather you never learn.
Ingest
Turning a bucket into something you can reason about.
01
Inventory and provenance
A catalog of what you actually have, and where every second of it came from.
You don't need to think about
Why exact-hash deduplication catches almost nothing on scraped video, or why the licensing question you get asked in a year is answerable only if provenance survived all eleven stages below.
02
Validate and normalize
Files that decode, with timestamps you can trust.
You don't need to think about
Containers that lie about frame counts, the duration-times-frame-rate arithmetic that silently misaligns on part of every corpus, or the one unapplied display matrix that trains half your data sideways without a single error in the logs.
03
Profile encode shape
The cost model for all nine stages after it.
You don't need to think about
GOP length, B-frame pyramids, or variable frame rate. In a 200-clip sample of real web video we profiled, the median GOP ran 128 frames and 57.8% of frames were non-reference, skippable outright. Nobody who has not measured this knows it about their own corpus.
Curate
Deciding what deserves to be trained on.
04
Quality and safety
A corpus that will not teach your model habits you spend months unlearning.
You don't need to think about
Bitrate per pixel, and whether it predicts your model learning to reproduce compression artifacts. A clip can pass every resolution and aesthetic filter you own and still degrade what you generate.
05
Motion and aesthetics
A motion distribution you chose, rather than one you inherited.
You don't need to think about
Near-static clips, which produce the single most recognizable failure mode in video generation, or the slideshows and tripod interviews that sail through conventional quality filters untouched.
06
Shots and chunking
Clips that do not contain cuts.
You don't need to think about
Why a chunk boundary stored as a timestamp costs you a container parse at every stage afterward, and why the same boundary stored as a frame index costs nothing, forever.
Derive
The expensive assets, and the ones with the shortest shelf life.
07
Captions and annotation
Text your model can condition on.
You don't need to think about
Which of your derived assets die the day you swap the captioning model. We keep that list, and rebuild only what expired.
08
Embeddings, dedup and retrieval
Search, clustering, dataset balancing, and an eval set that is not leaked.
You don't need to think about
Train and eval contamination that is semantic rather than hash-based, or embedding generation as a sparse-sampling read, exactly the access pattern where ordinary decoding wastes the most work.
09
Latents
Precomputed latents, ready to train against.
You don't need to think about
The latent cache that quietly freezes your resolution, frame rate, aspect handling and clip length until somebody rebuilds the whole thing.
Train
Keeping expensive accelerators busy, and knowing what they saw.
10
Bucketing and sharding
Uniform batches, without a manifest that has to fit in memory.
You don't need to think about
Why aspect-homogeneous shards make a local shuffle buffer hand you correlated batches, and why baking one sampling policy into storage is the decision that costs the most later.
11
Loading
Tensors, fast enough to keep your accelerators busy.
You don't need to think about
The five to seven hardware decode engines a datacenter GPU actually exposes, or what happens when you reach that ceiling and decoding fewer frames is the only lever left.
12
Lineage, eval and cost
An answer to “what data was in run 47?”
You don't need to think about
Any of it, until a regression, a takedown request, or an auditor asks. Then it is the only thing that matters.
Expiry
What forces you to do it all again.
The question that decides your compute bill is not how fast you build these assets. It is how often you have to rebuild them, and how much you rebuild by accident because nobody could say what went stale.
| Derived asset | New footage | Filter change | New captioner | New embedder | New VAE | New res / fps |
|---|---|---|---|---|---|---|
| Encode profile | Rebuild required | No rebuild | No rebuild | No rebuild | No rebuild | No rebuild |
| Quality + motion scores | Rebuild required | Rebuild required | No rebuild | No rebuild | No rebuild | No rebuild |
| Shot boundaries | Rebuild required | No rebuild | No rebuild | No rebuild | No rebuild | No rebuild |
| Captions | Rebuild required | No rebuild | Rebuild required | No rebuild | No rebuild | No rebuild |
| Embeddings + index | Rebuild required | No rebuild | No rebuild | Rebuild required | No rebuild | No rebuild |
| Latents | Rebuild required | No rebuild | No rebuild | No rebuild | Rebuild required | Rebuild required |
| Training shards | Rebuild required | Rebuild required | Rebuild required | No rebuild | Rebuild required | Rebuild required |
Training shards are the bottom row for a reason: materializing a sampling policy into storage couples every upstream decision to your physical layout. Keep the sampling policy a query instead, and most of this table stops firing.
What you get
Your videos, training-ready.
Point Spiral at the bucket you already have. Nothing moves, nothing gets re-encoded to start, and the files stay yours in the format they arrived in. What you get back is every access pattern above, from frames and clips to sampled windows, embeddings and latents, served fast enough to keep your accelerators saturated, with a straight answer about which data went into which run.
The engineers you hired to build models get to build models. And not mess with video data.