Feed your GPUs from object storage.
Spiral runs one query plan across client and server, CPU and GPU, from object storage to GPU memory.
Your GPUs can run in a different cloud or region from your data.
A live query becomes the data loader.
Your training loop reads a query, not an export. A recipe names the training sets, their weights, a seed, and a batch size, and TrainLoader streams each rank its batches from the bucket. In production at a video model company, the jobs are bound by compute, not by I/O. Spiral saturates the NIC.
Client and server
The data loader runs beside your GPUs. Compaction, derived writes, and index upkeep run on the server.
CPU and GPU
Video decodes on the GPU through NVDEC. NPP resizes the frames and converts color in the same pass.
Line rate from storage
Bytes go from object storage into pinned host memory, then to the GPU by DMA. The path makes one copy.
Any cloud
Spiral runs on Modal, AWS, Google Cloud, Azure, Nebius, CoreWeave, and other GPU clouds.
At line rate, a staged copy costs more than it saves.
Spiral reads object storage at 94.4 Gbps on a 100 Gbps link, fast enough to keep the GPUs fed straight from the bucket. A staged copy adds a staging job before every run, and it goes stale at the next commit.
A run resumes with the samples it would have read.
Yes: The same batch at every step
A seed and table versions fix each step’s batch, whatever the number of GPUs.
No: A loader for each project
Each new project used to start with its own loader and shuffle job. A recipe now names the mix, and TrainLoader plans the reads, the decode, and the order.
Yes: A new mix in the next epoch
A change to the mix is a code change. Spiral rewrites no data.
No: Shard files and staged copies
Tar and TFRecord shards exist because object storage was too slow to read directly. At line rate the job reads the table, so nothing gets resharded or staged on local disk.
What it runs on.
Workflows:Train