Skip to content
Workflows / Train

Train on any mix of your data.

Give each training set a weight, and set a seed. Spiral streams the batches from storage, in the same order on one GPU or many, with no export step.

Exact table versions, a fixed order, and device batches

Run recordrun_2048 · ready
samplestablesfailed_graspsnominal_graspsplanseed 20484 ranks disjoint
2.4Msamples
04ranks
00exports
The current way

Each new mix starts with an export.

A mix is usually a set of weights over named datasets or exported shards. To train on 30% failures and 70% nominal episodes, a team first exports both subsets. The team then manages how stale they get, how they shuffle, and how a stopped run resumes.

“Training SmolVLA spends more time waiting for the dataloader (1s) than running backprop (0.7s).”
LeRobot issue #1623
“The changes in dataset versions seem to be creating fragmentation in the hub.”
LeRobot issue #2446
The input plan

Declare the mix, and Spiral plans every batch.

You name each table, its weight, and a seed. Nothing is exported.

Python · training input

Docs
robots = client.project("robotics").branch("main")
fails = robots.table("failed_grasps")
nominal = robots.table("nominal_grasps")
windows = Windows({"wrist_camera": Video(range(90))})

recipe = Recipe(
    seed=2048,
    batch_size=64,
    components=[
        Component(fails, windows, weight=0.3),
        Component(nominal, windows, weight=0.7),
    ],
)

for batch in TrainLoader(recipe, device="cuda"):
    loss = policy.step(batch)

2 tables · weights 0.3 and 0.7 · seed 2048

01Pin each table to a version.
failed_grasps @ main (01a1a71d) → 9,412 momentsnominal_grasps @ 01a1a71d → 81,006 moments
02Plan weighted draws.Each table keeps its share in every batch, not only on average.
draw(failure) = 0.30draw(nominal) = 0.70seed = 2048
03Divide the work among ranks.Each rank reads its share of every batch, so the batch is the same at any rank count.
rank 0 [0000, 0004, 0008, …]rank 1 [0001, 0005, 0009, …]rank 2 [0002, 0006, 0010, …]rank 3 [0003, 0007, 0011, …]
04Deliver device batches.On a 100 Gbps link, Spiral reads object storage into GPU memory at 94.4 Gbps.

Spiral blog · Sep 30, 2026

object ranges → NVDEC → fused transform→ DLPack → torch.Tensor[batch, time, …]
Delivery guarantees

Spiral reproduces every batch.

The run record names every table version, the seed, and the weights, so every batch is fixed before the first step. None of these three events changes what the model sees.

01A commit lands mid-runThe run still reads failed_grasps at 01a1a71d. The new rows wait for the next run record.
02A node dies mid-epochThe run restarts from the position saved with the checkpoint, and the next step gets the batch it would have had.
03You restart on 8 GPUs, not 4The run continues at its exact step, with the batches it would have read on 4.
The on-ramp

Keep the training loop you have.

Start with ScanDataset in a standard PyTorch DataLoader. Move to TrainLoader for weighted mixes, a fixed order, and exact resume.

Python · existing trainer

Docs
robots = client.project("robotics").branch("main")
fails = robots.table("failed_grasps")
scan = fails.scan_request(["wrist_camera", "action"])

dataset = ScanDataset(scan, batch_size=64)
loader = DataLoader(dataset, batch_size=None)

WebDataset and MosaicML Streaming inputs convert to tables

Existing stack

Spiral feeds PyTorch and imports the dataset formats you have.

PyTorchWebDatasetWebDatasetMosaicML StreamingLeRobotHugging Face Datasets
faqs

Before you connect the trainer.

A change to the seed, the weights, or a table version starts one. A restart or a new rank count does not.

Yes. A table can hold one checkpoint in each row: the weights as a file, the step, the run record, and the saved position. A restart reads one row and continues at its exact step.

You choose. Tables live in your own bucket or in Spiral-managed storage. Training reads them in place, with no copy into a separate training store.

The plan can decode H.264 with NVDEC, convert NV12 frames to tensors in fused CUDA kernels, and give the tensors to PyTorch through DLPack.

At the second mix. The first export is a script. By the second, you maintain an export pipeline, whatever the size of the corpus.

Continue the loop

The model’s failures start the next loop.

InvestigateFind every failure like this one.EnrichAdd columns without rewriting the table.CurateCurate and test a training set live, and reproduce it forever.

How it works:GPU data loadingand Reversibility & isolation