Reversibility is the bottleneck

I've watched enough coding agents work to understand why software velocity is exploding: code work is made of small, reversible, compounding steps. Every change is a commit, every commit can be walked back, and nothing is ever truly lost. Data work has none of this. Too much of it happens in large, throw-away iterations: copies, scratch buckets, one-off datasets that never add up to anything.

Here's a morning that will feel familiar. Your training script skips a list of files by name, because those episodes were recorded before the firmware fix — the IMU reported degrees instead of radians. Someone has since reprocessed them, but the fix isn't in your copy of the dataset. You try to migrate. Three hours in, your script fails: it turns out they changed something else too. You tell yourself you'll ask on Slack what dataset-fixed-v0 actually contains, and you restart on the old version.

Later that day you have an idea worth trying — an agent can write the pipeline in minutes. But the dataset doesn't quite fit, and handling the mismatch at runtime would make every run crawl. The bitter part: you had a similar idea last month, and the transform you ran then would have covered this one too, but it lives in a scratch bucket nobody dares reuse. The idea goes into your backlog, a graveyard of missed opportunities, and you go back to watching the current run.

Frontier work is a search process, and the team that runs the most investigations and experiments per week wins. Ideas are cheap and compute is rentable; iteration throughput is what compounds. The way to better iterations is more iterations. In physical AI, a single experiment's input is terabytes of multimodal episodes, camera and lidar in MCAP, labels in Parquet, calibration in JSON, spread across more buckets than anyone will admit. The thing standing between you and the next iteration is almost always data logistics.

And the worst failures aren't the loud ones. Somebody "fixes" bad calibration by rewriting files in place, and every model trained on the old bytes quietly becomes unreproducible. Nothing noticed, because the paths didn't change. These failures are invisible on day one, catastrophic at month eighteen, and unfixable retroactively. You can't add provenance to data you already lost.

A robot fleet produces 5TB of MCAP a day without a human touching anything, but deciding whether last Tuesday's episodes are safe to train on is still a person spelunking a bucket. Machines are the primary producers and consumers of the data; humans still run every investigation and every experiment.

If we want investigations and experiments to run at machine scale too, the thing that has to change first is that a change to shared data can't be a one-way door.

Every change to your data would have to be a transaction. Every past state would have to stay addressable, not just the latest one, and with retention that respects time travel. The position your shipped model trained on stays hot for as long as the model matters, while the 30TB from last spring that no run ever pinned quietly ages into cold storage.

The record of those transactions can't be something the system merely writes down. The record has to be the only way changes happen.

There's a name for that: a journal.

Here's how that morning goes with a journal.

The teammate who reprocessed the broken episodes didn't make a dataset-fixed-v0. The fix landed as one transaction on the shared history, 312 episodes reprocessed with calibration v2, position 1042, with nothing rewritten in place and nothing for you to migrate to.

Your running experiment doesn't notice. It pinned position 1038 when it started, and nothing anyone does can disturb the position you're standing at. No freeze window, no "don't touch the dataset" Slack message; your teammate didn't need to know your run existed.

Trying the fix is launching your next run at 1042. And the "something else" that killed your migration three hours in? It's in the diff between 1038 and 1042, a transaction sitting right next to the calibration pass. You read what changed before you launch instead of discovering it when your script fails. If v2 makes the model worse, that's a result, not a mess: walking it back is launching at 1038 again, and "did we ever try recalibrating those?" is a query.

And the afternoon idea? The transform you ran last month isn't rotting in a scratch bucket nobody dares reuse. It landed as a transaction, so it's already part of the history your next run starts from. Today's idea runs on top of last month's work in the time it takes to type a position. Every change that lands moves the starting line for everyone, and negative results accumulate alongside the wins. The copies never added up to anything. The history adds up by default.

Table formats have understood this for a long time; transactions and time travel are why Iceberg and Delta won the warehouse. But time travel over a Parquet table doesn't version the 50TB of MCAP and point-cloud bytes the table points at. The table says episode 4711's camera stream lives at s3://…/cam0.mcap; the calibration "fix" rewrites that file in place; your snapshot now faithfully points at bytes that didn't exist when the snapshot was taken. The moment your rows reference external blobs, a snapshot is a snapshot of pointers to mutable things. And a real robotics dataset isn't a table. It's MCAP episodes plus URDF robot models plus calibration files plus sim scene descriptions; many assets that only mean something together, and no single container format with transactions can hold them all.

Branching bolted onto the bucket attacks the problem from the other side, and misses for the same reason. It diffs object keys, but the unit of meaning is the episode. One episode spans MCAP, video, labels, and calibration files, while one file holds a thousand episodes. The diff is too coarse and too fine at the same time; it answers what bytes changed, never what happened. Unless you can branch your whole data stack and changes land as commits small enough to understand and clean enough to merge, branching just enables more copies. The problem with copies was never the copying; storage is cheap. It's that they never add up to anything. A fork that can't merge is a scratch bucket with a nicer name.

So one approach versions the table but not the bytes, and the other versions the bytes but not the meaning. "Keep everything immutable and manage references carefully" is the C++ of data management.

With coding agents, generating experiments is no longer the bottleneck — risk is.

The question that keeps a team slow isn't "can we build it?" but "can we walk it back?" Agents can only be let loose on your data if every action they take is a transaction you can revert. A mutable pile of files is unsafe, but how about immutable? Fifty agents, each forking the dataset, produce fifty scratch buckets; you've automated the graveyard.

What makes a fleet productive is that each agent's work lands as a transaction the others can see, build on, and walk back. One asset that compounds instead of fifty copies that diverge.

Every workaround teams invented to go faster is actually a defense against irreversibility. Copies are defensive. Freeze windows are defensive. The scratch bucket is defensive. Teams aren't slow because they lack tools; they're slow because destructive operations on shared data are scary. Speed is what's left when the fear and the waiting are removed.

We want to get the primitives for machine-scale data work right, and we want adopting them to require no leap of faith. The journal is the first one we're deploying with design partners at Spiral. If you think this will help you move faster, we should talk: hello@spiraldb.com.