Spiral — Physical AI Infrastructure

From Sensors to GPUs,
Without the Pipeline Tax

Data infrastructure for teams training multimodal, world, and VLA models
🦾 Robotics 🌍 World Models 🧠 VLA Models 🔬 Life Sciences
The Challenge
📷
RGB + Depth Camera
30 – 60 Hz
Tensor[H,W,3] frames
Tensor[H,W] depth maps
+ timestamps
🦾
Robotic Arm State
500 Hz
Joint angles · torques
Gripper XYZ · actuator state
Tensor[6] per tick
📡
LiDAR / IMU
10 – 100 Hz
Point clouds · orientation
Velocity estimates
Categorical metadata
🏷️
Annotations
Sparse / async
Object labels · confidence
Velocity estimates
One row per detection
Each modality has a different rate, shape, and cardinality — yet training requires them aligned to a shared timeline
The Data Model
robot_demonstrations
└── sessions {_id: utf8}
├── operator: utf8
├── task: utf8
└── clips {_id: utf8}
├── start_time: f64
├── duration: f64
├── frames [] ← 30fps
│ ├── rgb: tensor<u8>[H,W,3]
│ ├── depth: tensor<f32>[H,W]
│ ├── rgb_patches: tensor<f32>[N,D]
│ └── depth_patches: tensor<f32>[N,D]
├── arm_state [] ← 500Hz
│ ├── joint_angles: tensor<f32>[6]
│ └── gripper_xyz: tensor<f32>[3]
└── detections [] ← sparse
├── label: utf8
├── confidence: f32
└── ts: f64
Sibling collections share a parent but have independent row counts — no forced joins at ingest time
Patch embeddings appended to frames after initial ingest — existing fields untouched
500Hz arm state stored separately — querying clip metadata never reads a single tensor
Annotations as a sibling collection — one row per detected object, not per frame
Robot arm with LiDAR point cloud and sensor coordinate frames
Timeline Alignment
RGB Frames
30 fps · first-in-bin
Arm State
500 Hz · mean-in-bin
Detections
sparse · list-in-bin
timeline(fps=10) · zip
Training Spine
10 fps unified · flat table
All streams share one tl object — combined via zip (lockstep, no searching). Ancestor fields broadcast via sort-merge on shared key prefix.
GPU-Direct Data Loading
Traditional
☁️
S3 Full Scan
full object download
↓
💾
NVMe Staging
materialize to local disk
↓
🔲
CPU Decompression
brotli / zstd / gzip — serial
↓
⬛
Copy to GPU
PCIe transfer — CPU-bound
Spiral
☁️
S3 Filtered Scan
predicate pushdown · byte ranges only
↓ direct stream
⚡
GPU Decompression
on-device decode · zero CPU cores
↓
🧠
Train / Eval
checkpointed · shard-aware DDP
No NVMe staging. No CPU decompression. Spiral streams the minimal byte ranges directly to GPU memory — approaching NIC line-rate.
Faster Iteration
Traditional Pipeline
Query 1
☁️
S3 Object Storage
full object download
↓
💾
NVMe Staging
materialize to local disk
↓
🔲
CPU Decompression
brotli / zstd / gzip
↓
⬛
Copy to GPU
PCIe — CPU-bound
New Query →
Query 2
☁️
S3 Object Storage
full object download
↓
💾
NVMe Staging
materialize to local disk
↓
🔲
CPU Decompression
brotli / zstd / gzip
↓
⬛
Copy to GPU
PCIe — CPU-bound
New Query →
Query 3
☁️
S3 Object Storage
full object download
↓
💾
NVMe Staging
materialize to local disk
↓
🔲
CPU Decompression
brotli / zstd / gzip
↓
⬛
Copy to GPU
PCIe — CPU-bound
Spiral
☁️
S3 Filtered Scan
predicate pushdown · byte ranges only
↓ direct stream
⚡
GPU Decompression
on-device decode · zero CPU cores
↓
🧠
Train / Eval
checkpointed · shard-aware DDP
where=sessions.task == "pick_and_place"
+ AND arm.gripper_xyz[2] > 0.3
One line. Nothing else changes.
Spiral Capabilities
🌳

Collection Tree

Sensor streams at different rates as sibling collections. Querying clip metadata never reads a frame tensor.

SCHEMA-NATIVE
📐

Timeline + Resample

Define a shared fps spine. Downsample video, mean-aggregate arm state, list-collect annotations — all in one declarative pass.

ZIP JOIN
➕

Live Column Appends

Write patch embeddings or labeling-pass annotations directly into existing collections. No re-ingestion, no ETL glue.

IN-PLACE
🎯

Window Sampling

Sample random windows using key tables of (session, clip, start). Reproducible, bias-free, shard-aware for DDP.

TRAINING-READY
⚡

GPU-Direct Stream

Predicate pushdown to S3. Decompress on-GPU. No NVMe staging. map_workers for tokenization without blocking I/O.

NIC LINE-RATE
🔁

Checkpointed Loader

Resume from the last processed record after interruption. No wasted epochs, no duplicate records across restarts.

FAULT-TOLERANT
End to End Training Loop
📷
Capture Sensors
cameras · arm · LiDAR
ingest
🏷️
Prepare & Annotate
embed · label · filter
resample
⏱️
Align Timeline
timeline · zip · flat table
window sample
🎲
Sample Windows
key table · sharded
stream
🧠
Train
GPU-direct · checkpointed
adjust filters & re-scan in seconds, not hours
No preprocessing step. Change your filters or embeddings —

re-scan the full dataset instantly.
Enabled by Vortex
Vortex State of the art
32.5 GB/s
Parquet Legacy standard
1 GB/s
0
8
16
24
32
GB/s throughput
32.5×
faster than Parquet
State-of-the-art columnar file format — built for GPU-direct reads, zero-copy decompression, and predicate pushdown at wire speed.
Models Improve Faster
0
IO Bottlenecks
No idle cycles waiting on CPU decompression or NVMe staging
5x
More Throughput
GPU-direct streaming eliminates CPU bottlenecks and staging I/O entirely
10x
Faster Iteration
Change a filter or embedding — re-scan the full dataset in seconds, not hours