Spiral Logo

Latest posts

From S3 to GPU in One Copy

Sep 30, 2026 · by Onur Satici · 10 min read

Vortex is an extensible, open-source columnar file format and toolkit. One feature of Vortex is that it stores data in ways well suited for processing on both CPUs and GPUs. In particular, it stores data in a format that GPUs can decompress efficiently, so we can transfer compressed data to the GPU without worrying about decompression performance. When processing cold data, the rate at which we can move data from cloud storage to the GPU becomes the new bottleneck. In this blog post, I explain how Vortex can read from S3 into GPU memory fast enough to saturate a 100 Gbps network link.1
While the Vortex team and I had to address many technical challenges to achieve this performance, the biggest win was ensuring that bytes from cloud storage were copied only once between the kernel's socket buffer and the GPU. On machines with few cores, excess copies and inefficient toolchain usage created so much unnecessary CPU work that the CPU could no longer keep enough network requests in flight to saturate network bandwidth. This is why we built the one-copy path, which uses less than half the CPU per gigabyte that the original reader did; that is what lets us fill the network link.
Two rows of buffers between the network card and the GPU. The original path: the NIC writes into a kernel buffer by DMA, then the CPU copies the data five times, into a TLS encrypted buffer, a TLS decrypted buffer, HTTP chunks, a segment buffer, and CUDA staging memory, before DMA to the GPU. The one-copy path: the NIC writes into the kernel buffer by DMA, the kernel decrypts it once into a pinned segment buffer provided by our HTTP client, and that buffer goes to the GPU by DMA.

The original reader

In our original reader, requested data arrives at a network card, which facilitates writing this over-the-wire data into a kernel-owned receive buffer via DMA. Then, five copies occur before we can begin to send data to the GPU:
  1. The kernel copies the TLS ciphertext into the TLS library's buffer.
  2. A decryption library (such as rustls) decrypts it into a plaintext buffer.
  3. An HTTP client strips the framing and puts the body into its own chunks.
  4. Vortex joins those chunks into one contiguous buffer for the segment, the unit of storage Vortex reads from a file.
  5. The CUDA driver copies that buffer into its own pinned staging memory, because the GPU can only DMA from page-locked memory.
Finally, the host-to-device copy can start.
Every byte passes through all five copies, so the copying work grows with throughput: moving data twice as fast takes twice the CPU. We measure this cost in cores per Gbps, the number of CPU cores it takes to sustain one gigabit per second. Throughput alone hides this cost whenever the network is the limit. Cores per Gbps shows it either way, and on a machine that runs short of CPU, it sets the throughput.

Removing the copies

Let's discuss what was required to remove each copy listed above and the impact of doing so. We worked backward from the GPU: we first addressed the staging copy (5), then the HTTP client's copies (4 and 3), and finally TLS, which consolidates copies 2 and 1.

Pooling pinned buffers in CPU memory

The last copy exists only because our segment buffer was ordinary pageable memory. If the buffer is pinned from the start, CUDA can transfer directly from it. However, pinned memory is expensive to allocate, so Vortex maintains a pool. To reuse buffers, we need a mechanism to indicate when a buffer is no longer in use. Thus, after queuing each transfer, we queue a CUDA event on the same stream. Once the event completes, all work queued before it has finished, and the buffer returns to the pool.
On its own, removing the staging copy barely changed throughput on our test machine and slightly raised the CPU cost per byte. Its value is what it enables: the next two steps read straight into these pinned buffers, which the GPU can DMA from without a staging copy.

Writing our own HTTP client

Copies 3 and 4 happen because the HTTP client puts the body (the response payload, without HTTP headers) into its own chunks, which Vortex then joins into the segment buffer. Note that Vortex knows every segment's offset and length from the file footer before any request goes out; thus, when retrieving scanned data, we can pass the network layer a pinned buffer of exactly the right size. We wrote a small HTTP/1.1 client that issues range requests over keep-alive connections, parses the response headers, and reads the body directly into that buffer.
This step cut the CPU cost per gigabyte by almost 60% and improved throughput by about a quarter. When profiling this optimization, we found that only about a sixth of the saving is memcpy. Most of it is per-chunk work in the HTTP stack and the async runtime:2 polling, bookkeeping, locking, and allocation for roughly 2,000 small chunks per request. Reading the body into one buffer removes that work.

Moving TLS to the kernel

Even with our own HTTP client, TLS still costs a copy: the kernel copies the ciphertext into a rustls buffer, and rustls decrypts it into ours. Linux can decrypt in the kernel instead (kTLS). rustls performs the handshake and passes the session keys to the kernel, which then decrypts each record directly into the buffer we pass it. That leaves one copy between the socket and the GPU.
The kernel's AES-GCM code is slower than the one rustls uses, by about a factor of two on our testing platform, but kTLS also removes the last userspace copy and rustls's record handling. The net effect was a further 7% cut in CPU overhead per byte, with throughput staying within run-to-run variation.

Overall results

We measured each step on a Google Cloud VM with an NVIDIA RTX PRO 6000 GPU, reading Vortex files from Cloud Storage into GPU memory.3
Compared with the original reader, the one-copy path moved 1.25 times as much data and needed 58% fewer CPU cores per Gbps. Further, based on measured network utilization, we find that we're using about 100% of the available data bandwidth.4
Three bar charts, one row per step. Throughput relative to the original reader: 1.00 for the original reader, 0.99 with a pinned buffer pool, 1.22 with our own HTTP client, 1.25 with kernel TLS. CPU per Gbps relative to the original reader: 1.00, 1.08, 0.44 and 0.42. Link utilization, payload as a share of what the 100 Gbps link can carry: 80%, 79%, 98% and about 100%.
This machine has more than enough CPU to handle the extra copies. Instead, our changes drastically reduce overall CPU utilization. When profiling the original reader, the copies we removed accounted for about a seventh of the CPU; decryption, the network stack, zeroing new memory, and the async machinery took the rest. The one-copy path removes both the copies and most of that machinery. For CPU-bound machines, these optimizations reduce the computational cost of managing individual buffers, leading to further gains in data-transfer throughput. Even with CPU to spare, the original reader left a fifth of the link idle; the one-copy path fills it, using less than half the CPU per byte.

Closing thoughts

By building a path from cloud storage to the GPU that saturates the network link, we learned a lot about what makes data movement hard. We not only eliminated unnecessary movement by reducing the copy count but also minimized the work the CPU has to do along the way to drive data transfers. Most of what our one-copy path saves is per-chunk overhead rather than pure memcpy overhead. On a machine with few cores, that freed CPU keeps more requests moving, which is where higher throughput comes from.
Making each byte cheaper to move raises the next question: did we have to download all of those bytes in the first place? Often not, though many existing file formats don't help us express that we only need a small portion of each file. We built this transfer machinery on top of Vortex because it can fetch just the parts of a file a query needs, which we've talked about in other blog posts (1, 2, 3).
I talked about this topic at QCon London 2026. The talk transcript is available here.

Footnotes

  1. The QCon London 2026 talk presented 60 Gbps. We have optimized the path further since then, and can now fully saturate a 100 Gbps network link. ↩
  2. The original reader uses object_store, which runs reqwest and hyper on the Tokio runtime. ↩
  3. We used a g4-standard-96 VM with 96 vCPUs. Our experiment read 36 GB of Vortex files from a Cloud Storage bucket (in the same region) into the device memory of one of its two (NVIDIA RTX PRO 6000) GPUs, sixteen files at a time. The results shown are the median of five runs. ↩
  4. About 5.7% of the bytes the network interface received (per its /proc/net/dev counter) were overhead: TCP/IP headers, TLS, and HTTP framing, on Google Cloud's 1,460-byte MTU. A 100 Gbps link therefore carries at most about 94.3 Gbps of data, which matches the one-copy path's median of 94.4 Gbps. ↩