Vortex: The Storage Layer for the Composable Data Stack

Suppose you're building a database.

Not a general-purpose database. A database for observability, genomics, retrieval, robotics, or whatever particular problem caused you to conclude that Postgres wasn't quite going to cut it.

You choose Apache DataFusion for the query engine. Sensible. It gives you SQL, a DataFrame API, joins, aggregates, spilling, an optimizer, object-store support, and enough extension points to add your own functions, operators, types, and data sources.

Then you reach for the default file format. Usually Parquet.

Also sensible.

And yet, somewhere between your custom functions and your generic files, you have accidentally built a specialized database on top of a storage format that knows nothing about the thing your database specializes in.

This is usually where the trouble starts.

The composable data stack is missing a layer.

DataFusion is interesting because it is not really a database. It is a box of high-quality database parts.

You bring the catalog, transaction model, distribution layer, and whatever makes your system useful. DataFusion provides the machinery beneath: parsing, planning, optimization, vectorized execution, scheduling, memory management, and a respectable collection of relational operators.

This is the basic argument for the composable data stack. Your observability database should not need to write its own hash join. Your vector database should not need to invent spilling. Your genomics database should not need to maintain a SQL parser because someone added QUALIFY.

Reuse the boring parts. Specialize the important ones.

This works extremely well until we reach storage, where the industry's commitment to composability often becomes:

Custom database
  → custom functions
  → custom operators
  → custom optimizer rules
  → standard file format

The file format is treated as the neutral choice. But a file format is not neutral. It fixes a large part of the physical plan before the query engine sees the first byte.

Page sizes, row groups, encodings, statistics, indexes, compression, and nesting all determine which reads are possible and how much data must be decoded to answer them.

A clean TableProvider boundary does not make those decisions disappear. It merely makes them somebody else's decisions.

The clearest evidence that general-purpose formats are not always sufficient is what sophisticated data systems do once their workloads become sufficiently specific.

They write another one.

Google built Capacitor around BigQuery's runtime: it uses workload-aware row ordering, chooses shards for massive parallelism, and can rewrite the physical layout as the system changes. Snowflake's format was shaped by S3 and elastic compute: immutable files expose columns independently and serve as units of caching, scheduling, and MVCC. Meta built Nimble for thousands-wide ML tables, recursive encodings, and parallel decoding. Datadog's Husky lays out whole columns sequentially so compaction can stream each fragment in a single GET with bounded memory.

Storage is part of the database design. Different workloads, different formats.

None of these formats was designed independently of its reader.

An engine does not abstractly "read a file." A Volcano-style engine pulls records through iterators. A vectorized engine operates on batches. A morsel-driven engine needs enough independent pieces of input to keep every worker busy.

The reader has to turn bytes into those units of work.

ClickHouse's MergeTree makes this explicit: parts are units of merging; granules are units of indexing and scanning.

Formats such as Parquet, ORC, and Avro solve an important problem: lots of systems can read the same bytes.

Parquet may be the most successful example. Spark can write a file that DuckDB, DataFusion, Trino, Pandas, and a Python script found in a forgotten Airflow job can all understand.

That kind of agreement requires formats to change carefully. A new encoding or pruning structure is a coordination problem across a specification and many independent implementations. Custom metadata can move faster, but generic readers cannot use metadata they do not understand.

This conservatism is useful. It makes Parquet dependable for interchange and long-lived storage. It also means Parquet cannot absorb every new encoding, index, or hardware-specific idea on the timetable of a query engine.

Compatibility and specialization are different contracts. A composable data stack should support both.

Vortex handles this with Editions. An edition is a frozen set of layouts and encodings with a minimum reader version. The default edition moves deliberately. An engine that controls both ends can opt into unstable editions—or its own encodings—and move much faster.

We plan to make that boundary a little more forgiving. Future files may carry WASM implementations of new encodings and indexes, giving an older reader enough code to keep working. Native support will be faster and capable of more push-down. But “slower” is still preferable to “cannot open the file.”

Once you accept that specialized storage is useful, the obvious response is to write a file format.

Please don't.

A file format starts pleasantly enough. You serialize a header, write a few buffers, and have something benchmarkable by Friday.

Then you need projection push-down. Predicate push-down. Statistics. Schema evolution. Async I/O. Read coalescing. Caching. Checksums. Object-store range requests. Parallel decoding. Work scheduling. Limits. Sparse row selection. Corruption handling. Metrics. A writer that does not use 3× the input size in memory.

Eventually you discover late materialization, which means your scan is now a small query engine. Then you add multiple indexes and need to choose between them, which means your file format has an optimizer.

At this point the format is no longer a description of bytes. It is a storage engine with an unusually inconvenient API.

The expensive part was never writing the integer little-endian. It was all the machinery around it.

Vortex exists to make specialized storage composable too.

At the bottom, Vortex has pluggable encodings: FSST strings, ALP floats, bit-packed integers, dictionaries, run-end encoding, and custom encodings for data Vortex has never heard of.

"Custom encoding" here does not mean picking a different integer codec from a menu. You can define a new physical array representation, give it children and metadata, and implement kernels that operate on it without first converting it to Arrow. A geometry type might have a spatial encoding. A posting list might use a bitmap. An embedding can have a quantized representation with its own dot product kernel.

Above arrays are layouts. A layout describes how arrays are partitioned and placed over out-of-memory storage. Layouts are hierarchical and user-extensible. A Vortex file is, approximately, a serialized layout tree plus its data segments.

That sounds abstract, so consider a few trees.

To reproduce Parquet's broad shape, start with row groups, split each row group into columns, then split each column into pages:

rows
  → row groups
    → columns
      → pages

For a scan-heavy workload, put the column split first, give every column its own chunk sizes, and add logical zones that are independent of physical I/O blocks:

columns
  → zone map
    → column-specific chunks
      → compressed arrays

For a very wide table, you might put each column in a separate object so reading three columns never touches the metadata or bytes of the other 9,997. For a local embedded database, one file with aligned segments may be better. For point lookups, you might avoid row groups entirely. For a network scan, you might buffer several compressed arrays into object-store-friendly ranges.

These are different formats in the sense that matters for performance. In Vortex, they are different compositions of the same layouts, arrays, scan engine, and I/O runtime.

Indexes and zone maps are another part of the layout rather than privileged features baked into the file specification.

It helps to think of an index as filter evidence. The scan asks whether a region of rows can possibly match a predicate. A minimum and maximum can prove that x = 42 will not match. A Bloom filter can provide better evidence for an unclustered identifier. Neither structure returns the rows; it lets the scan avoid paying to fetch and decode them.

Evidence has a cost. It occupies bytes, takes time to read, and only helps some predicates. The writer should therefore choose which evidence to collect, its granularity, and where to put it based on the workload. A timestamp may deserve fine-grained minima and maxima. A UUID may deserve a Bloom filter. Another column may deserve nothing.

Vortex zone maps store the partial results of aggregate functions for each logical zone. Min and max are merely the built-in examples. Aggregate functions are themselves pluggable, as are the rewrite rules that turn a filter into a falsification test over those aggregates.

Suppose an observability database frequently evaluates predicates such as message LIKE '%connection refused%'. A custom aggregate could hash every trigram present in a zone into a compact bitmap signature. A rewrite rule would extract the trigrams that must occur in the pattern. If even one is definitely absent from the signature, the entire zone can be skipped; possible matches continue to the normal row filter.

This is not an exotic data structure. PostgreSQL's pg_trgm uses bitmap signatures and extracted trigrams to accelerate LIKE and regular expressions. The interesting part is making that idea a pluggable zone aggregate. The database can choose the signature size, change the tokenizer, or try a different approximate containment structure without waiting for a file format committee and every reader implementation to agree.

Another workload can provide different evidence for its own predicates. Each domain-specific index lives beside the data it describes and participates in the same recursive pruning process.

This is the useful meaning of a custom file format: custom physical decisions and custom evidence, without a custom implementation of everything around them.

DataFusion and Vortex fit together because they specialize at different levels.

DataFusion knows about SQL, joins, aggregates, ordering, repartitioning, spilling, and whole-query optimization.

Vortex knows how to turn a request for a subset of rows and columns into the smallest useful set of reads.

The integration translates DataFusion expressions into Vortex expressions. Supported filters participate in file pruning and row-level filtering; unsupported expressions remain in DataFusion and execute after the scan. Projection push-down ensures only required columns are fetched. DataFusion schedules files and partitions while Vortex handles the physical work within each scan. The current integration's mechanics are described in the Vortex DataFusion documentation.

This boundary matters for custom functions.

A custom DataFusion function can always run after the scan. But if the function corresponds to something your storage understands—a geospatial predicate, a search operation, an embedding filter, or a domain-specific transform—you can also teach the expression converter and Vortex compute system about it.

Now the function can participate in push-down. A layout can prune around it. A compressed array can provide a specialized kernel. Columns that fail the predicate need not be materialized merely to cross the engine boundary.

You have not just added a UDF. You have extended the physical plan all the way down to the bytes.

There is an important implication here: Vortex does not have to begin and end with a .vortex file.

Layouts are bound to abstract segments, and scans are bound to sources. The file format is one useful way to serialize the layout tree and place its segments. It is not the boundary of the system.

Two pieces of upcoming work make this concrete at the table-format layer.

Iceberg owns schemas, snapshots, transactions, and the inventory of data files; the data-file format owns the physical representation inside each file. Iceberg's new File Format API makes that second layer extensible, and a full Vortex integration is already underway. This will let an Iceberg table use Vortex data files without asking Vortex to become a table catalog.

We are also beginning work on Vortex data files for DuckLake. The current DuckLake specification stores catalog state in SQL and requires Parquet for data. Supporting Vortex would preserve DuckLake's compact catalog and transaction model while making its physical storage a choice.

That is the composable boundary: table formats manage tables; Vortex manages how their data is physically represented and read.

The same abstractions open up several other directions.

The first is a Parquet reader built on the Vortex scan framework. Parquet would remain the bytes at rest, while its row groups, columns, pages, and statistics would be exposed through the same source and split abstractions used by Vortex. The format-specific reader would still understand Parquet. It would not also need to invent a new contract with every query engine for scheduling, push-down, and moving arrays across the boundary. The Vortex Scan API is being designed around exactly this N-by-M problem.

The second is putting Vortex layouts inside another database. A layout tree can be bound to segments stored in Postgres block storage instead of offsets in a standalone file. Postgres can own the relation and its durability model, while Vortex supplies columnar encodings, layout-aware reads, and compute push-down over those blocks.

The third is shuffle. Distributed engines often materialize an uncompressed interchange representation merely to move a batch from one stage to the next. Vortex uses the same serialized array representation in memory, on disk, and over the wire. Using Vortex as a shuffle interchange format could keep arrays compressed between workers and preserve useful encodings across the network boundary.

These are future directions, not features hidden behind an undocumented flag. But they are why the abstraction matters. Once storage is expressed as arrays, layouts, segments, and scans, a file becomes one deployment of the storage layer rather than the thing that defines it.

Spice built its Cayenne data accelerator around DataFusion and Vortex. In its Cayenne announcement, Spice calls out Vortex's pluggable compression, encoding, and layout strategies as the foundation for segment-level access and a multi-file architecture.

LangChain took the same components in a very different direction. Its SmithDB database uses DataFusion and Vortex, with "heavy customizations" for agent observability. LangChain reports that SmithDB made core LangSmith experiences up to 15× faster than before.

The more interesting detail came later. SmithDB implemented full-text search by making its inverted index another Vortex layout. Its planner pushes search predicates through the same expression interface; the layout registers the required object-store ranges with the same segment scheduler; a query that does not use the index never opens the index file.

These are whole-system results, not controlled benchmarks attributing every millisecond to Vortex. That is rather the point. A useful storage layer composes with the rest of the system. Spice and LangChain kept the shared machinery and specialized the parts their workloads made important.

Vortex gives a new database somewhere to put its physical opinions.

Start with the built-in encodings, layouts, scan engine, and query-engine integrations. Then specialize one layer at a time. Add an encoding for your data type. Add evidence for your dominant predicate. Change the layout when the next storage device makes today's page size look ridiculous. Push a custom function down until it reaches the representation that knows how to execute it.

The point is not that Vortex has discovered the one correct file format. It is that storage can keep learning. Encodings, evidence, layouts, and kernels can evolve with the workload instead of being frozen at the moment an ecosystem reached consensus.

The consequential choices remain yours, without also making you the owner of an I/O scheduler, a pruning engine, and another pile of bespoke readers.

The composable data stack freed new databases from writing their own query engine.

The same idea should free them from writing their own storage engine.

You probably do need a custom file format.

You just shouldn't have to build one from scratch.

Vortex is open source, and we want you to build on it. The team at Spiral also works on it full-time. If you're building a data stack around DataFusion, DuckDB, or Spark—and storage is beginning to look suspiciously like a research project—come talk to us. We can help design the layout, carry your custom functions through push-down, benchmark the result, and get the unglamorous I/O details right.