Spiral Logo

Latest posts

Vortex: One Format for Any Shape

Sep 16, 2026 · by Connor Tsui · 15 min read

This is a companion to a talk I'll give at EuroRust 2026, and it goes into more detail than I have time for on stage.

New data, old formats

There's a good chance your database has at least one of these: a JSON document in a TEXT column, a UUID in a VARCHAR(36), or a geometry packed into a BLOB. We don't store data this way on purpose.
Analytical databases used to hold columns of simple values: booleans, integers, decimals, strings, and the occasional timestamp. The data we put in them is no longer that simple! A single database can now contain raw images, videos, 1536-dimensional embeddings, and deeply nested JSON documents (with potentially hundreds of fields).
Not long ago, applications had to interpret much of this complex data themselves. Data lakes and lakehouses now give this data its own types and operations, so we can query it alongside simple data. Complex data became first-class.1
However, the columnar file formats underneath have not kept up.2 For example, Parquet has added support for modern types over time, but those types still have to fit its prescribed encodings and storage layouts.3 For something like a video, the table probably just holds a URI pointing at a file somewhere else.
Building on a simple storage model makes the formats easier to implement and optimize, but it means that your complex data must fit the format instead of the format fitting your data. When the existing encodings or layouts are a poor fit, adding a logical type doesn't solve that problem. And if you need to extend Parquet with a new type or encoding, it requires a lengthy process to change the specification. Each independent reader (Java, C++, Rust, Go) then needs its own implementation to support it.4

Vortex

Our claim is that one format can fit any shape of data and still be fast and small. Our proof of this is Vortex, an extensible columnar data format, built in Rust, zero-copy compatible with Apache Arrow, and open source under the Linux Foundation.5
Vortex files are largely self-describing, which makes the format customizable. The writer can make choices that other formats fix in the spec (such as how columns and chunks are nested) and record them in the file. You can combine the existing layouts without writing new code for the reader (provided it supports the file's version, encodings, and layouts).
Almost everything in Vortex is also extensible, including its types, encodings, and layouts. This means you don't have to convince everyone that your new type or encoding is worth adding. You don't have to wait for a committee to standardize features before you can rely on them. And you certainly don't have to build on a format that is still figuring out how to version itself.6
Note that readers will still need your code to understand the extension, but the important idea is that you don't have to build a new file format from scratch.
You can also contribute the extension back to Vortex. And because Vortex's language bindings share the same Rust implementation, an accepted extension doesn't need to be implemented again for every language.7
Vortex's extensibility falls out of a handful of design decisions. We'll follow an array through compression and computation, then see how to add encodings, storage layouts, and logical types of our own.

Logical types

Vortex's main data structure is an Array. You can think of an array as a column in a table: a list of values that all have the same logical data type.
But what is a "logical type"? It tells you what kind of values are in the array: integers, strings, and timestamps, for example. However, it does not tell you how these values are laid out in memory or on disk.
A 32-bit unsigned integer array contains only integers that fit in 32 bits. However, I could choose to physically store these 32-bit integers as 64-bit integers (maybe I want to waste half of my space 🤷).
Or more realistically, if I know that every integer is between 0 and 255, then I could "bitpack" the 32-bit integers into 8 bits, which gives me a 4x compression ratio.8
Vortex represents this logical description with a Rust enum called DType. The logical data type of the array in the example above would be DType::Primitive(PType::U32, Nullability::NonNullable), no matter how we physically store its values.9 DType has these variants:
Code Icon
Rust
// `Nullability` is shortened to `_` for readability.
pub enum DType {
    Null,                              // every value is null
    Bool(_),                           // true or false
    Primitive(PType, _),               // every integer and float width
    Decimal(DecimalDType, _),          // exact decimal numbers
    Utf8(_),                           // strings
    Binary(_),                         // raw bytes
    List(Arc<DType>, _),               // variable-length lists
    FixedSizeList(Arc<DType>, u32, _), // lists of exactly N elements
    Map(MapDType, _),                  // objects of key-value pairs
    Struct(StructFields, _),           // named and typed fields
    Union(UnionVariants, _),           // discriminated union values
    Variant(_),                        // dynamically typed values
    Extension(ExtDTypeRef),            // YOUR types go here...
}
Primitive covers Vortex's integer and floating-point types. Struct and List contain other DTypes, so types can nest. Nullability is also part of DType: nullable and non-nullable strings are different logical types. The Extension variant holds user-defined types, and we'll come back to that near the end of this article.
Every array carries a DType, but its DType still says nothing about how the values are physically represented. The same non-nullable U32 array can therefore take several different forms!

Physical encodings

We've written more about this distinction in Logical vs Physical data types, but here's a high-level overview.
The physical representation of an array is called its encoding. The non-nullable U32 array from above keeps the same logical values and DType no matter which physical encoding we choose.
Vortex represents different encodings with different Rust types. For example, dictionary encoding, run-end encoding, and bitpacking use DictArray, RunEndArray, and BitPackedArray.
Logical array · non-nullable U32
3331212777715153
DictArray
values: ArrayRef
312715
codes: ArrayRef
000112222330
RunEndArray
ends: ArrayRef
3591112
values: ArrayRef
3127153
BitPackedArray
bit_width: u8
4
packed: Buffer
001100110011110011000111011101110111111111110011
Fields are simplified. Bits do not show the FastLanes byte layout; metadata and padding are omitted.
You can think of each encoding as implementing a shared Array trait, with an ArrayRef hiding its concrete type. The code below is a simplified model, not Vortex's actual API (see the footnote for more information).10
Code Icon
Rust
type ArrayRef = Arc<dyn Array>;
If you are unfamiliar with Rust, dyn Array is a trait object, which allows us to use an object that implements the Array trait without knowing its concrete type (via dynamic dispatch).
The important part here is that each encoding supplies its own implementation of the shared interface. Vortex can then pass it around as an ArrayRef without adding a concrete encoding to a central enum or match expression.11
New encodings have generally taken us only a few weeks to add (sometimes days!), even before LLMs could help with the implementation.12
Vortex already ships several state-of-the-art encodings. Check out our other blogs!

Cascading compression

If you looked closely at the diagram, you may have noticed that the children of DictArray and RunEndArray are also ArrayRefs. That means we can compress each child with another encoding! A dictionary can replace repeated strings with integer codes, and those codes may have patterns of their own.
Consider a column with seven city names but only two distinct values, "Pittsburgh" and "New York City". The dictionary stores the two city names once, and each row holds a small integer code that selects one of them (an index into the dictionary).
Dictionary encoding
Store each distinct city once, then write its index for every row.
Uncompressed
ROWCITY
0Pittsburgh
1Pittsburgh
2Pittsburgh
3New York City
4New York City
5Pittsburgh
6Pittsburgh
Dictionary encoded
Values
INDEXVALUE
Codes
0
1
2
3
4
5
6
Seven input strings
Step 1 / 5
We've removed the repeated strings, but the dictionary codes are [0, 0, 0, 1, 1, 0, 0], which still contain three runs. Because the codes are themselves an array, Vortex can encode them as a RunEndArray. This replaces the seven codes with three exclusive ends, [3, 5, 7], and one value per run, [0, 1, 0].13
Each end is the index immediately after the run: the first run occupies rows 0, 1, and 2, so its end is 3.
Run-end encoding
Record one exclusive end and one value for each run of repeated codes.
Dictionary values
0:Pittsburgh1:New York City
Codes
000
11
00

Ends are exclusive: end 3 covers rows 0, 1, and 2.

codes: RunEndArray
ENDVALUEROWS
Seven dictionary codes
Step 1 / 5
7 is the largest run end, so each end fits in three bits. The run values are only 0 and 1, so each fits in one bit. Vortex can therefore bitpack both children.
Bitpack each integer child
Keep three bits for each run end and one bit for each run value.
ends
DECIMAL8-BIT WORD
3
00000
011
5
00000
101
7
00000
111
BitPacked<3>
values
DECIMAL8-BIT WORD
0
0000000
0
1
0000000
1
0
0000000
0
BitPacked<1>

Illustrative 8-bit inputs. Packing metadata and padding are omitted.

Two integer children
Step 1 / 5
The encoded array is now a tree! Dictionary encoding decomposes the original strings into values and codes. Run-end encoding decomposes those codes into ends and run values. Bitpacking then compresses both of those integer arrays.
DictArray
  • values: ArrayRef
    String values
    PittsburghNew York City
  • codes: ArrayRef
    RunEndArray
    • ends: ArrayRef
      BitPackedArray
      bit_width: 3 · exclusive ends
      357
    • values: ArrayRef
      BitPackedArray
      bit_width: 1
      010
Instead of asking one encoding to compress the whole column, Vortex can combine several encodings across the array tree. Each level exposes a simpler component of the data to the next encoding below. This allows encodings to compose, compressing the column better.14
This recursive process is known as cascading compression. After choosing an encoding, the compressor "cascades" down to repeat the process for each child array.
Vortex's compressor builds these cascades automatically by sampling the data to decide which encodings are the best to apply to an array. Vortex's compressor is inspired by the BtrBlocks research paper.15

Computation over compressed data

Writing is only one side of a data format. Compression saves storage and I/O, but decompressing an array for every query usually means spending more CPU time and memory in return. In the worst case, we might decompress and reconstruct values we don't even need.
However, we can be smart about this. The encodings generally give us interesting information about the data we have compressed, such as which values repeat and how often. Vortex can use that implicit structure to perform operations while the data is still compressed.

Sum of RunEnd

We can start simple by summing an integer array: [10, 10, 30, 30, 30, 50, 50, 50, 50]. If we encoded it as a RunEndArray, it would have ends [2, 5, 9] and values [10, 30, 50].
The simple way to do this is to decompress the entire array and add up all nine integers.
Decompress, then sum
Expand all nine integers, then add them one at a time.
RunEnd input
Ends
2
5
9
Values
10
30
50
Decompressed integers
Sum
0
Run-end input
Step 1 / 14
But the run ends already tell us how many times to count each value! Subtracting consecutive ends, starting from 0, gives us run lengths [2, 3, 4]. A specialized compute kernel can then multiply each run's value by its length and add the products: 10 * 2 + 30 * 3 + 50 * 4 = 310.16
Sum over compressed runs
Specialized compute kernel
Derive run lengths, multiply by each value, and add three products.
RunEnd input
Ends
2
5
9
Values
10
30
50
Run lengths and products
LengthValue × length
Sum
0
Run-end input
Step 1 / 7
In this example, we only save a few extra operations because the data is small. But hopefully you can imagine that a very large column with very long runs would benefit massively from this optimization!

Compare with Dict

Now suppose we want to find the rows where city == "New York City" in the original seven-row city array. We need a selection mask with one boolean per row. The naive approach is to decompress the array and compare each of the seven strings.
Decompress, then compare
Decode seven city names, then compare every string with New York City.
Dict input
Values
0Pittsburgh
1New York City
Codes
0001100
== "New York City"
RowDecoded cityMatch?
0
1
2
3
4
5
6
Selection mask
Dictionary input
Step 1 / 10
Vortex can do a lot better. It can represent an unevaluated operation as an array node, with its inputs as children. Here, the comparison sits above the dictionary array.
Because the array is dictionary encoded, Vortex can use a rewrite rule to push the comparison into the dictionary's values. It only needs to compare "New York City" with "Pittsburgh" and "New York City", which produces [false, true]. The codes child (which holds the RunEnd and BitPacked arrays cascaded beneath it) does not have to change for the comparison.17
Compare dictionary values
Rewrite rule
Push the predicate into values, then use unchanged codes to select booleans.
city == "New York City"
Dict input
Values
0
Pittsburgh
1
New York City
Codes
0001100

Logical codes; RunEnd -> BitPacked unchanged.

Boolean lookup
RowCodeMatch?
0
0
1
0
2
0
3
1
4
1
5
0
6
0
Selection mask
Predicate above Dict
Step 1 / 11
[false, true] tells us which dictionary values match, but we need to know which rows match. Each row's code selects one of those booleans: code 0 selects false, and code 1 selects true. For our original codes, that gives us the selection mask [false, false, false, true, true, false, false].
With seven rows, this only saves five string comparisons, but with one million rows and d dictionary values, Vortex would only need to compare d strings!

Filter Dict

Now suppose a different query gives us the selection mask [false, true, false, true, true, false, true] for the city array.
We could decompress all seven strings and then discard the three rows that the mask excludes.
Decompress, then filter
Reconstruct seven strings, then keep four rows selected by the mask.
Dict input
Values
0Pittsburgh
1New York City
Codes
0001100
Decoded strings and row mask
RowDecoded cityMask
0
false
1
true
2
false
3
true
4
true
5
false
6
true
Kept strings
Dictionary and selection mask
Step 1 / 10
But Vortex can instead use another rewrite rule to push the filter down to the dictionary's codes.
Applying the mask to the codes [0, 0, 0, 1, 1, 0, 0] gives us [0, 1, 1, 0]. Notably, we don't need to touch the values child at all to perform this selection filter!18
Filter dictionary codes
Rewrite rule
Push the selection mask into codes and reuse the dictionary values.
Dict input
Values
0Pittsburgh
1New York City
Codes
Filter
RowCodeMask
0
0
false
1
0
true
2
0
false
3
1
true
4
1
true
5
0
false
6
0
true
Result: Dict
Values
Filtered codes
Dictionary and selection mask
Step 1 / 9
These are all forms of late materialization: evaluating operations while the data is still compressed.19 Our post on push-down compute looks at this kind of encoding-level pushdown in more detail.
Vortex implements these optimizations in two ways: specialized compute kernels that run an operation directly over an encoding, and rewrite rules that rearrange the array tree.20 The optimizer will keep applying rewrite rules until the array tree stops changing.21
A rule first checks that the arrays meet its requirements. If no rule or kernel fits, Vortex can convert the array to its canonical form (the plain representation for its logical type) and run the operation there.22 We'll explain how this works in the next section.

Adding a new physical encoding

The encodings we have talked about so far all ship with Vortex by default. But that is not a requirement! A third-party encoding from another crate uses the same plugin interface as Dict, RunEnd, or BitPacked.
Canonicalization lets us normalize the physical encodings for a logical type to a shared form that Vortex's compute operations already know how to work with. This means we can implement canonicalization for a new encoding to get those operations working, then add specialized kernels to make them faster.
An array's execute method moves it towards its canonical form. For Dict, decompression means using each code to gather one value from the dictionary. In (heavily) simplified pseudocode, it looks like this:
Code Icon
Rust
// Execute into the "canonical" encoding, which everyone knows how to
// work with.
fn execute(dict: DictArray) -> CanonicalArray {
    dict.codes.map(|code| dict.values[code]).collect()
}
This gives every operation a fallback. If Vortex does not have a specialized compute path for the encoding, it can call execute until it reaches the canonical form and continue there.23
You can then register the encoding with a session, which holds the encodings and other extensions available to Vortex:
Code Icon
Rust
session.arrays().register(MyEncoding);
Vortex uses this registry when it reads a file and encounters the encoding.24
You can also register specialized compute kernels for the operations you want to optimize. The canonical fallback already makes everything else work.
Of course, registering the encoding is the easy part: you still have to validate child arrays, serialize metadata, teach the compressor when to choose the encoding, optimize computations, etc. All of this can live in your own crate (which readers will need to include), and you don't have to add the encoding to a central enum or fork the Vortex repository.

Adding a new storage layout

So far, our city array has lived entirely in memory. We compressed it into a tree of Dict, RunEnd, and BitPacked arrays, then ran operations over that tree without decompressing it. Eventually, though, we have to put those bytes somewhere.
We've written more about storage layouts and their tradeoffs in Data Layouts: Where Bytes Find Their Forever Home. Here, we'll stick to a high-level overview of how Vortex lets us compose and extend them.
The least clever option is to serialize the whole table into one big buffer (we use StructArray to represent multiple columns). Vortex calls this a FlatLayout: one serialized array in one segment. Suppose we only want a few rows from city. We still have to fetch and load the entire table, because we've given the reader just one segment to work with.
A Layout describes how Vortex breaks data into pieces. You can think of a Layout as an Array whose data may not be in memory yet. Like an Array, it has a dtype and children, so it also forms a tree. But its leaves point to segments in a local file, an in-memory cache, or an object on S3. The reader can then selectively fetch specific segments that a query needs instead of the entire file.25
Thus, Vortex has two kinds of trees: Array and Layout. The Array tree is where we combine encodings such as Dict and RunEnd. To change which bytes get fetched together, we change the Layout tree. Row groups, column chunks, pages, and zone maps are choices we can make via Layouts.26
Logical table
rowcityid
0Pittsburgh1
1Pittsburgh2
2Pittsburgh3
3New York City4
4New York City5
5Pittsburgh6
6Pittsburgh7
Requestcity · rows 0, 1, 2
FetchedNot fetched
Flat layout
FlatLayout
whole table
Segment · fetched
city · rows 0–6id · rows 0–6
Layout tree
Struct
  • city
    Chunked
    • rows 0–2
      Flat
      Segment · fetched
      city · rows 0–2
    • rows 3–6
      Flat
      Segment · not fetched
      city · rows 3–6
  • id
    Flat
    Segment · not fetched
    id · rows 0–6
Segment sizes are schematic.
Vortex ships Chunked, Struct, and a few other layouts by default, and we can compose them to create custom data layouts on disk. For example, by composing Chunked, Struct, and Flat layouts, we can recreate Parquet's PAX-style row group -> column chunk -> page hierarchy exactly if we want!27
The composition of layouts means that we can customize the I/O and storage for data stored by Vortex. For example, an S3 object and a local file do not necessarily want the same storage layout, so a crate can define and register layouts for different storage systems / access patterns without changing the logical type (or any of the encodings inside it).28
There are many tradeoffs when it comes to storage layout tuning. Smaller segments can improve pruning and random access but create more metadata and I/O overhead. We've improved Vortex's layouts a lot over the past year and a half, and there's more to this topic than we can cover here, so we'll come back to it in a follow-up post.

Adding a new logical type

We can now change how an array is compressed in memory and how it is arranged on disk. But without another abstraction, Vortex would still see a 1536-dimensional embedding as just a FixedSizeList of floats.
That difference matters because a vector has its own valid values and operations. If Vortex sees only a list of floats, every application has to enforce those rules itself.
This is the same issue as storing JSON in a TEXT column! The values "fit", but the column's type doesn't tell the system that they must be valid JSON. An embedding can fit into a list of floats in the same way, but the system does not know which operations belong to a Vector.
The Extension dtype allows us to extend the existing types provided by Vortex with new meaning. We can define Vector as a logical extension type over the existing FixedSizeList storage, with its own invariants and operations.
For example, Vector only accepts fixed-size lists whose elements are non-nullable floats (we likely do not want a bunch of Utf8 values in our vectors!). The ExtVTable trait lets us give the type an ID and check that its storage has the expected shape. In (heavily) simplified code, it looks like this:29
Code Icon
Rust
impl ExtVTable for Vector {
    fn id(&self) -> ExtId {
        ExtId::new("vortex.tensor.vector")
    }

    fn validate_dtype(dtype: &ExtDType<Self>) -> VortexResult<()> {
        let DType::FixedSizeList(element, ..) = dtype.storage_dtype()
        else {
            return error("storage must be a fixed-size list");
        };

        if !element.is_float() || element.is_nullable() {
            return error("elements must be non-nullable floats");
        }
        Ok(())
    }
}
The ID tells Vortex that this extension is a Vector. Vortex can keep using its existing list and float encodings underneath, while query engines see a Vector instead of an ordinary list.
Logical type
Vector · 3 dimensions
Storage type
FixedSizeList
3 non-nullable F32 elements per row
row 0
1.52.53.5
row 1
1.52.53.5
Physical encoding
DictArray
values: ArrayRef · F32
1.52.53.5
codes: ArrayRef
BitPackedArray
row 0
012
row 1
012
packed: Buffer · 2 bits per code
row 0
000110
row 1
000110
Bits are illustrative; metadata and padding are omitted.
You can then register the type with the session:
Code Icon
Rust
session.dtypes().register(Vector);
The same crate can also provide operations for the type. vortex-tensor, for example, registers cosine similarity, inner product, and L2 norm scalar functions with the session.
vortex-spatial uses the same approach for geospatial types such as Polygon, with operations including contains, distance, and intersects. Both Vector and Polygon are defined in their own crates, without adding variants to Vortex's central DType enum.
We've used these extension points for the more complex types we started with, too. In SpiralDB, our closed-source database, we've implemented video, image, and geospatial types on top of Vortex. Our video deep dive takes a closer look at how we use video's internal structure to fetch and decode the frames a query needs.

One format for any shape

If your data needs a type, encoding, or storage layout that Vortex doesn't ship, you can provide it in your own crate. Readers need that crate to use the extension, but you don't have to fork the container format.
Raw images and video, 1536-dimensional embeddings, deeply nested JSON documents, polygons, sparse data, repetitive values, and repeating sequences can all live inside the same format. Logical types can describe the complex values, and encodings can represent the different patterns in the values, including the Dict -> RunEnd -> BitPacked tree from earlier. A layout for a large table on S3 can fetch only the segments a query needs.
After all these extension points, you might expect Vortex to pay for its flexibility with a lot of overhead. But Vortex dispatches at array granularity, so a compute kernel still processes a whole vector at a time. And as we've seen, operating on compressed data can avoid work that decompressing it first would require.
Our benchmarks show compressed sizes comparable to Parquet, with significantly faster writes, scans, and random access. The suite also includes Lance and Arrow IPC: Vortex leads both in the hot random-access results while producing substantially smaller files. Results vary by workload and query engine, but Vortex's flexibility leaves plenty of room to outperform Parquet.
Vortex is available as both a Rust crate and a Python package. The documentation is at docs.vortex.dev.
Check out the repository and contact us if you are interested in using Vortex!

Footnotes

  1. You can see this across the major analytics platforms. Complex data now has its own types and functions:
    ↩
  2. Apache Arrow's format introduction shows the same table laid out by row and by column in memory. The idea is the same for files. ↩
  3. Parquet has eight physical types, and logical types add meaning to those primitives. JSON, for example, uses BYTE_ARRAY. Even VARIANT hides its structure behind a binary encoding: without shredding, it stores value and metadata as two BYTE_ARRAY fields. Shredding lets you pull parts of the value out into typed columns. Parquet also stores fixed-size lists as ordinary LISTs, so Arrow writers preserve the original type in ARROW:schema metadata. See Parquet's physical types, logical types, and the open FIXED_SIZE_LIST issue. ↩
  4. Parquet adds features by mailing-list discussion and vote. Older readers cannot decode new encodings. See Parquet's format-version documentation. ↩
  5. Each Vortex logical type has one canonical encoding, and each canonical encoding maps to an Arrow type. Conversions reuse the underlying buffers wherever the two layouts match. The canonical encoding rationale explains why we chose these representations. The vortex-arrow crate handles the conversions in both directions. ↩
  6. After months of discussion, Parquet voted to use major version numbers for changes that older readers can't read. But the vote left the implementation details to a separate working document, where several decisions are still under debate. ↩
  7. Vortex's language bindings guide describes four levels of support, from reading and writing Arrow data to registering plugins from the language you're using. The Python, C, Java, and DuckDB bindings all wrap the same Rust crates. ↩
  8. That 4x figure leaves out array metadata. Vortex uses the FastLanes bitpacking algorithm, which works on chunks of 1,024 values so it can unpack them efficiently with SIMD. We pad the final chunk to 1,024 values, so short arrays may not compress as well as the bit widths alone would suggest. ↩
  9. We've shortened the real DType enum here. You can find all eleven integer and float widths in PType. ↩
  10. In the real API, an encoding implements VTable and its related traits, which define the encoding's behavior. Vortex derives the private dispatch layer from those implementations. ArrayRef wraps Arc<ArrayInner<dyn DynArrayData>>, a single 16-byte fat pointer. The length, dtype, and encoding ID are fields we can read directly, without virtual calls. We've also drawn the child arrays as named fields, though the real API uses indexed slots. The Vtables and Dispatch guide goes into the details. ↩
  11. If you've used arrow-rs, this simplified model may look familiar. Its ArrayRef is exactly Arc<dyn Array>. DataFusion physical plans also have the same shape: each operator implements ExecutionPlan, and plans store their children as Arc<dyn ExecutionPlan>. Vortex wraps its arrays differently, but the idea is the same: callers use a shared interface without needing to know the concrete type behind it. ↩
  12. For comparison, Parquet's ALP specification PR took nearly five months to merge, while FSST support is still only a proposal. ↩
  13. Run-end encoding stores where each run ends rather than how long it is. This lets us use binary search to find the run containing a given row. Apache Arrow's Run-End Encoded layout makes the same choice, and Vortex imports Arrow run-end arrays directly into RunEndArray. ↩
  14. You might notice that this looks similar to a query execution plan. Vortex uses many of the same ideas, but its scope is smaller: it evaluates array operations such as scalar functions and filters, not relational operators such as sorts and joins. We have taken a lot of inspiration from classical query engines when designing Vortex's execution model. ↩
  15. The compressor's overview walks through how it chooses encodings. Each compression scheme first checks whether it can handle the array's canonical form, then estimates how well it will compress, usually by compressing a sample of about 1% of the values. Cascades stop at three layers, and we don't try the same scheme twice in one chain. Registering an encoding doesn't automatically teach the compressor to choose it. You can provide a compression scheme and add it through the compressor builder. ↩
  16. The RunEndSumKernel first executes the ends and values children into primitive arrays. It can then sum the runs without expanding them. sum_all_valid does the arithmetic from the example: subtract consecutive ends to get each run's length, multiply by the run's value, and add the products. The kernel is registered as an aggregate kernel for RunEnd. ↩
  17. The comparison pushdown is implemented by DictionaryScalarFnValuesPushDownRule. It applies the scalar function to values and reuses the existing codes. If the function can fail, we only do this when Vortex knows that every dictionary value is used. We don't want a query to fail on a value that no row refers to! ↩
  18. The dictionary filter implementation passes the selection mask to codes and reuses the existing values array. The filter is registered as a rewrite rule. This is similar to filter pushdown in a query engine. There, we move a filter through a query plan to discard rows earlier. Here, we move it through an encoding tree to avoid reconstructing strings. ↩
  19. The term comes from research on column stores. Abadi, Myers, DeWitt, and Madden's Materialization Strategies in a Column-Oriented DBMS (ICDE 2007) compares rebuilding rows early with waiting until after the filters run. Vortex applies the same idea one level lower: inside an encoding rather than across a query plan. ↩
  20. Vortex's ExecuteParentKernel interface lets an encoding execute the operation in its parent node. We call this fused execution. For example, bitpacking's fused comparison kernel compares each 1,024-value block against a constant as it unpacks the values. It puts the results straight into a bitmask, without building an intermediate array of integers. Aggregations use a separate kernel interface, but can also work directly over an encoding. ↩
  21. The try_optimize loop tries to simplify individual nodes and parent-child pairs until nothing changes. A 101-iteration limit guards against rules that keep undoing each other's changes. ↩
  22. Every DType has exactly one canonical encoding, listed in the Canonical enum. Strings and binary use Arrow's byte views (sometimes called German strings) as their canonical form. These are faster to filter than arrays that use offsets, though they occasionally need garbage collection. ↩
  23. A dictionary-encoded array executes in stages. First, it executes codes into a primitive array, then values into its canonical form. Then it calls take_canonical(values, codes) to look up the dictionary value for each code (roughly output[i] = values[codes[i]]). One call to execute doesn't have to finish the job, but it must bring the array closer to its canonical form. You can follow this through VTable::execute, Dict::execute, and take_canonical. ↩
  24. You can allow Vortex to read an unknown encoding, but it can only keep the structure as a placeholder. The reader still needs the encoding's code to decode the values or compute with them. See SerializedArray::decode. ↩
  25. The reader asks a SegmentSource for each segment, which can come from a file, an object store, a cache, or an in-memory buffer. FlatLayoutStrategy serializes one array and writes its buffers as one segment. The file stores those segments alongside a serialized layout tree and the footer metadata needed to find them. The vortex-file overview shows how these pieces fit together. The magic bytes VTXF live in a module named forever_constant, guarded by a test called never_change_these_constants. ↩
  26. Zone maps are the Zoned layout. It has a data child and a zones child with one row of aggregate statistics per zone. During a scan, Vortex uses those statistics to check whether any rows in a zone could match the filter. If it can rule out the whole zone, it skips reading that zone's data. ↩
  27. The original PAX paper explains the storage model behind Parquet's file hierarchy. Vortex's built-in layouts give us the pieces to build a similar hierarchy. The WriteStrategyBuilder is where you configure how the writer lays out the file. ↩
  28. To add a layout, you implement a VTable for reading and a LayoutStrategy for writing, then add the layout to the session registry. If you allow unknown layouts, Vortex can keep them as placeholders, but their reader returns an error. ↩
  29. In the real code, validate_vector_storage_dtype does these checks. We've written them out in the pseudocode so you can see what makes a valid vector. The full ExtVTable also handles metadata and scalar values; you can see that in the Vector implementation and its session registration. As the ExtVTable documentation explains, an extension dtype wraps a storage DType and adds metadata. It gives the data new meaning without defining a new physical array layout. ↩