Vortex: One Format for Any Shape
Connor Tsui
Published Sep 16, 2026 · 15 min read
This is a companion to a talk I'll give at EuroRust 2026, and it goes into more detail than I have time for on stage.
New data, old formats
There's a good chance your database has at least one of these: a JSON document
in a TEXT column, a UUID in a VARCHAR(36), or a geometry packed into a
BLOB. We don't store data this way on purpose.
Analytical databases used to hold columns of simple values: booleans, integers, decimals, strings, and the occasional timestamp. The data we put in them is no longer that simple! A single database can now contain raw images, videos, 1536-dimensional embeddings, and deeply nested JSON documents (with potentially hundreds of fields).
Not long ago, applications had to interpret much of this complex data themselves. Data lakes and lakehouses now give this data its own types and operations, so we can query it alongside simple data. Complex data became first-class.1
However, the columnar file formats underneath have not kept up.2 For example, Parquet has added support for modern types over time, but those types still have to fit its prescribed encodings and storage layouts.3 For something like a video, the table probably just holds a URI pointing at a file somewhere else.
Building on a simple storage model makes the formats easier to implement and optimize, but it means that your complex data must fit the format instead of the format fitting your data. When the existing encodings or layouts are a poor fit, adding a logical type doesn't solve that problem. And if you need to extend Parquet with a new type or encoding, it requires a lengthy process to change the specification. Each independent reader (Java, C++, Rust, Go) then needs its own implementation to support it.4
Vortex
Our claim is that one format can fit any shape of data and still be fast and small. Our proof of this is Vortex, an extensible columnar data format, built in Rust, zero-copy compatible with Apache Arrow, and open source under the Linux Foundation.5
Vortex files are largely self-describing, which makes the format customizable. The writer can make choices that other formats fix in the spec (such as how columns and chunks are nested) and record them in the file. You can combine the existing layouts without writing new code for the reader (provided it supports the file's version, encodings, and layouts).
Almost everything in Vortex is also extensible, including its types, encodings, and layouts. This means you don't have to convince everyone that your new type or encoding is worth adding. You don't have to wait for a committee to standardize features before you can rely on them. And you certainly don't have to build on a format that is still figuring out how to version itself.6
Note that readers will still need your code to understand the extension, but the important idea is that you don't have to build a new file format from scratch.
You can also contribute the extension back to Vortex. And because Vortex's language bindings share the same Rust implementation, an accepted extension doesn't need to be implemented again for every language.7
Vortex's extensibility falls out of a handful of design decisions. We'll follow an array through compression and computation, then see how to add encodings, storage layouts, and logical types of our own.
Logical types
Vortex's main data structure is an Array. You can think of an array as a
column in a table: a list of values that all have the same logical data type.
But what is a "logical type"? It tells you what kind of values are in the array: integers, strings, and timestamps, for example. However, it does not tell you how these values are laid out in memory or on disk.
A 32-bit unsigned integer array contains only integers that fit in 32 bits. However, I could choose to physically store these 32-bit integers as 64-bit integers (maybe I want to waste half of my space 🤷).
Or more realistically, if I know that every integer is between 0 and 255, then I could "bitpack" the 32-bit integers into 8 bits, which gives me a 4x compression ratio.8
Vortex represents this logical description with a Rust enum called DType. The
logical data type of the array in the example above would be
DType::Primitive(PType::U32, Nullability::NonNullable), no matter how we
physically store its values.9 DType has these variants:
// `Nullability` is shortened to `_` for readability.
pub enum DType {
Null, // every value is null
Bool(_), // true or false
Primitive(PType, _), // every integer and float width
Decimal(DecimalDType, _), // exact decimal numbers
Utf8(_), // strings
Binary(_), // raw bytes
List(Arc<DType>, _), // variable-length lists
FixedSizeList(Arc<DType>, u32, _), // lists of exactly N elements
Map(MapDType, _), // objects of key-value pairs
Struct(StructFields, _), // named and typed fields
Union(UnionVariants, _), // discriminated union values
Variant(_), // dynamically typed values
Extension(ExtDTypeRef), // YOUR types go here...
}
Primitive covers Vortex's integer and floating-point types. Struct and
List contain other DTypes, so types can nest. Nullability is also part of
DType: nullable and non-nullable strings are different logical types. The
Extension variant holds user-defined types, and we'll come back to that near
the end of this article.
Every array carries a DType, but its DType still says nothing about how the
values are physically represented. The same non-nullable U32 array can
therefore take several different forms!
Physical encodings
We've written more about this distinction in Logical vs Physical data types, but here's a high-level overview.
The physical representation of an array is called its encoding. The
non-nullable U32 array from above keeps the same logical values and DType no
matter which physical encoding we choose.
Vortex represents different encodings with different Rust types. For example,
dictionary encoding, run-end encoding, and bitpacking use DictArray,
RunEndArray, and BitPackedArray.
You can think of each encoding as implementing a shared Array trait, with an
ArrayRef hiding its concrete type. The code below is a simplified model, not
Vortex's actual API (see the footnote for more information).10
type ArrayRef = Arc<dyn Array>;
If you are unfamiliar with Rust, dyn Array is a
trait object, which
allows us to use an object that implements the Array trait without knowing its
concrete type (via dynamic dispatch).
The important part here is that each encoding supplies its own implementation of
the shared interface. Vortex can then pass it around as an ArrayRef without
adding a concrete encoding to a central enum or match expression.11
New encodings have generally taken us only a few weeks to add (sometimes days!), even before LLMs could help with the implementation.12
Vortex already ships several state-of-the-art encodings. Check out our other blogs!
Cascading compression
If you looked closely at the diagram, you may have noticed that the children of
DictArray and RunEndArray are also ArrayRefs. That means we can compress
each child with another encoding! A dictionary can replace repeated strings with
integer codes, and those codes may have patterns of their own.
Consider a column with seven city names but only two distinct values,
"Pittsburgh" and "New York City". The dictionary stores the two city names
once, and each row holds a small integer code that selects one of them (an index
into the dictionary).
We've removed the repeated strings, but the dictionary codes are
[0, 0, 0, 1, 1, 0, 0], which still contain three runs. Because the codes are
themselves an array, Vortex can encode them as a RunEndArray. This replaces
the seven codes with three exclusive ends, [3, 5, 7], and one value per run,
[0, 1, 0].13
Each end is the index immediately after the run: the first run occupies rows
0, 1, and 2, so its end is 3.
Ends are exclusive: end 3 covers rows 0, 1, and 2.
7 is the largest run end, so each end fits in three bits. The run values are
only 0 and 1, so each fits in one bit. Vortex can therefore bitpack both
children.
Illustrative 8-bit inputs. Packing metadata and padding are omitted.
The encoded array is now a tree! Dictionary encoding decomposes the original strings into values and codes. Run-end encoding decomposes those codes into ends and run values. Bitpacking then compresses both of those integer arrays.
- values: ArrayRefString valuesPittsburghNew York City
- codes: ArrayRefRunEndArray
- ends: ArrayRefBitPackedArraybit_width: 3 · exclusive ends357
- values: ArrayRefBitPackedArraybit_width: 1010
Instead of asking one encoding to compress the whole column, Vortex can combine several encodings across the array tree. Each level exposes a simpler component of the data to the next encoding below. This allows encodings to compose, compressing the column better.14
This recursive process is known as cascading compression. After choosing an encoding, the compressor "cascades" down to repeat the process for each child array.
Vortex's compressor builds these cascades automatically by sampling the data to decide which encodings are the best to apply to an array. Vortex's compressor is inspired by the BtrBlocks research paper.15
Computation over compressed data
Writing is only one side of a data format. Compression saves storage and I/O, but decompressing an array for every query usually means spending more CPU time and memory in return. In the worst case, we might decompress and reconstruct values we don't even need.
However, we can be smart about this. The encodings generally give us interesting information about the data we have compressed, such as which values repeat and how often. Vortex can use that implicit structure to perform operations while the data is still compressed.
Sum of RunEnd
We can start simple by summing an integer array:
[10, 10, 30, 30, 30, 50, 50, 50, 50]. If we encoded it as a RunEndArray, it
would have ends [2, 5, 9] and values [10, 30, 50].
The simple way to do this is to decompress the entire array and add up all nine integers.
But the run ends already tell us how many times to count each value! Subtracting
consecutive ends, starting from 0, gives us run lengths [2, 3, 4]. A
specialized compute kernel can then multiply each run's value by its length
and add the products: 10 * 2 + 30 * 3 + 50 * 4 = 310.16
| Length | Value × length |
|---|---|
In this example, we only save a few extra operations because the data is small. But hopefully you can imagine that a very large column with very long runs would benefit massively from this optimization!
Compare with Dict
Now suppose we want to find the rows where city == "New York City" in the
original seven-row city array. We need a selection mask with one boolean per
row. The naive approach is to decompress the array and compare each of the seven
strings.
| Row | Decoded city | Match? |
|---|---|---|
| 0 | ||
| 1 | ||
| 2 | ||
| 3 | ||
| 4 | ||
| 5 | ||
| 6 |
Vortex can do a lot better. It can represent an unevaluated operation as an array node, with its inputs as children. Here, the comparison sits above the dictionary array.
Because the array is dictionary encoded, Vortex can use a rewrite rule to
push the comparison into the dictionary's values. It only needs to compare
"New York City" with "Pittsburgh" and "New York City", which produces
[false, true]. The codes child (which holds the RunEnd and BitPacked
arrays cascaded beneath it) does not have to change for the comparison.17
Logical codes; RunEnd -> BitPacked unchanged.
| Row | Code | Match? |
|---|---|---|
| 0 | 0 | |
| 1 | 0 | |
| 2 | 0 | |
| 3 | 1 | |
| 4 | 1 | |
| 5 | 0 | |
| 6 | 0 |
[false, true] tells us which dictionary values match, but we need to know
which rows match. Each row's code selects one of those booleans: code 0
selects false, and code 1 selects true. For our original codes, that gives
us the selection mask [false, false, false, true, true, false, false].
With seven rows, this only saves five string comparisons, but with one million
rows and d dictionary values, Vortex would only need to compare d strings!
Filter Dict
Now suppose a different query gives us the selection mask
[false, true, false, true, true, false, true] for the city array.
We could decompress all seven strings and then discard the three rows that the mask excludes.
| Row | Decoded city | Mask |
|---|---|---|
| 0 | false | |
| 1 | true | |
| 2 | false | |
| 3 | true | |
| 4 | true | |
| 5 | false | |
| 6 | true |
But Vortex can instead use another rewrite rule to push the filter down to
the dictionary's codes.
Applying the mask to the codes [0, 0, 0, 1, 1, 0, 0] gives us
[0, 1, 1, 0]. Notably, we don't need to touch the values child at all to
perform this selection filter!18
| Row | Code | Mask |
|---|---|---|
| 0 | 0 | false |
| 1 | 0 | true |
| 2 | 0 | false |
| 3 | 1 | true |
| 4 | 1 | true |
| 5 | 0 | false |
| 6 | 0 | true |
These are all forms of late materialization: evaluating operations while the data is still compressed.19 Our post on push-down compute looks at this kind of encoding-level pushdown in more detail.
Vortex implements these optimizations in two ways: specialized compute kernels that run an operation directly over an encoding, and rewrite rules that rearrange the array tree.20 The optimizer will keep applying rewrite rules until the array tree stops changing.21
A rule first checks that the arrays meet its requirements. If no rule or kernel fits, Vortex can convert the array to its canonical form (the plain representation for its logical type) and run the operation there.22 We'll explain how this works in the next section.
Adding a new physical encoding
The encodings we have talked about so far all ship with Vortex by default. But
that is not a requirement! A third-party encoding from another crate uses the
same plugin interface as Dict, RunEnd, or BitPacked.
Canonicalization lets us normalize the physical encodings for a logical type to a shared form that Vortex's compute operations already know how to work with. This means we can implement canonicalization for a new encoding to get those operations working, then add specialized kernels to make them faster.
An array's execute method moves it towards its canonical form. For Dict,
decompression means using each code to gather one value from the dictionary. In
(heavily) simplified pseudocode, it looks like this:
// Execute into the "canonical" encoding, which everyone knows how to
// work with.
fn execute(dict: DictArray) -> CanonicalArray {
dict.codes.map(|code| dict.values[code]).collect()
}
This gives every operation a fallback. If Vortex does not have a specialized
compute path for the encoding, it can call execute until it reaches the
canonical form and continue there.23
You can then register the encoding with a session, which holds the encodings and other extensions available to Vortex:
session.arrays().register(MyEncoding);
Vortex uses this registry when it reads a file and encounters the encoding.24
You can also register specialized compute kernels for the operations you want to optimize. The canonical fallback already makes everything else work.
Of course, registering the encoding is the easy part: you still have to validate child arrays, serialize metadata, teach the compressor when to choose the encoding, optimize computations, etc. All of this can live in your own crate (which readers will need to include), and you don't have to add the encoding to a central enum or fork the Vortex repository.
Adding a new storage layout
So far, our city array has lived entirely in memory. We compressed it into a
tree of Dict, RunEnd, and BitPacked arrays, then ran operations over that
tree without decompressing it. Eventually, though, we have to put those bytes
somewhere.
We've written more about storage layouts and their tradeoffs in Data Layouts: Where Bytes Find Their Forever Home. Here, we'll stick to a high-level overview of how Vortex lets us compose and extend them.
The least clever option is to serialize the whole table into one big buffer (we
use StructArray to represent multiple columns). Vortex calls this a
FlatLayout: one serialized array in one segment. Suppose we only want a few
rows from city. We still have to fetch and load the entire table, because
we've given the reader just one segment to work with.
A Layout describes how Vortex breaks data into pieces. You can think of a
Layout as an Array whose data may not be in memory yet. Like an Array, it
has a dtype and children, so it also forms a tree. But its leaves point to
segments in a local file, an in-memory cache, or an object on S3. The reader
can then selectively fetch specific segments that a query needs instead of the
entire file.25
Thus, Vortex has two kinds of trees: Array and Layout. The Array tree is
where we combine encodings such as Dict and RunEnd. To change which bytes
get fetched together, we change the Layout tree. Row groups, column chunks,
pages, and zone maps are choices we can make via Layouts.26
| row | city | id |
|---|---|---|
| 0 | Pittsburgh | 1 |
| 1 | Pittsburgh | 2 |
| 2 | Pittsburgh | 3 |
| 3 | New York City | 4 |
| 4 | New York City | 5 |
| 5 | Pittsburgh | 6 |
| 6 | Pittsburgh | 7 |
- cityChunked
- rows 0–2FlatSegment · fetchedcity · rows 0–2
- rows 3–6FlatSegment · not fetchedcity · rows 3–6
- idFlatSegment · not fetchedid · rows 0–6
Vortex ships Chunked, Struct, and a few other layouts by default, and we can
compose them to create custom data layouts on disk. For example, by composing
Chunked, Struct, and Flat layouts, we can recreate Parquet's PAX-style row
group -> column chunk -> page hierarchy exactly if we want!27
The composition of layouts means that we can customize the I/O and storage for data stored by Vortex. For example, an S3 object and a local file do not necessarily want the same storage layout, so a crate can define and register layouts for different storage systems / access patterns without changing the logical type (or any of the encodings inside it).28
There are many tradeoffs when it comes to storage layout tuning. Smaller segments can improve pruning and random access but create more metadata and I/O overhead. We've improved Vortex's layouts a lot over the past year and a half, and there's more to this topic than we can cover here, so we'll come back to it in a follow-up post.
Adding a new logical type
We can now change how an array is compressed in memory and how it is arranged on
disk. But without another abstraction, Vortex would still see a 1536-dimensional
embedding as just a FixedSizeList of floats.
That difference matters because a vector has its own valid values and operations. If Vortex sees only a list of floats, every application has to enforce those rules itself.
This is the same issue as storing JSON in a TEXT column! The values "fit", but
the column's type doesn't tell the system that they must be valid JSON. An
embedding can fit into a list of floats in the same way, but the system does not
know which operations belong to a Vector.
The Extension dtype allows us to extend the existing types provided by Vortex
with new meaning. We can define Vector as a logical extension type over the
existing FixedSizeList storage, with its own invariants and operations.
For example, Vector only accepts fixed-size lists whose elements are
non-nullable floats (we likely do not want a bunch of Utf8 values in our
vectors!). The ExtVTable trait lets us give the type an ID and check that its
storage has the expected shape. In (heavily) simplified code, it looks like
this:29
impl ExtVTable for Vector {
fn id(&self) -> ExtId {
ExtId::new("vortex.tensor.vector")
}
fn validate_dtype(dtype: &ExtDType<Self>) -> VortexResult<()> {
let DType::FixedSizeList(element, ..) = dtype.storage_dtype()
else {
return error("storage must be a fixed-size list");
};
if !element.is_float() || element.is_nullable() {
return error("elements must be non-nullable floats");
}
Ok(())
}
}
The ID tells Vortex that this extension is a Vector. Vortex can keep using its
existing list and float encodings underneath, while query engines see a Vector
instead of an ordinary list.
You can then register the type with the session:
session.dtypes().register(Vector);
The same crate can also provide operations for the type.
vortex-tensor, for example,
registers cosine similarity, inner product, and L2 norm scalar functions with
the session.
vortex-spatial uses the same approach for geospatial types such as
Polygon, with
operations including contains,
distance, and intersects. Both Vector and Polygon are defined in their own
crates, without adding variants to Vortex's central DType enum.
We've used these extension points for the more complex types we started with, too. In Spiral, our closed-source database, we've implemented video, image, and geospatial types on top of Vortex. Our video deep dive takes a closer look at how we use video's internal structure to fetch and decode the frames a query needs.
One format for any shape
If your data needs a type, encoding, or storage layout that Vortex doesn't ship, you can provide it in your own crate. Readers need that crate to use the extension, but you don't have to fork the container format.
Raw images and video, 1536-dimensional embeddings, deeply nested JSON documents,
polygons, sparse data, repetitive values, and repeating sequences can all live
inside the same format. Logical types can describe the complex values, and
encodings can represent the different patterns in the values, including the
Dict -> RunEnd -> BitPacked tree from earlier. A layout for a large table
on S3 can fetch only the segments a query needs.
After all these extension points, you might expect Vortex to pay for its flexibility with a lot of overhead. But Vortex dispatches at array granularity, so a compute kernel still processes a whole vector at a time. And as we've seen, operating on compressed data can avoid work that decompressing it first would require.
Our benchmarks show compressed sizes comparable to Parquet, with significantly faster writes, scans, and random access. The suite also includes Lance and Arrow IPC: Vortex leads both in the hot random-access results while producing substantially smaller files. Results vary by workload and query engine, but Vortex's flexibility leaves plenty of room to outperform Parquet.
Vortex is available as both a Rust crate and a Python package. The documentation is at docs.vortex.dev.
Check out the repository and contact us if you are interested in using Vortex!
Footnotes
-
You can see this across the major analytics platforms. Complex data now has its own types and functions:
- Snowflake has
VARIANT,GEOGRAPHY, andVECTORtypes. Cortex AI functions such asAI_TRANSCRIBEandAI_PARSE_DOCUMENTreadFILEvalues directly in SQL. - Databricks has
VARIANTin Delta Lake, along with nativeGEOMETRYandGEOGRAPHYtypes. AI functions such asai_parse_documentrun over files in Unity Catalog volumes. - BigQuery has
JSONandGEOGRAPHYtypes, plusVECTOR_SEARCHfor embedding arrays. Its object tables holdObjectRefvalues, which letAI.GENERATErun over the images and documents that a table column points to. - Redshift stores nested documents as
SUPERvalues that you can query with PartiQL. It also hasGEOMETRYandGEOGRAPHYspatial types.
- Snowflake has
-
Apache Arrow's format introduction shows the same table laid out by row and by column in memory. The idea is the same for files. ↩︎
-
Parquet has eight physical types, and logical types add meaning to those primitives. JSON, for example, uses
BYTE_ARRAY. EvenVARIANThides its structure behind a binary encoding: without shredding, it storesvalueandmetadataas twoBYTE_ARRAYfields. Shredding lets you pull parts of the value out into typed columns. Parquet also stores fixed-size lists as ordinaryLISTs, so Arrow writers preserve the original type inARROW:schemametadata. See Parquet's physical types, logical types, and the open FIXED_SIZE_LIST issue. ↩︎ -
Parquet adds features by mailing-list discussion and vote. Older readers cannot decode new encodings. See Parquet's format-version documentation. ↩︎
-
Each Vortex logical type has one canonical encoding, and each canonical encoding maps to an Arrow type. Conversions reuse the underlying buffers wherever the two layouts match. The canonical encoding rationale explains why we chose these representations. The
vortex-arrowcrate handles the conversions in both directions. ↩︎ -
After months of discussion, Parquet voted to use major version numbers for changes that older readers can't read. But the vote left the implementation details to a separate working document, where several decisions are still under debate. ↩︎
-
Vortex's language bindings guide describes four levels of support, from reading and writing Arrow data to registering plugins from the language you're using. The Python, C, Java, and DuckDB bindings all wrap the same Rust crates. ↩︎
-
That
4xfigure leaves out array metadata. Vortex uses the FastLanes bitpacking algorithm, which works on chunks of 1,024 values so it can unpack them efficiently with SIMD. We pad the final chunk to 1,024 values, so short arrays may not compress as well as the bit widths alone would suggest. ↩︎ -
We've shortened the real
DTypeenum here. You can find all eleven integer and float widths inPType. ↩︎ -
In the real API, an encoding implements
VTableand its related traits, which define the encoding's behavior. Vortex derives the private dispatch layer from those implementations.ArrayRefwrapsArc<ArrayInner<dyn DynArrayData>>, a single 16-byte fat pointer. The length, dtype, and encoding ID are fields we can read directly, without virtual calls. We've also drawn the child arrays as named fields, though the real API uses indexed slots. The Vtables and Dispatch guide goes into the details. ↩︎ -
If you've used arrow-rs, this simplified model may look familiar. Its
ArrayRefis exactlyArc<dyn Array>. DataFusion physical plans also have the same shape: each operator implementsExecutionPlan, and plans store their children asArc<dyn ExecutionPlan>. Vortex wraps its arrays differently, but the idea is the same: callers use a shared interface without needing to know the concrete type behind it. ↩︎ -
For comparison, Parquet's ALP specification PR took nearly five months to merge, while FSST support is still only a proposal. ↩︎
-
Run-end encoding stores where each run ends rather than how long it is. This lets us use binary search to find the run containing a given row. Apache Arrow's Run-End Encoded layout makes the same choice, and Vortex imports Arrow run-end arrays directly into
RunEndArray. ↩︎ -
You might notice that this looks similar to a query execution plan. Vortex uses many of the same ideas, but its scope is smaller: it evaluates array operations such as scalar functions and filters, not relational operators such as sorts and joins. We have taken a lot of inspiration from classical query engines when designing Vortex's execution model. ↩︎
-
The compressor's overview walks through how it chooses encodings. Each compression scheme first checks whether it can handle the array's canonical form, then estimates how well it will compress, usually by compressing a sample of about 1% of the values. Cascades stop at three layers, and we don't try the same scheme twice in one chain. Registering an encoding doesn't automatically teach the compressor to choose it. You can provide a compression scheme and add it through the compressor builder. ↩︎
-
The
RunEndSumKernelfirst executes theendsandvalueschildren into primitive arrays. It can then sum the runs without expanding them.sum_all_validdoes the arithmetic from the example: subtract consecutive ends to get each run's length, multiply by the run's value, and add the products. The kernel is registered as an aggregate kernel forRunEnd. ↩︎ -
The comparison pushdown is implemented by
DictionaryScalarFnValuesPushDownRule. It applies the scalar function tovaluesand reuses the existingcodes. If the function can fail, we only do this when Vortex knows that every dictionary value is used. We don't want a query to fail on a value that no row refers to! ↩︎ -
The dictionary filter implementation passes the selection mask to
codesand reuses the existingvaluesarray. The filter is registered as a rewrite rule. This is similar to filter pushdown in a query engine. There, we move a filter through a query plan to discard rows earlier. Here, we move it through an encoding tree to avoid reconstructing strings. ↩︎ -
The term comes from research on column stores. Abadi, Myers, DeWitt, and Madden's Materialization Strategies in a Column-Oriented DBMS (ICDE 2007) compares rebuilding rows early with waiting until after the filters run. Vortex applies the same idea one level lower: inside an encoding rather than across a query plan. ↩︎
-
Vortex's
ExecuteParentKernelinterface lets an encoding execute the operation in its parent node. We call this fused execution. For example, bitpacking's fused comparison kernel compares each 1,024-value block against a constant as it unpacks the values. It puts the results straight into a bitmask, without building an intermediate array of integers. Aggregations use a separate kernel interface, but can also work directly over an encoding. ↩︎ -
The
try_optimizeloop tries to simplify individual nodes and parent-child pairs until nothing changes. A 101-iteration limit guards against rules that keep undoing each other's changes. ↩︎ -
Every
DTypehas exactly one canonical encoding, listed in theCanonicalenum. Strings and binary use Arrow's byte views (sometimes called German strings) as their canonical form. These are faster to filter than arrays that use offsets, though they occasionally need garbage collection. ↩︎ -
A dictionary-encoded array executes in stages. First, it executes
codesinto a primitive array, thenvaluesinto its canonical form. Then it callstake_canonical(values, codes)to look up the dictionary value for each code (roughlyoutput[i] = values[codes[i]]). One call to execute doesn't have to finish the job, but it must bring the array closer to its canonical form. You can follow this throughVTable::execute,Dict::execute, andtake_canonical. ↩︎ -
You can allow Vortex to read an unknown encoding, but it can only keep the structure as a placeholder. The reader still needs the encoding's code to decode the values or compute with them. See
SerializedArray::decode. ↩︎ -
The reader asks a
SegmentSourcefor each segment, which can come from a file, an object store, a cache, or an in-memory buffer.FlatLayoutStrategyserializes one array and writes its buffers as one segment. The file stores those segments alongside a serialized layout tree and the footer metadata needed to find them. Thevortex-fileoverview shows how these pieces fit together. The magic bytesVTXFlive in a module namedforever_constant, guarded by a test callednever_change_these_constants. ↩︎ -
Zone maps are the
Zonedlayout. It has a data child and azoneschild with one row of aggregate statistics per zone. During a scan, Vortex uses those statistics to check whether any rows in a zone could match the filter. If it can rule out the whole zone, it skips reading that zone's data. ↩︎ -
The original PAX paper explains the storage model behind Parquet's file hierarchy. Vortex's built-in layouts give us the pieces to build a similar hierarchy. The
WriteStrategyBuilderis where you configure how the writer lays out the file. ↩︎ -
To add a layout, you implement a
VTablefor reading and aLayoutStrategyfor writing, then add the layout to the session registry. If you allow unknown layouts, Vortex can keep them as placeholders, but their reader returns an error. ↩︎ -
In the real code,
validate_vector_storage_dtypedoes these checks. We've written them out in the pseudocode so you can see what makes a valid vector. The fullExtVTablealso handles metadata and scalar values; you can see that in theVectorimplementation and its session registration. As theExtVTabledocumentation explains, an extension dtype wraps a storageDTypeand adds metadata. It gives the data new meaning without defining a new physical array layout. ↩︎
Train on everything you record.
Spiral keeps video, sensor streams, agent traces, spatial data, and rows in one system. Analytics, search, and training read them straight from object storage.