Statistics are aggregates

Suppose we want to run this query over data stored on disk:

SELECT * FROM measurements WHERE x < 10;

Columnar formats split each column into chunks of rows and store statistics, such as the minimum and maximum, for each one. Vortex calls these chunks zones.¹ If the values of x in a zone are between 50 and 90, no row in that zone can satisfy x < 10, so statistics let readers skip the zone without decoding it.

This is one of the oldest techniques in analytical databases. Modern questions focus on which statistics to keep and how many rows each should cover, both of which depend on the data and the query (e.g., Prammer et al., CIDR '25). Moerkotte's original paper from 1998 called these statistics small materialized aggregates (SMA). While many systems call these statistics zone maps, we think it's worthwhile to take the name SMA literally, since we compute an aggregate function (e.g., MIN(x)) over the zone's rows when the file is written.

However, many query engines don't treat SMA that way. They use statistics to skip zones that can't match a filter, but then compute aggregates from the rows a scan returns, as if the statistics were never there. We think that separation keeps engines from using work the file has already done. A statistic is an aggregate, and an engine that treats it as one can use it to answer part of a query rather than only to avoid reading data.

SELECT MIN(x) FROM measurements;

First, let's talk about using statistics to answer simple queries (without filters). If every zone stores its minimum, the file's minimum is the smallest of the zone minimums, and the table's minimum is the smallest of the file minimums. We can answer this query from metadata at any level we have it, without reading x. MAX and COUNT merge the same way, with MAX taking the maximum and COUNT adding.

AVG doesn't work this way. Take two zones, [10, 20] and [100]. Their averages are 15 and 100, and averaging those yields 57.5, whereas the average of all three values is about 43.33. Averaging the averages gives the one-row zone the same weight as the two-row zone.

Instead, we should keep the sum and count for each zone: (30, 2) and (100, 1). We add the sums and counts to get (130, 3), and only then do we divide. These pairs are aggregate partials: the state we maintain for a set of rows so results can be merged before the final answer is produced. For MIN, the partial is the minimum itself. For AVG, it is (SUM(x), COUNT(x)), where the count excludes nulls.

To merge statistics up from zones to files to tables, we have to store partials rather than final answers. Vortex already defines its statistics this way: each one is the partial state of an aggregate function, computed by the same code that runs during query execution.

Of course, few queries aggregate an entire table. A more typical query first filters the rows:

SELECT MIN(x)
FROM measurements
WHERE recorded_at >= TIMESTAMP '2026-09-01 00:00:00';

The table's stored minimum of x might come from a row in August rather than from the September-or-later filter; thus, table-level statistics can't help us here. The further up we merge, the more rows a partial covers, and the more likely it is that some of them fail the filter. Because a table-wide minimum answers almost no query with a WHERE clause, engines rarely attempt to use them during query processing. DataFusion's aggregate-statistics rule answers a query from metadata only under strict conditions, such as exact statistics and no GROUP BY clause. After the WHERE clause above, its statistics for x are no longer exact, and the query falls back to scanning x.

Merging statistics from zone-level to file-level took us up the hierarchy. To answer filtered queries, we go back down, where zones are small enough that the filter treats all of a zone's rows the same way.

Suppose the data is clustered by time. A zone entirely before September contributes nothing, so we skip it. If every row in a zone is from September or later, the zone's stored minimum of x is the answer for that zone. For a zone that spans the cutoff, we read recorded_at to identify the matching rows and compute the minimum of their x values.

Sorting zones into these three cases is the same min-max check we use for skipping, applied to recorded_at. If the time statistics aren't stored, we must read recorded_at to decide. Decomposing an outer statistic that couldn't be used back into its component statistics works because the data is clustered on the filter column. Most zones fall entirely before the cutoff, where statistics let us skip them, or entirely after it, where statistics answer for them. Only the few zones that span the cutoff need a scan.²

Stored minimums also help with zones that don't fully match. Suppose every row in one zone matches and its stored minimum is 4. If the next zone has a stored minimum of 9, nothing in it can beat the 4 we already have, so we skip it without checking the dates. If its stored minimum were 1 instead, that 1 might belong to a row from August. It tells us only that the minimum of the matching rows is at least 1, so we still have to find those rows.

The 4 and the 1 represent different kinds of results. The 4 is exact for the rows in its zone. The 1 is only a bound. Statistics from other systems need the same distinction: Iceberg's column bounds needn't be values that occur in the data, and an exact minimum becomes a bound once the row that held it is deleted. DataFusion's dynamic filters for MIN and MAX already use a running result to skip zones like the one with 9, but they don't accept the 4 as part of the answer.

The same mix of stored and computed partials works for AVG. Suppose the first zone in our earlier example has its (sum, count) stored, while the second doesn't, so we scan it:

In the diagram, the source merges the two partials and hands the engine (130, 3). Alternatively, it could return the stored (30, 2) and let the engine scan [100] and merge. Either way, the engine needs to understand the partial state to perform the final division and to know that (30, 2) replaces a scan of [10, 20]. Counting both would count those rows twice.

For a source to hand an engine work it has already done, it has to specify three things for each partial: which rows it covers, whether it is exact or only a bound, and what kind of partial state it is.

Once a stored statistic is a partial computed earlier, the difference between metadata and execution comes down to when the work was done. The merge is the same whether (30, 2) was computed at write time or during the query. A statistic is a memoized aggregate.

That changes what a new statistic costs. If our old files store only MIN and MAX, we can start writing (sum, count) into new files and use it for AVG queries straight away. The old files still require a scan, but their partials merge with the stored ones. Nothing about the merge requires that a partial be written by the file writer, so a partial computed by one query could be kept for the next.

A new kind of statistic is a new aggregate function. That's how Vortex stores bounding boxes for spatial data. The partial for a two-dimensional bounding box consists of four values: xmin, ymin, xmax, ymax. To merge two boxes, we take the smaller xmin and ymin and the larger xmax and ymax:

We already use these boxes to skip zones that can't match a spatial predicate. When merged, they also answer a query for the bounding box of every geometry in zones A and B without decoding any geometry. If a zone has no stored box, Vortex computes one from the encoded geometries, using kernels for specific encodings to compute over compressed data where possible. One query can take a stored box from one zone and a computed box from the next, and an engine that accepts the partials can merge them without knowing the layouts or encodings behind them.

Memoized partials needn't be exact. Merging exact COUNT(DISTINCT id) results would require keeping each zone's set of IDs, which could be as large as the column. A HyperLogLog sketch is a small, mergeable partial for an approximate count, so approximate joins exact and bound as something an engine has to track.

Which partials are worth storing, at what granularity, and which of them a query engine can use are the questions we're exploring in Vortex as a file format.