Benchmarks
The MerRec queries compare Algeon with CPU pipelines over the 1.27-billion-event dataset. The TPC-H suite at scale factors 100 and 500 and the TPC-DS suite at scale factor 100 compare the CPU and GPU paths, and the server load runs drive concurrent TPC-H streams through Arrow Flight SQL.
GPU algorithms
The GPU algorithms behind cugraph_*, cuvs_* and cuml_* UDTFs run NVIDIA's native
libraries inside DataFusion physical plans alongside relational operators. NVIDIA
publishes upstream kernel benchmarks for cuGraph,
cuVS,
and cuML.
MerRec queries against CPU pipelines
Each query reads the full six-month MerRec dataset, 1,268,092,738 events and 82,614,392 items on local NVMe, with the files in the page cache. Each CPU pipeline runs the relational steps in DuckDB 1.5.6 on 32 threads and the algorithm in the named library. Both sides ran on one host: an Intel Xeon Platinum 8559C with 32 vCPUs, 256 GiB RAM and an NVIDIA RTX PRO 6000 with 96 GB.
The table lists every algorithm query in the MerRec guide. Algeon times are
medians of runs 2 to 4 over Flight SQL with a release binary and no resident
GPU cache (cache_cap_bytes = 0); CPU times are medians of three runs, none
excluded. The data file keeps the first GPU run of each query, which also
pays process start-up costs.
Stage breakdown across the eight queries
- Barbie tie-ins builds two undirected graphs from 23 million distinct
user-brand pairs and runs personalized and global PageRank on each. The CPU
pipeline takes 2.3 s in the DuckDB join, 13.0 s renumbering vertices and
building the igraph graphs, and 7.7 s in the four PageRank calls. NetworkX
3.7 takes 158 s once its tolerance matches cuGraph's absolute L1 tolerance
(
tol = 1e-5 / N); its defaulttol=1e-6is a mean per-vertex error and stops after two iterations on 2.1 million vertices. - Session audit finds weakly connected components over 39.8 million
user-session edges. DuckDB deduplication and renumbering take 9.3 s and
scipy
connected_componentstakes 3.0 s. - Audience funnel combines graph affinity, taste vectors and purchase propensity in one plan. The CPU pipeline passes intermediate tables between igraph, FAISS and XGBoost in host memory; graph construction and personalized PageRank take 14 s. Shoppers tied at the 20th-neighbor distance fall on either side of the cutoff, and the lookalike count varies between 19,096 and 19,104.
- Propensity model takes 14.7 s in DuckDB to aggregate features over the events before September, and that aggregation takes most of the 6.60 s on the GPU. Forest inference takes 1.1 s in XGBoost and under 0.01 s in cuML. Tied scores at the shortlist cutoff give 16,087 buyers on the GPU and 16,090 on the CPU.
- Brand twins compares the fan sets of the 2,000 largest brands by Jaccard similarity. The DuckDB self-join takes 2.9 s to collect the fan sets and 3.4 s to join them.
- Pokemon lookalikes builds a 64-dimensional taste vector for each of 2.1 million shoppers in SQL, then searches 20 neighbors of each seed buyer. DuckDB takes 19.5 s to build the vectors. The search takes 6.4 s in FAISS and 0.03 s in cuVS.
- Shopping worlds groups the 5,000 largest brands into worlds with Leiden over a Jaccard similarity graph. DuckDB takes 8.3 s for the fan sets and Jaccard, and igraph 0.5 s for Leiden. The two Leiden implementations return different partitions; the result lists are not compared row by row.
- World bridges scores 12.5 million brand pairs with Jaccard similarity, splits the similarity graph into worlds with Leiden and ranks bridge brands by exact betweenness. The DuckDB self-join takes 6.2 s and NetworKit's betweenness 1.7 s.
Measurement details and reproduction
The GPU runs were measured on
2026-11-05 at revision
6932478ae.
The GPU server runs a release build for
x86-64-v3 on driver
595.99.02 and CUDA
13.2, configured by
server.toml.
GPU and FAISS medians exclude the first of four runs. The 100,000-query FAISS
case ran five times and also excludes its second run, which overlapped a
compile job on the same host. FAISS used 8 threads,
chosen in a tuning run against 16 and 32, with NumPy 2.5.3 and PyArrow 25.0.1. The
algorithm scale guide
documents SQL generation, CPU tuning and result validation, and the
data file
keeps every run.
Exact kNN over 2.1 million vectors
Each search finds the 10 nearest of the
2,080,74364-dimensional Float32 taste vectors
in mr_taste64 for the first 100 to 100,000 shoppers. The distance column
counts query rows times taste rows.
Algeon runs the search through the cuvs_brute_force_knn table function in
SQL. The Algeon time covers the whole query, from the Parquet scan on local
NVMe to the last result batch over Flight SQL. The CPU time is FAISS 1.15.1
IndexFlatL2 on 8 threads,
timed from Parquet load to the Arrow result.
GPU and FAISS neighbor distances agree within 1e-6 after both are recomputed in Float64. Neighbors at tied distances may differ by ID.
Item-user PageRank
The query below builds a weighted, undirected graph from six months of item
views. It scans the 1,268,092,738 events, groups the item views by
(item_id, user_id) for the interaction weights, and runs cugraph_pagerank
over the resulting 87,055,073 vertices and
1,682,503,472 directed edges.
PageRank converges in 75 iterations at the default
alpha => 0.85 and epsilon => 1e-5.
SELECT vertex, value
FROM cugraph_pagerank(
edges => (
SELECT item_id AS src, -(user_id + 1) AS dst,
CAST(COUNT(*) AS DOUBLE) AS weight
FROM merrec_events
WHERE event_type = 'item_view'
GROUP BY item_id, user_id
),
directed => false
)
WHERE vertex >= 0
ORDER BY value DESC, vertex
LIMIT 20;
The whole query, edge preparation included, takes a median of
20.77 s; the cuGraph PageRank call
accounts for 7.94 s of it. The aggregated
edges and the graph stay in device memory between the GROUP BY and the
PageRank call.
Relational path
The TPC-H suite on three datasets and the TPC-DS suite at scale factor 100,
CPU path against GPU path, on an AWS EC2 g7e.8xlarge.
Both suites were measured on 2026-11-13 at revision
1a7c85a7f, the build of the server load runs below.
These runs are derived from the TPC-H and TPC-DS benchmarks. They are not
audited TPC results and are not comparable to published TPC results. TPC-H
and TPC-DS are trademarks of the Transaction Processing Performance Council.
The query text is the DuckDB rendering of the TPC specifications (MIT), with
Q11's FRACTION substituted as the specification defines it, 0.0001 / SF.
Each query ran 1 warm-up and 5 measured iterations on each
path, and the tables and charts report the median measured iteration.
A measured time starts when the query text is passed to SessionContext::sql
and ends when the last result batch has been collected. It covers SQL parsing,
logical optimization, physical planning and execution; DataFusion's dfbench
times its TPC-H runs over the same span. Session construction and table
registration run before the timer starts.
- CPU: Intel Xeon Platinum 8559C, 32 vCPU.
- Memory: 256 GiB.
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB.
- Storage: local NVMe Parquet files. GPU reads use KvikIO compatibility mode with Algeon's default 64 I/O threads, without GPUDirect Storage. CPU reads use DataFusion's Arrow Parquet reader.
- DataFusion version
55.1.
Both paths read the Parquet files through the OS page cache. Before each
dataset ran, the other datasets' files were evicted from the cache, and the
resident share of the dataset's own files, by bytes, was recorded:
SF100 sorted 64% before the first query and 64% after the last; SF100 dbgen order 0.0% before the first query and 65% after the last; SF500 sorted 0.0% before the first query and 52% after the last;
TPC-DS 0.0% before the first query and 68% after the last. The SF500 files total
166.9 GiB on a host with 256 GiB of memory,
which also holds the CPU path's working set. No cold-storage run was measured.
The GPU keeps no data between queries (cache_cap_bytes = 0), so each measured
iteration reads the column pages again, copies them to the device and decodes
them. Each TPC-H query ran in its own process and built a new session for every
iteration. The TPC-DS CPU and GPU runs each reuse one DataFusion session for all
99 queries, so planning on either path can find a file's footer in the
session's metadata cache from the warm-up; the GPU path's cuDF reader still
parses the footer of each file it reads once per query. Caches along the
path describes what each layer holds.
TPC-H
TPC-H ran on three datasets. SF100 sorted and SF500 sorted come from
fixture/tpc-h/generate.py:
DuckDB sorts each table by its pruning keys, l_shipdate first for
lineitem, and Spark writes one Parquet file per table with parquet-java.
SF100 dbgen order (unsorted) holds the same SF100 rows through the same
writer in tpchgen's generation order, which is primary keys ascending and not
sorted by date. The sections below call it dbgen order. The TPC-DS files are
written by DuckDB.
The benchmark compares the measured CPU and GPU outputs of each query value
by value, with the ordering rules of the TPC-DS check below.
Floating-point columns accept an absolute and relative error of
1e-8;
every other column must match exactly. The table counts the queries that ran
on the GPU path and the queries whose output matched the CPU path.
Times sum the median time of each query, planning included. The planning share is the summed planning time divided by the summed time.
- SF100 sorted
- SF100 dbgen order
- SF500 sorted
- Result: CPU 50.3s, GPU 9.3s, CPU / GPU 5.4x, planning 0.2% of CPU time and 1.5% of GPU time, largest task memory peak 42.1 GiB.
- Per-query runtime
- Speedup
- Memory pressure
- Per-query results
Median of 5 measured iterations after 1 warm-up; each time covers SQL planning and execution. Parquet files 64% resident in the OS page cache before the first query and 64% after the last; no GPU resident cache.
CPU time divided by GPU time, both including planning.
Largest cuDF allocator peak that one compiled task reported, per query.
22 TPC-H queries. Times include planning; the planning columns show the share.
TPC-H Parquet layout
The values below were measured from the file footers, offset indexes and page
headers by fixture/tpc/describe_layout.py, which also checks them against
each directory's _generation.json.
No dataset is partitioned. Each table is one Parquet file in one directory, with no Hive partition keys. The generator sets no dictionary option, so parquet-java's default applies, and the table counts the column chunks that carry a dictionary page.
Both SF100 datasets returned the same output fingerprint and row count for all
22 queries; the data generator refuses to
publish otherwise. In lineitem, the row-group value ranges are disjoint on
l_linestatus, l_shipdate in the sorted files and on
l_orderkey in dbgen order. Against the sorted
files, the dbgen-order files took
in total on the CPU path (50.3 s and 51.7 s) and
33% longeron the GPU path (9.3 s and 12.3 s).
TPC-DS SF100
All 99 queries ran on the GPU path in NativeRequired mode, and every result matched the CPU path.
- Result: CPU 174.6s, GPU 47.8s across 99/99 successful native queries, CPU / GPU 3.65x, largest task memory peak 41.4 GiB.
- Per-query runtime
- Speedup
- Memory pressure
- Per-query results
Median of 5 measured iterations after 1 warm-up; each time covers SQL planning and execution. Parquet files 0.0% resident in the OS page cache before the first query and 68% after the last; no GPU resident cache. The Y-axis skips 15–60s.
Scroll horizontally to inspect all 99 queries.
CPU time divided by successful GPU time, sorted ascending. Green is GPU faster; red is GPU slower.
Largest cuDF allocator peak that one compiled task reported, per query.
99 TPC-DS queries. GPU faster on 62; slower on 37.
GPU execution was faster for 62 queries and CPU for 37. The largest absolute CPU lead was Q44 (0.62s CPU, 1.19s GPU). The CPU / GPU ratio sums the median query times under the CPU configuration below.
TPC-DS Parquet layout
The files come from
fixture/tpc-ds/generate.py.
fixture/tpc/describe_layout.py measured the values below the same way as for TPC-H.
Each TPC-DS table is one Parquet file in one directory, with no Hive partition keys.
DataFusion CPU baseline configuration
The TPC-DS CPU baseline ran DataFusion 55.1.0 with these settings. Settings not listed keep their defaults. The TPC-H runs above used 8,192-row batches.
Reproduction and TPC-DS parity verification
Both paths use release builds with target-cpu=x86-64-v3. The TPC-DS runs use
the same 24 Parquet files,
totalling 41,372,641,702 bytes and
959,037,905 rows. Each query's 1 warm-up
precedes its 5 measured iterations. wall_ms of a measured row covers SQL
planning and execution through result collection, including reads and GPU
result transfer; planning_ms and execution_ms split it.
GPU runs use the sf100_gpu_throughput profile with
parquet_throughput source planning, an explicit 32 GiB
max_source_chunk_bytes target and at most 1,024 row groups per chunk.
The byte target groups source data; the query's separate
memory grant limits
device allocations. This benchmark profile sets its row-group cap directly.
The server's default chunk policy
derives its byte target from the admitted grant. Statistical aggregates use
gpu_native_tolerant.
The datasets were generated with these recorded commands:
python3 fixture/tpc-h/generate.py --scale 100 --out /mnt/algeon-nvme/algeon/fixture/tpch/sf100-spark/parquet
python3 fixture/tpc-h/generate.py --scale 100 --unsorted --out /mnt/algeon-nvme/algeon/fixture/tpch/sf100-unsorted/parquet
python3 fixture/tpc-h/generate.py --scale 500 --out /mnt/algeon-nvme/algeon/fixture/tpch/sf500-spark/parquet
scripts/dev/relational_benchmarks_driver.py
measured every number in the TPC-H, TPC-DS and server load sections from one
build. Its module docstring lists the prerequisites, the host-facts file and
the stage directories.
python3 scripts/dev/relational_benchmarks_driver.py \
--out-dir target/relational-benchmarks/RUN \
--tpch-sf100-sorted "$TPCH_SF100_SORTED" \
--tpch-sf100-dbgen-order "$TPCH_SF100_DBGEN_ORDER" \
--tpch-sf500-sorted "$TPCH_SF500_SORTED" \
--tpcds-sf100 "$TPCDS_SF100" \
--host-facts host-facts.json run-all
run-all runs the stages build, tpch sf100_sorted, tpch sf100_dbgen_order,
tpcds, tpch sf500_sorted, server-load and website in order. Each stage
writes DONE or STOP with its reasons. The website stage writes the three
data files only when the worktree stayed clean and the binaries unchanged, every
TPC-H row ran on the GPU path with matched results and 5 samples, the two SF100
datasets returned identical outputs, the TPC-DS parity report has no mismatch,
every measured TPC-DS iteration succeeded, describe_layout found no difference
between a manifest and its footers, and no server-load run stopped.
The tpcds stage sets these CPU and GPU settings and runs the parity report
before the CPU and GPU timing runs:
export ALGEON_TPCDS_CPU_TARGET_PARTITIONS=32
export ALGEON_TPCDS_CPU_BATCH_SIZE=32768
export ALGEON_TPCDS_NATIVE_PROFILE=sf100_gpu_throughput
export ALGEON_TPCDS_NATIVE_STATISTICAL_AGGREGATE_EXECUTION_MODE=gpu_native_tolerant
native_tpcds_parity_report checks the result values. It runs each query on
both paths in one process, with the CPU side on the same partition and batch
settings, and compares every value. A top-level ORDER BY preserves the
sequence of sorting keys. Rows may exchange positions only within contiguous
groups equal on every key; unresolved keys retain emitted row order. Queries
without top-level ordering use multiset comparison, preserving duplicate
counts. Tie groups require exact key equality. Floating-point
columns, which are the GPU variance and standard deviation outputs under
gpu_native_tolerant, accept an absolute and relative error of
1e-8;
every other column must match exactly. The result-values column in the table
names the mode used for each query.
The generator refuses a parity TSV whose dataset root, query set, table sizes or CPU settings differ from the timing run, a timing TSV from a bench binary without the planning boundary, and a measured query without 5 rows:
python3 website/scripts/generate_tpcds_sf100_benchmark_ts.py \
--cpu-tsv RUN/tpcds/cpu.tsv --gpu-tsv RUN/tpcds/gpu.tsv \
--parity-tsv RUN/tpcds/parity.tsv --environment RUN/environment.json \
--layout RUN/tpcds/layout.json --page-cache RUN/tpcds/page-cache.json \
--out website/src/data/tpcdsSf100Benchmark.ts
Concurrent TPC-H streams through Algeon server
The runs above time one query at a time inside a benchmark process. In this
run, 1 to 16 concurrent Flight SQL clients ran the 22 TPC-H SF100 queries
against Algeon server on the same g7e.8xlarge. Each client ran one stream: the 22
queries in its own seeded order, twice, with the first pass as warm-up. The
GPU server ran every query in native_required mode. The CPU path is the same
binary with execution_mode = "functions_only", which plans every query
through DataFusion on the CPU. Queries that call GPU UDTFs go through the same
admission queue and memory caps; this run uses relational SQL only.
- Throughput
- p95 latency
- All values
Completed queries per second over the measured pass, median of 3 runs. Parquet in the OS page cache before each run.
95th percentile query latency in seconds, from request to the last result batch.
Median of 3 runs with the range in parentheses.
With the server's default settings, the GPU peaked at 2.97 queries per second (10,704 per hour) at 16 streams. Throughput was within 2% of that peak from 2 streams up, while median latency rose from 0.5 s to 5.4 s at 16 streams. The CPU path peaked at 0.56 queries per second, and the GPU default completed 5.3 times as many.
The admission setting max_active_attempts sets where throughput levels off.
The server runs up to that many queries at once and queues later submissions
in FIFO order. Each admitted query receives a
fixed memory cap equal to device managed capacity divided by max_active_attempts:
94.4 GiB at concurrency limit 1,
47.2 GiB at the default limit 2, and
31.5 GiB at limit 3 on this GPU. Concurrency limit 1
stayed near 2.45 queries per second from
2 streams up. Concurrency limit 3 reached
3.45,
16%
above the default. All 22 queries completed under each of the three caps. A
query that allocates more than its cap fails with query_memory_limit_exceeded
while the other admitted queries continue. Raising max_active_attempts raises
throughput for queries that fit the smaller cap and lowers the largest query the
server can admit. Memory and query lifetime
describes the queue, the caps and how the allocator enforces them.
Server overload and admission queue behavior
In these overload runs, the client submitted queries to the default configuration at sustained rates from 1.0× to 2.0× of rated capacity (2.97 queries per second) for five minutes after 30 seconds of warm-up. The client submitted each query on schedule whether or not earlier queries had finished.
Completed throughput held between
2.95 and
3.03 queries per
second from 1.0× to 2.0× offered load. The server rejected the excess with
service_overloaded. With all 64 queue positions
occupied, a new query was rejected after a median of 11 ms.
max_queue_wait_ms (30 s) in these runs.
At 1.5× load and above, admitted queries waited in the queue before execution, and p95 completion latency reached 25.7 s. Every admitted query result matched the CPU reference output. After each run, the server reported no active queries and no remaining memory grants.
Server load measurement methodology and reproduction
- Client and server ran on one machine over loopback.
tasksetpinned the client to 4 vCPUs and the server to the other 28. The Parquet files were fully resident in the page cache before each run. - The runs were measured on 2026-11-13 at
revision
1a7c85a7f. The server runs a release build configured bytpch-sf100-server.toml, which sets onlynative_requiredand the table registrations. The run's copy registers the Spark-written files of the TPC-H SF100 run. A=1 and A=3 add one[[admission.device_profiles]]entry. The CPU path ran withdatafusion.execution.target_partitions = 32anddatafusion.execution.batch_size = 32768. - Each point is the median of 3 runs, with
a fresh server for each run. Throughput counts queries completed in the
measured pass, from the moment every stream finished warm-up to the last
completion. Latency runs from
GetFlightInfoto the last result batch. - Every result was compared with references recorded by the CPU path. 22,919 GPU results matched, warm-up passes included.
- This run is derived from TPC-H and is not a TPC-H Throughput Test. It runs no refresh functions, orders each stream from seed 20261013 and reports no TPC metric.
- The stress runbook
documents the server, reference, streams and overload commands for
algeon_tools flight-sql-stress. scripts/dev/server_load_driver.pyran this matrix; its module docstring lists the prerequisites and the commands that turn a run directory into this page's data.