Skip to main content

Inside GPU execution

The engine runs a compiled schedule for each native fragment, coordinated from a worker thread in the device's GPU execution pool (Runtimes and threads). Operators execute GPU kernels on decoded columns in VRAM, retaining intermediate results until downstream consumers finish.

CUDA streams and dependencies​

CudaStream identifies an ordered queue of GPU work. Work on separate streams requires explicit completion dependencies before exchanging data.

Default inline chunk executionDefault chunk execution with source_work_per_attempt set to one. The coordinator submits chunk work on the root CUDA stream and synchronizes that stream before accepting each chunk's output. Partial states are merged on the root stream under the query's grant.CPU · coordinator on the GPU execution runtimeGPU · root CUDA stream · one query grantsubmitsynchronize, then accept outputSubmit and accept each chunkEnqueue work on the root CUDA streamWait in cudaStreamSynchronize before accepting the outputChunk 1Decode input columnsCompute partial stateChunk 2Decode input columnsCompute partial stateMerge partialsCombine completed statesProduce the grouped result
The default chunk dispatcher runs inline on the coordinator. It accepts each chunk's output after synchronizing its execution stream. This drawing follows chunk work; source prefetch and RAPIDS internal streams are omitted.

The coordinator runs as a QueryExecution job on one pool worker, executing root tasks on the attempt's root stream. Expanded chunk work (aggregate chunks, join probes, and fused top-k candidates) executes on blocking-lane jobs. An execution lane consists of one auxiliary permit, one leased CUDA stream, and a derived context. Its width is bounded by source_work_per_attempt, available permits, and pending work items; runs allocated fewer than two lanes execute inline on the coordinator. At the default source_work_per_attempt = 1, chunk work executes inline. The coordinator merges lane outputs in dispatch order under the required SQL semantics; DataFusion CPU partitions do not dictate lane allocation or stream width. A lane's stream lease remains held for the entire run, while chunk jobs may execute on any blocking-lane thread.

Related terms distinguish lane scopes:

  • An attempt lane (lane_count, which equals max_active_attempts) is the identity slot an admitted attempt holds for its lifetime (Capacity and allocation).
  • An execution lane is the permit, stream, and context described above.
  • Chunk lane work is a blocking-lane job executing on an execution lane.

Cross-stream dependencies use completion fences such as CUDA events. Buffer ownership lasts through final consumption, and all lanes allocate within one query memory cap. An aggregate's source chunks on separate lanes are live at the same time, so its chunk target divides the grant by the regions one chunk keeps live times the run's lane width; two lanes halve the target a serial run plans. Within composed island executions, shared execution scopes coordinate multi-reader lifecycles, retaining resident handles until the final reader completes.

Partitioned joins​

A partitioned hash join splits its build side by key hash into one frame per partition and splits each probe piece the same way, so each pair holds only matching keys. When the probe side streams in, the join builds the hash table of each build partition at the first probe piece and probes it with every later piece. The attempt's kept tables stay within the share of its grant described in kept join tables. A partition outside that share builds one table per batch of probe input instead of one per piece. The join copies that partition's rows out of each probe piece and holds them. When the rows held by the attempt's joins would pass one partitioned-join target, the join concatenates the rows of its partition holding the most bytes, joins them against a table built for that batch, and drops the table. It repeats until the rest fits, and joins the rows still held when the probe input ends. A probe of a single piece keeps no table and holds no rows. A retry that repartitions a pair builds its own tables for the smaller pairs.

A build side that streams in chunk by chunk decides after each chunk whether to partition. Plan row and byte estimates bound the build from above and may far exceed it, for example after a LIKE filter over a large dimension. The build commits to partitions once its counted rows or bytes reach the large-build thresholds of KeyedHashJoinStrategyOptions. It also commits when its estimate allows a large build and the whole chunks it holds reach the quarter of the grant that also bounds kept join tables. From then on each chunk is hash-partitioned as it arrives. A build that never commits is concatenated, and its strategy is chosen from the counted rows and bytes alone, where a large estimate no longer counts as large-build evidence. A build of medium size whose probe is large still ends partitioned, after that one concatenation.

The coordinator builds the kept tables on the root stream before probe chunks reach execution lanes. Lanes probe them concurrently on their own streams, which only read the table. A lane returns its rows of partitions without a kept table to the coordinator, which holds them and joins their batches on the root stream. The outputs of one probe piece's partitions, with the batches its rows pushed out, are concatenated into one frame. The next operator therefore receives at most one frame per probe piece, none for a piece whose rows were all held, and one for the batches joined at the end of the input. That concatenation holds the piece's output twice at its peak.

Runtime dynamic filters​

The compiled executor derives dynamic filters from the producer's complete join build or aggregate group-key state. max_build_rows bounds the producer rows admitted to this work. Exact membership payloads retain distinct non-null keys in the attempt's query memory domain and share them across consumers and chunks. Empty and dense range payloads retain no key column. cuDF orders membership reads against the retained producer stream and retains completion dependencies until consumption ends.

Route discovery places a consumer only on a join input whose unmatched rows the join discards. Inner and semi joins discard them on both inputs. A left outer, anti or mark join discards them on its right input alone; each left row reaches the output, or the mark column, whether it matches or not. A null-aware anti join reads the nulls and the emptiness of its right input and takes no route on either input. Only single-key joins under NullsDoNotMatch produce a key route. Native execution builds the right input of every keyed left outer and mark join. Build-side selection may move a keyed semi or anti join's build to its left input, and the route then reduces the right input. A join built on its left input keeps one match flag per left row: each right chunk marks the left rows it matches and is released, and the output is one gather of the marked rows for a semi join or the unmarked rows for an anti join. The join retains no right chunk after marking it. A null-aware anti join always builds its right input. A key route at a left outer or mark join, or at an anti join that builds its right input, would consume the left input and is rejected as UnsupportedJoinShape.

Integer bounds and distinct counts classify empty, dense range, small-IN-list and larger exact keysets without copying the key column to the host. Resident exact payload metadata carries the existing kind, count and range diagnostics; host predicate helpers cannot apply that payload. A source route requiring reader or row-group pruning downloads the distinct keys, retains the device keyset for post-decode membership, and preserves the source pruning policy. The large reader-only producer policy continues to derive a scalar range. Producer key limits and consumer type bounds follow native lowering contracts.

Relational data into GPU algorithms​

SQL table functions integrate cuGraph, cuVS, and cuML within queries. cuDF columns supply input relations, and algorithm output columns feed downstream relational operators. GPU table functions details buffer ownership, zero-copy interop, and layout requirements across RAPIDS libraries.

Result egress and host synchronization​

When a query fragment completes its relational and algorithmic stages, execution reaches its terminal sink (ResultSink::HostArrow or a deferred host batch cursor). The engine transfers the output column buffers from VRAM to host memory, packaging them into Arrow RecordBatch instances for DataFusion.

Result egress and host synchronizationResult egress for one native fragment. On the host runtime, the first poll of the DataFusion result stream runs admission and the native schedule. Each chunk then takes one GPU pool job on algeon-gpu-{d}. The job sizes the chunk to at most 4 MiB of Arrow buffers and reserves that estimate in DataFusion's MemoryPool, enqueues cudaMemcpyAsync on the execution stream after the upstream kernels, waits in cudaStreamSynchronize, wraps the host buffers as Arrow arrays and shrinks the reservation to the batch size. The stream yields the RecordBatch, the next poll moves the next chunk, and the last chunk ends the stream.Host runtime · DataFusion result streamGPU pool · one algeon-gpu-{d} job per chunkGPU VRAM · execution streamnext pollpool jobyield batchenqueuecopy completeDataFusion pollFirst poll: admission, native scheduleEach chunk: one pool jobRecordBatch to the callerHolds one chunk reservationLast chunk ends the stream1. Size and reserveEstimate ≤ 4 MiB of Arrowtry_grow in MemoryPool2. Copy and waitEnqueue cudaMemcpyAsynccudaStreamSynchronize3. Build the batchWrap host buffers as ArrowShrink reservation to sizeResult columnsWritten by upstream kernelsDevice-to-host copyStarts after upstream kernelsCrosses PCIe into host buffers
Each chunk is one GPU pool job. The job reserves its size estimate before the copy, enqueues the copy on the execution stream and holds its worker in cudaStreamSynchronize until the copy completes. The result stream keeps one chunk reservation at a time.

The device-to-host copy is queued on the execution stream after the upstream kernels and starts once they complete.

The host runtime polls the DataFusion stream and awaits each pool job without blocking a host worker. The first poll runs admission and the fragment's schedule. Each chunk then takes one pool job, which sizes a row range of the result to at most 4 MiB of Arrow buffers, reserves that estimate from DataFusion's MemoryPool (one live reservation per stream), enqueues the device-to-host copy and waits in cudaStreamSynchronize. Sizing reads only the string and list offsets of the range, and the copy reads each column through a view of it instead of first copying the range on the device. The wait holds the pool worker until the copy completes. Algeon sets no CUDA device scheduling flag. Under the runtime default, cudaDeviceScheduleAuto, the waiting thread spins on its CPU core while the process has no more active CUDA contexts than logical processors, and yields to other threads otherwise.

Once cudaStreamSynchronize returns, the worker hands fixed-width values and packed null masks to Arrow arrays without another copy. Boolean values are packed to bits, Decimal32 and Decimal64 columns are widened when the schema asks for Decimal128, and string columns are rebuilt row by row. The last chunk ends the result stream, and the attempt releases its grant once no fragment is running.