Error Handling Design

Every Algeon-owned adapter failure carries a stable machine-readable code, a coarse kind, and a status that says whether the condition is permanent, temporary, or persistent, in addition to a human-readable message. An application can find that identity inside a DataFusionError that wraps a DataFusion adapter error, or read it from gRPC metadata over Flight SQL, and should make decisions from those typed fields, never from formatted error text. Algeon keeps an error typed at the layer that owns the failure and converts it only when crossing a real boundary. Ordinary DataFusion errors that carry no Algeon identity, and transport errors below the server, keep their own policies: nothing in this guide upgrades them to Algeon errors.

The typical native-execution path is:

cuDF / cuGraph / cuVS
  -> shared native QueryError taxonomy
  -> algeon_datafusion::error::Error
  -> DataFusionError::External
  -> embedded application or Flight SQL status

Normal native-planning declines do not enter this path. NotSupported and NotSelectedByCost are candidate outcomes in a PlanningReport, not error codes or DataFusionErrors, and the original DataFusion plan remains executable. A malformed plan, an unsatisfied final-plan requirement, or a runtime failure is an error.

Ownership by layer

Layer	Error type	Responsibility
query-planning	QueryError	Canonical DataFusion-independent error taxonomy plus IR, optimizer, capability, proof, and admission failures.
query-runtime	planning's QueryError	Attempt lifecycle, query service, device ledger, grant, cache, observation, and metric failures; it does not define another taxonomy.
query-engine	planning's QueryError	Native execution failures and the compatible public facade re-export.
DataFusion adapter	algeon_datafusion::error::Error	DataFusion planning/execution context and projection of native errors into the public adapter taxonomy.
DataFusion traits	DataFusionError	Framework boundary only. Algeon errors cross it as a typed External source.
DataFusion server module	ServerError / tonic::Status	Server lifecycle, client-safe messages, gRPC codes, metadata, and retained diagnostics under algeon_datafusion::server; the private server package only launches it.

Fix a missing classification at its owner. If a cuDF or cuGraph error lacks the typed information Algeon needs, extend the owning component crate instead of parsing its message in the adapter.

One identity, several projections

An adapter Error carries:

Field	Use
code	Most specific stable machine identity.
kind	Coarse grouping derived from code; useful for protocol and aggregate policy.
status	Coarse source condition derived from code: Permanent, Temporary, or Persistent.
operation	Stable dotted identifier for the operation that added this boundary context.
message	Human diagnostic text.
facts	Structured diagnostic context. Raw by default.
source	Original typed cause chain.

kind and status are mappings, not independent classifications. Add a new code first, then update the exhaustive mappings and contract tests.

Do not parse Display or Debug. Adapter Error::Display intentionally emits only its human message. Native QueryError formatting is for trusted in-process diagnostics only and can contain raw facts and source text.

Crossing DataFusion

From<Error> for DataFusionError performs one conversion:

DataFusionError::External(Box::new(algeon_error))

DataFusion may wrap that value in context, diagnostics, shared errors, or error collections. Use DataFusionErrorExt::find_algeon_error() rather than matching one External layer:

use datafusion::common::DataFusionError;
use algeon_datafusion::error::DataFusionErrorExt;

fn record_failure(error: &DataFusionError) {
    if let Some(algeon) = error.find_algeon_error() {
        tracing::warn!(
            code = algeon.code().as_str(),
            kind = algeon.kind().as_str(),
            status = algeon.status().as_str(),
            operation = algeon.operation(),
            "Algeon query failed"
        );
        return;
    }

    tracing::warn!(error = %error, "DataFusion query failed");
}

The search is deterministic depth-first pre-order and bounded by both depth and total node count. It follows standard source chains and visits every DataFusionError::Collection branch in order, returning the first Algeon error it reaches — not the most severe one. A None result only means no Algeon error was found within that budget; it does not prove that branches beyond the budget contain no Algeon identity. When there is no match, keep the application's existing DataFusion policy. There is deliberately no blanket conversion in the other direction.

Creating and wrapping errors

Create the error where the failed contract is known:

• Use the shared QueryError for DataFusion-independent planning, runtime, and engine behavior; add its canonical code and mappings in query-planning.
• Use adapter Error for DataFusion-specific planning, wrappers, and public integration behavior.
• A QueryError reaching the adapter is converted at that boundary (the constructor is contributor-internal, not a public integration entry point). The conversion maps the native code and preserves native code, kind, status, operation, message, non-colliding facts, and source. Contributors find the canonical construction rules in the owner rustdoc (crates/algeon-query-planning/src/error.rs, crates/algeon-datafusion/src/error.rs); integrators only need the public constructors and find_algeon_error() shown above.
• Use the existing typed lower-layer constructors and provenance at cuDF, cuGraph, or cuVS boundaries. Their mappings use enums and status, not message text.

Operations should be stable dotted identifiers. Messages explain the failure to a human. Facts carry machine-readable context needed for policy or diagnosis, and the source preserves the original cause. Do not create a second error taxonomy in a feature module or transport adapter.

Use the result type owned by the current layer: native Result<T> across planning, runtime, and engine, adapter Result<T> before DataFusion, and DataFusion's result type in DataFusion traits. Let ? perform the single adapter-to-DataFusion conversion.

Once native GPU execution starts, a failure closes the attempt; Algeon does not replay the retained CPU plan. Native-owned failures stay typed, while a DataFusion-owned boundary failure keeps its DataFusion identity.

Retry and recovery are evidence-based

ErrorStatus describes the source condition; it is not permission to retry:

Status	Meaning
Temporary	The condition may clear, but the error alone does not prove when or how to retry.
Persistent	A dependency failure is known or observed to continue; stop automatic retry and escalate.
Permanent	Do not automatically retry the same request. It normally needs different input, configuration, capability, or code; cancellation is also terminal.

There is no adapter Error::is_retryable(). Engine-internal bounded retries are owned by the exact execution operator and recognize only their explicit resource cases.

For a failure returned while admission is still being attempted, use QueryAdmissionOutcome::from_error (or from_error_with_queue_bounds). After a handle has been published, use from_execution_error; a post-admission OOM is an execution failure, not queue capacity that can be retroactively waited on. This projection is policy-neutral: the application still decides whether to queue, shed, or return a protocol error.

For an attempted query, the authoritative recovery signal is the sealed QueryAttemptReport::retry_advice. It is derived only after terminal outcome, GPU cleanup, capacity disposition, device health, native-fault isolation, and resource evidence are committed. Queue evidence can advise backoff; an exact reservation or allocation limit can require configuration change; proven device-local quarantine can allow another eligible device; unknown isolation can require backend recovery. Missing evidence remains indeterminate.

The engine never retries from this advice. A caller must satisfy its paired prerequisite and submit a new attempt. Cancellation, drop, and panic do not become retries, and transport delivery failure does not rewrite the sealed attempt outcome.

Recovering from an attempted query follows the attempt lifecycle, not the error value alone:

1. Classify the returned error at the correct boundary: QueryAdmissionOutcome::from_error while admission is still being attempted, from_execution_error after a handle has been published.
2. Retain the attempt completion the service published for the attempt.
3. Wait for the sealed QueryAttemptReport and read its retry_advice, derived only after terminal outcome, GPU cleanup, capacity disposition, device health, native-fault isolation, and resource evidence are committed. Satisfy the advice's paired prerequisite before submitting a new attempt.

Receiving an execution error is not proof that cleanup finished. The sealed report is the only signal that binds the recovery advice to completed cleanup.

Diagnostics and disclosure

Treat raw facts and source chains as trusted in-process diagnostics. They may contain query literals, object paths, identifiers, backend messages, or credentials.

• Adapter with_fact stays hidden from log and retained-diagnostic projections. with_log_safe_fact is an explicit opt-in for bounded operational classifications; sanitizing and truncating remain defense in depth.
• QueryError::to_json() includes raw native facts and source text. Do not use it as a client response.
• Sealed attempt reports project a bounded allowlist from the primary failure; they do not copy arbitrary messages or facts.
• Preserve the typed source for internal diagnosis unless the owning boundary deliberately omits it because the foreign text can expose a sensitive location.

Flight SQL boundary

The built-in server first looks for a typed adapter error. Ordinary DataFusion errors receive a server-private classification for gRPC presentation, but that does not give them a public Algeon identity for embedded callers.

The server maps cancellation, auth, not-found, invalid/planning, unsupported, and resource kinds to their matching gRPC codes. Execution, dependency, invariant, and internal failures use INTERNAL. Client messages are narrowed for source, dependency, invariant, and internal failures; logs and retained diagnostics keep the separate structured fields and bounded safe projection.

One code-specific exception: a non-converged native algorithm (algorithm_did_not_converge) returns FAILED_PRECONDITION with an actionable message naming the algorithm, the iterations reached, and the configured limits, plus matching algorithm, iterations, max_iterations, and epsilon metadata. It is a parameter problem to fix, not a condition to blindly retry.

Every status created from an adapter Error includes these metadata fields:

• x-algeon-error-code
• x-algeon-error-kind
• x-algeon-error-status
• x-algeon-error-operation

The server also projects only unsupported_column, unsupported_column_type, unsupported_column_role, predicate_role, reason, required_option, algorithm, iterations, max_iterations, epsilon, and native_message when present. native_message is projected only for client-actionable validation kinds (cancellation, not-found, permission, auth, planning, invalid-input, unsupported): internal, dependency, invariant, and execution failures never carry raw native text in metadata, however their status message reads. Query and correlation identifiers are added by the statement boundary. The source chain is retained locally, not emitted as metadata. Client-visible fact selection is independent of with_log_safe_fact. Attach native_message as a trusted in-process diagnostic whose content is already safe to expose on the allowlisted validation paths; the server gate decides the projection, not the producer.

Metadata values must be transmittable ASCII and are capped at 256 bytes per value with a ... marker. Values that cannot cross the wire (for example non-ASCII schema details) are omitted and listed under x-algeon-omitted-facts; the full diagnosis stays available in the status message and query stats.

Strict unsupported-type rejection

With the NoDataFusionCpu final-plan requirement, an unsupported native column cannot remain on the executable DataFusion plan. The outer adapter error code is final_plan_requirement_unsatisfied, and Flight returns InvalidArgument. For an unsupported list projection, the status message itself identifies the first blocking field in schema order, its exact diagnostic Arrow type, and its stable role:

GPU execution does not support projected column `countDetail` with Arrow type `List(Decimal128(10, 2))`

That status also carries the matching structured metadata:

Metadata key	Example value
x-algeon-reason	unsupported_type
x-algeon-unsupported-column	countDetail
x-algeon-unsupported-column-type	List(Decimal128(10, 2))
x-algeon-unsupported-column-role	projected

The terminal GetQueryStats structured error retains the same facts as fact.reason, fact.unsupported_column, fact.unsupported_column_type, and fact.unsupported_column_role. Flight metadata and query stats are additional machine-readable projections; they do not replace the actionable status message. The existing allowlist and masking rules still apply, so raw facts and the source chain are not exposed.

The four projections do not carry identical bytes. The status message keeps the full actionable text, including non-ASCII column names. Metadata carries only ASCII values capped at 256 bytes with a ... marker; a non-ASCII column name is omitted from metadata and listed under x-algeon-omitted-facts. Query stats retain the structured facts without the wire encoding limits. The column name never enters operational logs or Debug output. Compare projections per surface — message, metadata, logs, query stats — rather than asserting four equal strings.

Protocol policy outside the built-in server belongs to the application. The embedded REST example intentionally maps typed errors before applying its ordinary DataFusion fallback, and does not expose raw facts wholesale.

Extending the contract

When adding an error:

1. Add the most specific code at the owning layer.
2. Update code-to-kind and code-to-status mappings; do not infer either from the variant name.
3. Map native codes exhaustively at the adapter boundary while preserving the original native classification.
4. Decide explicitly whether any fact is raw, log-safe, attempt-report-safe, or Flight-visible. These are separate disclosure choices.
5. Extend the existing error, admission, attempt-report, or Flight contract tests. A useful regression test must fail when the mapping or boundary behavior is reverted.

The canonical implementation points are crates/algeon-datafusion/src/error.rs, crates/algeon-datafusion/src/error/code.rs, crates/algeon-query-planning/src/error.rs, crates/algeon-query-runtime/src/observability/attempt_report.rs, crates/algeon-query-engine/src/exec/, and crates/algeon-datafusion/src/server/error.rs.