HDBSCAN
UDTF: cuml_hdbscan
Official cuML reference: Python API
cuML HDBSCAN fits density-based clusters and returns query-local labels and membership probabilities.
Quickstart
Register the relations referenced by the parenthesized SELECT clauses below. For metadata validation, the descriptor names input_vectors.
SELECT id, cluster_id, probability
FROM cuml_hdbscan(
input => (SELECT id, d0, d1 FROM t),
min_cluster_size => 5
)
ORDER BY row_ordinal;
Inputs
Each relation argument is a parenthesized SELECT subquery. Metadata validation resolves a registered table or view for the same role without scanning its rows. See ML Inputs for the ID, feature element, Float64 narrowing, null, finite-value, and runtime-dimension contract.
Vector element types
Feature values must be Float32 or Float64.
Arguments and options
Scalar SQL arguments
Vector binding shapes
Each relation subquery projects an id and either feature columns in dimension order or one list column, with the element types listed under Vector element types. Nullable field types are accepted. A null ID, feature value, list row, or list element fails the query.
The list-column shape excludes other feature columns. See ML Inputs for the allowed list containers and runtime checks.
Output
The output schema depends on the call. Run gpu_validate_call against registered relations to inspect the schema that the validated ID types and literal options produce.
Examples
This example uses the EMBER2024 demo dataset.
Which missed Win32 files should an analyst group for batch review?
The 3,225 Win32 samples in this cohort initially escaped antivirus detection.
This query projects each sample's
512 byte and byte-entropy histogram features (f7 through f518) to 16 PCA
components, then fits HDBSCAN with min_cluster_size = 30 and min_samples = 10. The family labels do not enter either fit; SQL joins them after clustering
to summarize the resulting groups.
The capture contains seven clusters and 508 rows assigned to noise (cluster_id = -1). HDBSCAN keeps those noise rows in the output. Cluster IDs are local to
this fit. The 930-file cluster contains 736 snackarcin files, 193
opensupdater files, and one file without a family label.
All challenge files are positive examples, so this cohort cannot measure a
false-positive rate.
WITH reduced AS (
SELECT id, pc_0, pc_1, pc_2, pc_3, pc_4, pc_5, pc_6, pc_7,
pc_8, pc_9, pc_10, pc_11, pc_12, pc_13, pc_14, pc_15
FROM cuvs_pca(
input => (
SELECT v.id, v.histogram AS vector
FROM ember_research_vectors_challenge v
JOIN ember_metadata_challenge m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
n_components => 16
)
), assignments AS (
SELECT id, cluster_id
FROM cuml_hdbscan(
input => (SELECT * FROM reduced ORDER BY id),
min_cluster_size => 30,
min_samples => 10
)
), family_counts AS (
SELECT a.cluster_id, m.family, COUNT(*) AS files,
MIN(m.sha256) AS example_sha256
FROM assignments a
JOIN ember_metadata_challenge m ON m.id = a.id
GROUP BY a.cluster_id, m.family
), ranked AS (
SELECT *,
SUM(files) OVER (PARTITION BY cluster_id) AS cluster_files,
SUM(CASE WHEN family <> '' THEN files ELSE 0 END)
OVER (PARTITION BY cluster_id) AS family_labeled_files,
ROW_NUMBER() OVER (
PARTITION BY cluster_id
ORDER BY CASE WHEN family = '' THEN 1 ELSE 0 END,
files DESC, family
) AS family_rank
FROM family_counts
)
SELECT cluster_id, cluster_files, family_labeled_files,
family AS most_common_family, files AS most_common_family_files,
example_sha256
FROM ranked
WHERE family_rank = 1
ORDER BY cluster_files DESC, cluster_id;
The query orders the input by id, which fixes the input sequence represented
by this capture. It does not guarantee identical fitted memberships or label
numbers across hardware and software environments.
The saved capture retains the SQL, all eight
result rows, query ID and execution conditions. The observed execution
used a dedicated EMBER server and had partial_native disposition: PCA and
HDBSCAN ran on the GPU. The plan also included CPU window processing.
Limits
- L2SqrtExpanded distance and brute-force kNN are fixed.
- Float64 features, as wide columns or list elements, are narrowed to Float32 on the device before training or inference. Each value rounds to the nearest Float32, which keeps about seven significant digits; a value beyond the Float32 range becomes infinite and fails the query.
- Empty input returns an empty result. Other inputs require min_samples < rows, min_cluster_size <= rows, and max_cluster_size <= rows; row-dependent conditions are checked at execution.
- Cluster ID -1 denotes noise. The fit and output are query-local.