EMBER2024 A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
EMBER2024 provides extracted file features, metadata and labels. The reports use a later test period for model evaluation and a separate challenge cohort for file inspection.
How many mistakes does each classifier make in a later quarter?
Logistic regression and random forest train on 1,560,000 Win32 files from weeks 0 to 51, using the same 512 normalized byte and byte-entropy histogram features. Evaluation uses 359,994 later Win32 files from weeks 52 to 63 and counts both benign files flagged and malicious files missed.
The query fits both models, predicts labels for the later files, then
joins the known labels and aggregates the errors.
Predictions are class IDs 0 and 1. Test labels are joined after prediction.
-- Unified temporal comparison training both logistic regression and random forest
-- on the full deduplicated Win32 training dataset (~1.56M files) and predicting on
-- the full held-out Win32 test dataset (359,994 files).
-- Evaluates both query-local classifiers side-by-side in one GPU statement.
WITH logistic AS (
SELECT p.id, p.prediction, m.label, m.week_id, m.sha256, m.family
FROM cuml_logistic_regression(
training => (
SELECT v.id, v.histogram AS vector, v.label
FROM ember_research_vectors_train v
JOIN ember_metadata_train m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
predict => (
SELECT v.id, v.histogram AS vector
FROM ember_research_vectors_test v
JOIN ember_metadata_test m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
max_iter => 1000,
penalty_l2 => 1.0
) p
JOIN ember_metadata_test m ON m.id = p.id
),
random_forest AS (
SELECT p.id, p.prediction, m.label, m.week_id, m.sha256, m.family
FROM cuml_random_forest_classifier(
training => (
SELECT v.id, v.histogram AS vector, v.label
FROM ember_research_vectors_train v
JOIN ember_metadata_train m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
predict => (
SELECT v.id, v.histogram AS vector
FROM ember_research_vectors_test v
JOIN ember_metadata_test m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
n_trees => 100,
max_depth => 12,
seed => 42
) p
JOIN ember_metadata_test m ON m.id = p.id
),
classified AS (
SELECT 'logistic' AS model, id, prediction, label FROM logistic
UNION ALL
SELECT 'random_forest' AS model, id, prediction, label FROM random_forest
)
SELECT model, COUNT(*) AS files,
SUM(CASE WHEN prediction = 1 AND label = 1 THEN 1 ELSE 0 END) AS true_positive,
SUM(CASE WHEN prediction = 1 AND label = 0 THEN 1 ELSE 0 END) AS false_positive,
SUM(CASE WHEN prediction = 0 AND label = 1 THEN 1 ELSE 0 END) AS false_negative,
SUM(CASE WHEN prediction = 0 AND label = 0 THEN 1 ELSE 0 END) AS true_negative
FROM classified
GROUP BY model
ORDER BY model;
The later test set contains 179,994 benign and 180,000 malicious files.
The query took 1256.11 s.
Random forest flags 15,440 fewer benign files and misses 7,118 fewer malicious files in this fit. It still misses 23,468 malicious files. These results describe one fit with the recorded options. Repeated training can change the logistic result.
The full-cohort capture records the query ID, cohort counts and observed repeat range.
Which high-scoring files should an analyst inspect first?
The query scores all 6,315 challenge files with a pretrained LightGBM model using 2,568 features, then joins the three largest positive feature-block contributions for each file. A review queue built from this result carries both the score and the inputs behind it.
The model is loaded by cuml_forest_predict. CPU LightGBM pred_contrib was
run before SQL, with its 2,568 contributions summed into 12 named blocks.
cuvs_top_k ranks those prepared values. The query does not compute SHAP.
SELECT p.id, m.sha256, m.file_type, m.week_id, p.prediction,
t.rank, g.group_name, t.value AS positive_log_odds_contribution
FROM cuml_forest_predict(
input => (
SELECT id, vector
FROM ember_research_vectors_challenge
ORDER BY id
),
model => 'EMBER2024_all.model',
model_format => 'lightgbm'
) p
JOIN cuvs_top_k(
input => (SELECT id, vector FROM ember_attributions ORDER BY id),
k => 3,
select => 'max'
) t ON t.id = p.id
JOIN ember_feature_groups g ON g.group_index = t.position
JOIN ember_metadata_challenge m ON m.id = p.id
WHERE p.prediction >= 0.99 AND t.value > 0
ORDER BY p.prediction DESC, p.id, t.rank
LIMIT 30;
The 0.99 threshold includes 1,616 files before the output limit. The returned
30 rows represent ten files in this capture. The highest-ranked file is
20001348, with a score of 0.999851.
Its three returned contributions are:
The query took 0.24 s.
The returned blocks identify model inputs for further inspection. Negative contributions and model bias are absent from the table, and these values do not explain why a file is malicious.
GPU predictions matched CPU LightGBM within 3.0e-8 maximum absolute error
across all 6,315 files. Every challenge label is malicious, so this cohort
cannot measure false-positive rate or establish score calibration.
The capture retains the full SHA-256 records, all 30 rows, attribution preparation and CPU score comparison.
What connects these files to the seed?
The seed is the unlabeled Win32 file 20005740, with 58 retained imports.
The query returns candidates with related imports and the symbols behind
each match. The seed was selected by SHA-256 order among files with 10 to 100
retained imports and at least five neighbors in a separate histogram graph.
Family labels were not used to select it.
Personalized PageRank creates a 25-file queue. Jaccard ranks import overlap within that queue, and relational joins retrieve shared symbols and file metadata. The sample-to-symbol edges were prepared before SQL, retaining symbols occurring in 3 to 100 Win32 challenge files.
WITH candidates AS (
SELECT p.vertex
FROM cugraph_personalized_pagerank(
edges => (
SELECT src_id AS src, dst_id AS dst
FROM ember_48_challenge_import_edges
),
personalization => (
SELECT id AS vertex, CAST(1.0 AS DOUBLE) AS value
FROM ember_49_challenge_graph_seed
),
directed => false
) p
JOIN ember_48_challenge_import_sample_counts c ON c.id = p.vertex
CROSS JOIN ember_49_challenge_graph_seed seed
WHERE p.vertex <> seed.id AND c.retained_import_count >= 5
ORDER BY p.value DESC, p.vertex
LIMIT 25
), similarities AS (
SELECT * FROM cugraph_jaccard(
edges => (
SELECT src_id AS src, dst_id AS dst
FROM ember_48_challenge_import_edges
),
vertex_pairs => (
SELECT seed.id AS first, c.vertex AS second
FROM candidates c CROSS JOIN ember_49_challenge_graph_seed seed
),
directed => false
)
)
SELECT j.second AS candidate_id, m.sha256, m.family,
counts.retained_import_count, j.similarity,
COUNT(*) AS shared_rare_imports,
MIN(symbols.symbol) AS example_shared_import
FROM similarities j
JOIN ember_48_challenge_import_edges seed_edges ON seed_edges.src_id = j.first
JOIN ember_48_challenge_import_edges candidate_edges
ON candidate_edges.src_id = j.second AND candidate_edges.dst_id = seed_edges.dst_id
JOIN ember_48_challenge_import_symbols symbols ON symbols.symbol_id = seed_edges.dst_id
JOIN ember_metadata_challenge m ON m.id = j.second
JOIN ember_48_challenge_import_sample_counts counts ON counts.id = j.second
GROUP BY j.second, m.sha256, m.family, counts.retained_import_count, j.similarity
ORDER BY j.similarity DESC, j.second
LIMIT 15;
The first four candidates share 54 imports with the seed. Each has 130
retained imports, giving Jaccard 54 / (58 + 130 - 54) = 0.403.
The query took 0.10 s.
One shared symbol is msvbvm60.dll!MSVBVM60.DLL:ordinal521.
All 15 returned family fields are empty.
The common msvbvm60.dll runtime can account for strong import overlap
without establishing a shared malware family or campaign. The symbol join
lets an analyst inspect what produced a graph match before assigning meaning
to its score. An empty family field is missing attribution.
Edges exist for only 926 of 3,225 Win32 challenge files; 2,299 files have no retained edges and are outside this candidate search. Changing the import frequency bounds requires preparing a new edge table.
The capture records all results, preparation counts and CPU checks.
Data scope
The deduplicated train, test and challenge splits contain 2,626,000, 605,929 and 6,315 files. Every challenge file is malicious, and 3,225 of them are Win32. Research preparation retains the first occurrence of each SHA-256 within a split and assigns separate ID ranges to train, test and challenge records.
Related analyses
The historical reference search retrieves earlier Win32 files for a later query file. The HDBSCAN review groups the 3,225 Win32 challenge histograms, while the UMAP view shows a saved two-dimensional projection of these selected features.
Each report links to the SQL, returned rows and preparation records used for its conclusions.
The query timings include planning, execution and result receipt.