Note / 01 / Benchmark integrity
When perfect accuracy isn't proof
Why an extraordinary security-model score should begin a better question about the benchmark.
- Project
- Misleading Performance in Sysmon-Based Machine Learning
- Format
- Research field note
- Reading time
- 5 min read
The result that raised the question
Near-perfect accuracy is supposed to be the end of an experiment. In security machine learning, it can be the beginning of a forensic investigation into the benchmark itself.
My Sysmon study evaluated five models across 6,258 labeled events. The useful question was not simply whether the models could score well. It was what information made that performance possible, and whether the same information would exist when the detector met an unfamiliar execution.
Where an easy answer can hide
Event-level records can carry identifiers or execution-specific traces that correlate strongly with a label. If closely related events appear on both sides of a train-test split, a model can recognize the experiment instead of learning a durable behavioral distinction.
Removing identifier leakage changed the story. A result near 100 percent fell to roughly 61 percent. That was not a failed model; it was a more honest measurement of what the original setup had been rewarding.
Change the unit of analysis
The next step was to stop treating every event as an independent example and aggregate telemetry into temporal behavior. That changes the question from classifying a row to recognizing the pattern produced by an execution.
Under a balanced behavioral evaluation, the detector reached 82 percent. The number is lower than perfection and more useful because it corresponds to a task that better resembles the one the model is meant to perform.
A practical benchmark checklist
Before tuning an architecture, the representation can be audited: what counts as an independent example, which fields would exist at decision time, and whether the split separates underlying behaviors rather than only their rows.
Surprising performance deserves curiosity, not immediate celebration. A benchmark is part of the system, and it should be threat-modeled with the same care as the detector.
Working takeaways
What stays with the project.
- 01Audit the representation before optimizing the model.
- 02Define independence around behavior, not convenient rows.
- 03Treat extraordinary performance as a claim that needs explanation.