FP/noise baseline + noise-adjusted grades, 2026-10-27 wave (run-2026-10-27-001..033)
Owner: detection-analyst. Reproduce: python -X utf8 analysis/noise_baseline.py --date 2026-10-27 [--json]. Source: committed run.json alert summaries only (raw alerts.json is gitignored). Closes the "Scorecard needs the missing FP/noise baseline" carry-over from Advisory Board meetings 001/002.
Per-platform noise floor
| win-victim (29 runs) | lin-victim (4 runs) | |
|---|---|---|
| Baseline alerts/h (30 min pre-detonation window) | median >= 10,000/h; 27/29 runs hit the 5000-hit cap, so this is a LOWER BOUND. Only 2 uncapped runs: 836/h and 3,626/h | median 130/h (102-160), none capped |
| Alerts in the 90 s detonation window | median 1,813 (1,584-2,586) | 5 (all four = marker x2 + PAM/sudo) |
| Same, excluding rule 60107 | median 147 | 3 |
| Incidental alerts at level >= 10 per window | median 8 (2-45) | 0 |
Dominant Windows noise (hit share): rule 60107 (Security 4673 "Failed attempt to perform a privileged operation", L4) = 87.2% of baseline hits and 91.7% of window alerts; then the WEF 4624/4634/4672/4688 family (67022/67023/67028/67027, L3, ~2% each), then 751/752. Lab-owned noise worth noting: 100230 (SYSTEM-integrity process by a non-service parent, L10) fired 1,249x in baselines (~43 per run) -- a lab rule false-positive source at high severity. 92213 (Sysmon EID11 exe-dropped, L15) is also present in baselines. Hazard: 60107 carries mitre_id T1078, so any technique-tag matcher would "detect" T1078 with 1,683 FPs (run 014); matching must stay rule-id based, as the harness does.
Caveats: the baseline is the 30 min after a snapshot revert (boot + prior-step residue), not idle or in-use steady state; one lab VM, so this is a floor for the lab, not a fleet FP rate.
Noise-adjusted grades (vs each run's own baseline)
Rule: detected = a matched rule had zero hits in that run's baseline and (no expectation, or a matched rule was expected); partial = matched but noise-confounded, or expected rule silent and only an unplanned bundled rule hit; missed = no technique rule matched (harness partially-detected = marker + noise only).
| harness | noise-adjusted | |
|---|---|---|
| detected | 15 | 13 |
| partial | 0 (18 "partially-detected") | 2 (011, 013) |
| missed | 0 | 18 (14 Windows, 4 Linux) |
- 011 (T1047, 100407): rule also fired once in its own baseline -> confounded (needs raw events: residue vs benign wmic).
- 013 (T1059.001): expected 100405 silent; detected only by bundled 91822/92057 -> the planned rule did not fire.
- 024 (T1552.002) detected via 100411, expected 100409 silent (known: no Sysmon EID1 for reg.exe). 025 (T1555) via 100402, 100401 silent.
- 027 (T1566.002) expected 100405, nothing matched -> a true miss of a planned rule.
- Latency (detected, n=13, from window start so an upper bound incl. detonation runtime): median 43 s, p90 65 s, max 73 s.
- Signal vs noise: 1 matched alert per ~1,800 window alerts on Windows (median noise:signal 1,808:1). Matched rule level >= 10 in 9 of 13.
- Matched rule already present in own baseline: 13% of matched runs (chance floor).
- Missed techniques (gap list, 14 Windows): T1021.002, T1021.006, T1040, T1046, T1055, T1091, T1110.001, T1202, T1216, T1216.002, T1218.011, T1552, T1566.002 (expected-rule miss: highest priority), T1649; Linux: T1552.001, T1552.003, T1552.004, T1690.
Whole ledger (130 runs): 49 predate per-rule baselines (unadjusted); of 80 baseline-measured runs, 27 detected / 3 partial / 50 missed.
Deferred (needs raw events or a new lab run)
See DEFERRED in analysis/noise_baseline.py: causal attribution of matched alerts, uncapped per-rule baselines on the 27 capped runs, a true idle-host baseline (G1), and the 011 residue question.