What’s up with the 2023 MITRE ATT&CK Evaluations

Background on MITRE and the ATT&CK Framework

The MITRE Corporation is a non-profit organization that operates research and development centers sponsored by the United States federal government. MITRE has been a key player in cybersecurity for decades, with deep expertise in adversary techniques and defense strategies.

Their ATT&CK framework serves as a knowledge base of real-world adversary tactics, techniques, and procedures (TTPs) and provides a common language and model to describe cyber attacks in a structured and comprehensive way. “ATT&CK mapping” is now commonly integrated into cyber security products.

Overview of the 2023 MITRE ATT&CK Evaluations

In September 2023, MITRE Engenuity released results from its fifth round of ATT&CK-based product evaluations. The evaluations use a specialized testing methodology to reconstruct real-world adversary behavior in a typical corporate networks.

This round focused on emulating an advanced persistent threat (APT) group known as Turla, which has been active since at least 2007 targeting government and military organizations. The evaluations replicated Turla's end-to-end attack lifecycle across the ATT&CK model, leveraging 130+ TTPs observed in past Turla campaigns, with 2 scenarios: Carbon and Snake.

https://attackevalscdnendpoint.azureedge.net/publicsiteimages/Turla_Operational_Flow.png © MITRE Corporation

30 endpoint detection and response (EDR) solutions were evaluated in the enterprise category. MITRE's red team executed Turla's techniques within this environment while the security tools attempted detection and response.

Addressing detection gaps between vendors

The transparent results provide valuable technical insights into EDR capabilities and gaps against a sophisticated adversary:

• EDR tools showed significant variance in detection rates of Turla's techniques, highlighting differences in detection maturity.
• Most tools detected over 90% of steps, indicating progress in coverage of known ATT&CK techniques.
• However, several evasive techniques like process injection, DLL side-loading, and protocol tunnelling were missed by many vendors. These gaps indicate challenges detecting advanced adversary tradecraft.
• Behavioral and heuristic detection techniques continue to lag behind signature-based detection for most EDR products. Adversaries rely heavily on novel tools and behaviors, thus questioning the efficiency on threats variants…
• Alert triage and response features also need improvement. Many tools generated excessive alerts with insufficient context for analysts to prioritize. Delayed detection and containment opportunities were observed.

Even with these improvements, gaps remain in identifying and responding to sophisticated adversary tradecraft.

Validating and interpreting vendor claims

In the hyper-competitive cybersecurity market, vendors aggressively promote their capabilities. For example, several vendors boasted of "100% detection" of Turla's TTPs in their marketing announcements (do a quick internet search to convince yourself). However, the simulated attacks represent only a subset of threats seen in the wild (and again, Turla campaigns being known, all vendors should be able to detect it). Factors like diverse customer environments, usability, and stability also impact real-world efficacy.

As always, use their claims as a starting point and review:

• Transparent data that doesn't cherry-pick only favorable results
• Clear explanation of how metrics were measured
• Customer detection rates from real-world deployments

While useful, these metrics alone do not guarantee real-world protection. Enterprises should critically assess claims, validate through testing, and evaluate against business requirements and constraints.

Parting thoughts

MITRE ATT&CK evaluations provide valuable transparency and comparison of product capabilities against sophisticated threats. The technical insights highlight detection gaps that must be addressed through continued improvements by these vendors. However, users should be wary of inflated vendor claims, and instead rely on real-world testing, feedbacks and metrics tailored to their unique risk environment. These tools are no silver bullet and requires good processes, governance, deployment rate […] to work efficiently.