{"id":2930,"date":"2025-09-23T00:52:49","date_gmt":"2025-09-23T00:52:49","guid":{"rendered":"http:\/\/ft365.org\/index.php\/2025\/09\/23\/major-cyber-threat-detection-vendors-pull-out-of-mitre-evaluations-test\/"},"modified":"2025-09-23T00:52:49","modified_gmt":"2025-09-23T00:52:49","slug":"major-cyber-threat-detection-vendors-pull-out-of-mitre-evaluations-test","status":"publish","type":"post","link":"http:\/\/ft365.org\/index.php\/2025\/09\/23\/major-cyber-threat-detection-vendors-pull-out-of-mitre-evaluations-test\/","title":{"rendered":"Major Cyber Threat Detection Vendors Pull Out of MITRE Evaluations Test"},"content":{"rendered":"
\n
\n

Three major providers of cybersecurity solutions have decided not to take part in the 2025 edition of MITRE\u2019s annual endpoint detection and response (EDR) solution test.<\/p>\n

After Microsoft announced it would not participate in MITRE Engenuity ATT&CK Evaluations: Enterprise 2025 in June, SentinelOne and Palo Alto Networks confirmed on September 12 they were also pulling out of the test for this year.<\/p>\n

These decisions have raised concerns among the cybersecurity community about the program\u2019s future and relevancy. These moves are especially surprising since the three companies are major cybersecurity vendors and all performed well in the 2024 test, in which Microsoft\u2019s solution topped MITRE\u2019s tests, SentinelOne ranked fifth and Palo Alto ranked 12th.<\/p>\n<\/p><\/div>\n

\"Credit:
Credit: Poetra.RH \/ bluestork \/ Shutterstock.com<\/figcaption><\/figure>\n
\n

It is a particularly surprising decision for Microsoft, which used its ranking in the test to promote its solution, Microsoft Defender XDR, as recently as December 2024.<\/p>\n

Interestingly, all three companies justified the move by saying they wanted to prioritize product development and innovation.<\/p>\n

However, experts have suggested that other factors may also be at play, including the tests becoming increasingly seen as promotional rather than achieving real security gains.<\/p>\n

Infosecurity<\/em> spoke with Charles Clancy, MITRE CTO and SVP of MITRE Labs, who shared key elements of the evolution of the evaluation test that could explain the decisions ahead of the results of this year\u2019s test in December 2025.<\/p>\n

Backstory of\u00a0ATT&CK Evaluations: Enterprise<\/strong><\/h2>\n

MITRE Corporation is a US-based non-profit organization running many cybersecurity programs, including some on behalf of the US government.<\/p>\n

MITRE introduced its ATT&CK framework in 2015, which quickly became the standard tool in the cybersecurity industry for mapping real-world cyber adversaries\u2019 techniques, tactics and procedures (TTPs).<\/p>\n

In 2019, MITRE ATT&CK launched its first Evaluations program to \u201cfill a gap in the security testing market,\u201d Clancy argued.<\/p>\n

\u201cThere were many types of third-party testing out there for cybersecurity products, but each one of them had their own process and scoring methodology, leading to inconsistent results and a lack of rigor that wasn\u2019t driving the industry forward,\u201d he explained.<\/p>\n

MITRE Engenuity ATT&CK Evaluations: Enterprise is the most regular of all Evaluations tests, occurring every year since its launch.<\/p>\n

In a LinkedIn post, Igal Gofman, the director of engineering at CrowdStrike and a former security researcher at Microsoft and Tenable, called the test the \u201cOlympics of cybersecurity.\u201d<\/p>\n

Among the 1000 people working in MITRE\u2019s cybersecurity practice, 133 are dedicated to MITRE ATT&CK, of whom 12 to 15 people are working on the Evaluations tests, Clancy told Infosecurity<\/em>.<\/p>\n

Each year, the team behind the testing program picks one of several real-life adversaries and\/or attack chains based on their TTPs mapped in ATT&CK.<\/p>\n

They then test the EDR solutions of participating vendors in simulated attacks using Caldera, MITRE\u2019s own automated adversary emulation platform, according to several criteria, including detection results, false positives and true negatives.<\/p>\n

Although this test can be used to compare how effective EDR solutions are, Clancy noted it should not be seen as a longitudinal benchmark because each annual test differs greatly from the previous one.<\/p>\n

\u201cThe ethos we\u2019re trying to drive in the testing is comparison of an individual product to detect a particular threat actor. Simulating different adversaries year over year is really important to understand different classes of emerging threats,\u201d Clancy said.<\/p>\n

Inside the Test’s 2024 and 2025 Editions<\/strong><\/h2>\n

In 2024, MITRE ATT&CK Evaluations: Enterprise emulated 14 techniques across 7 tactics from known North Korean-affiliated hackers, 16 techniques across 7 tactics from the CL0P ransomware group and 31 techniques across 11 tactics from the LockBit ransomware group.<\/p>\n

CrowdStrike, one of the leading EDR providers, did not take part in that year\u2019s edition, with one member of the CrowdStrike subreddit \u2013 who claimed to be working for the company \u2013 suggesting that the evaluation was set to take place shortly after the July 19 global outage that affected the company\u2019s EDR product.<\/p>\n

That year, Microsoft, ESET and Cybereason topped the ranking, followed by ThreatDown, SentinelOne and Bitdefender.<\/p>\n<\/p><\/div>\n

\"Source:
Source: MITRE<\/figcaption><\/figure>\n
\n

In 2025, the ATT&CK Evaluations team has selected two scenarios:<\/p>\n

    \n
  • A Scattered Spider scenario: multi-faceted intrusion in a hybrid environment that features social engineering, cloud infrastructure exploitation, identity abuse and living off the land (LOTL) techniques<\/li>\n
  • A Chinese-aligned cyber-espionage scenario: evasive intrusion highlighting the adversary\u2019s adept use of social engineering, abuse of legitimate applications and services, establishing persistent mechanisms and employing custom malware to evade detection<\/li>\n<\/ul>\n

    While he admitted vendors can vary year-over-year, Clancy assured they can rely on \u201ca lot of repeat customers.\u201d<\/p>\n

    Why Vendors Are Pulling Out of MITRE\u2019s\u00a0Test<\/strong><\/h2>\n

    However, this year\u2019s edition, the results of which are expected in December, will be missing three major players: Microsoft, SentinelOne and Palo Alto Networks.<\/p>\n

    Microsoft announced it will not take part in this year\u2019s test on June 13, claiming that this decision \u201callows us to focus all our resources on the Secure Future Initiative and on delivering product innovation to our customers.\u201d<\/p>\n

    On September 12, SentinelOne and Palo Alto released similar statements.<\/p>\n

    The former said it wanted to \u201cprioritize our product and engineering resources on customer-focused initiatives while accelerating our platform roadmap,\u201d while the latter explained that this decision \u201cenables us to further accelerate critical platform innovations that directly address our customers’ most pressing security challenges and respond even faster to the evolving threat landscape.\u201d<\/p>\n

    When contacted by Infosecurity<\/em>, SentinelOne and Palo Alto Networks declined to provide further comment. Microsoft did not respond to a request for comment.<\/p>\n

    However, MITRE\u2019s Clancy said he is in close contact with the three vendors and believes he knows the reasons that made them pull out of this year\u2019s test.<\/p>\n

    First, as the vendors said in their statements, taking part in MITRE ATT&CK Evaluations program requires a resource-intensive commitment, suggesting that the time and personnel dedicated to it are lost on other projects.<\/p>\n

    Then, Clancy said that the team behind the test strives to make it harder every year and conceded they may have pushed it too far this year.<\/p>\n

    \u201cEach year, we want to design a test that\u2019s harder than the year before in order to drive the whole industry forward, since the test can offer an opportunity for vendors to upgrade their products in preparation for the test and once they get the results. And sometimes, we don\u2019t get the balance quite right,\u201d he explained.<\/p>\n

    Speaking to Infosecurity<\/em>, Vishal Santharam, a senior product manager for endpoint security products at ManageEngine, elaborated on Clancy\u2019s point.<\/p>\n

    \u201cIn 2024, MITRE started recording the volume of alerts in the evaluations, which is always a challenge for a vendor to tune in to. More alerts mean increased alert fatigue,\u201d he said, referring to a Forrester study decoding the 2024 MITRE Evaluations: Enterprise based on alert volume.<\/p>\n<\/p><\/div>\n

    \"Source:
    Source: Forrester<\/figcaption><\/figure>\n
    \n

    Additionally, Santharam noted that the 2025 Evaluations: Enterprise test included cloud environment, \u201cwhich is untested territory and requires even more attention from vendors.\u201d<\/p>\n

    Finally, Clancy told Infosecurity <\/em>that his team used to run a vendor forum each year to prepare for the MITRE ATT&CK Evaluations: Enterprise test.<\/p>\n

    \u201cThis forum, which was helpful in working with industry to set the objectives of the test each year, fell off over the last couple of years,\u201d Clancy admitted.<\/p>\n

    On LinkedIn, CrowdStrike\u2019s Gofman argued that the MITRE Evaluations tests were initially a great initiative to benchmark security solutions, but they turned into \u201cvendor theater\u201d in recent years.<\/p>\n

    \n

    \u201cVendors investing huge resources for PR wins, not real security improvements. With MITRE and CISA under pressure from budget cuts and changes, some vendors likely saw an opportunity to step back,\u201d he said.<\/p>\n

    \u201cThe concept of TTP-based testing is still valuable, but the way it\u2019s evolved, outdated, overly endpoint-focused, detached from real-world threats is far less so,\u201d he added.<\/p>\n<\/div>\n

    Patrick Garrity, a vulnerability researcher at VulnCheck, corroborated this view: \u201c[It] sounds like this benchmarking activity has become a giant distraction to building better products in exchange for publicity,\u201d he said in another LinkedIn post.<\/p>\n

    Despite these concerns, Clancy confirmed that a dozen cybersecurity vendors were still taking part in the 2025 edition of the test.<\/p>\n

    MITRE to Reboot Vendor Forum in 2026<\/strong><\/h2>\n

    Clancy told Infosecurity<\/em> that his team intended to re-establish the vendor forum ahead of MITRE ATT&CK Evaluations: Enterprise 2026.<\/p>\n

    \u201cThis is something we\u2019re already working to re-establish for the 2026 edition,\u201d he said.<\/p>\n

    He later made this ambition public in a LinkedIn post published on September 18, after SentinelOne and Palo Alto announced they would not participate in the 2025 edition.<\/p>\n<\/p><\/div>\n

    <\/figure>\n
    \n

    Santharam also told Infosecurity<\/em> that ManageEngine was\u00a0 working on an EDR solution and intended to participate in MITRE Engenuity ATT&CK Evaluations: Enterprise in 2026.<\/p>\n

    \n

    “The Advanced Anti-Malware and Next-Gen AV products from ManageEngine were certified by AV-Comparatives on their first try.\u00a0\u00a0The solution paves the way for our next EDR\u00a0offering while also providing comprehensive protection against malware and ransomware,”\u00a0he said.\u00a0<\/p>\n

    “We are also gearing up to take part in the forthcoming Gartner Magic Quadrant for Endpoint Protection Platform (EPP), and the MITRE ATT&CK tests.\u00a0In addition to proving the robustness and reliability of our technology, these independent assessments also assist clients in developing confidence in our EDR capabilities.”<\/p>\n<\/div><\/div>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"

    Three major providers of cybersecurity solutions have decided not to take part in the 2025 edition of MITRE\u2019s annual endpoint detection and response (EDR) solution test. After Microsoft announced it would not participate in MITRE Engenuity ATT&CK Evaluations: Enterprise 2025 in June, SentinelOne and Palo Alto Networks confirmed on September 12 they were also pulling<\/p>\n","protected":false},"author":2,"featured_media":2931,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2930","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"featured_image_urls":{"full":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"thumbnail":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220-150x150.jpg",150,150,true],"medium":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"medium_large":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"large":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"1536x1536":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"2048x2048":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"morenews-featured":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"morenews-large":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"morenews-medium":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220.jpg",300,300,false],"crawlomatic_preview_image":["http:\/\/ft365.org\/wp-content\/uploads\/2025\/09\/2930-5551346a-8115-4f32-93cd-24d517723220-146x146.jpg",146,146,true]},"author_info":{"display_name":"henry","author_link":"http:\/\/ft365.org\/index.php\/author\/henry\/"},"category_info":"Uncategorized<\/a>","tag_info":"Uncategorized","comment_count":"0","_links":{"self":[{"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/posts\/2930","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/comments?post=2930"}],"version-history":[{"count":0,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/posts\/2930\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/media\/2931"}],"wp:attachment":[{"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/media?parent=2930"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/categories?post=2930"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/ft365.org\/index.php\/wp-json\/wp\/v2\/tags?post=2930"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}