The testing of cybersecurity products is a contentious subject, with vendors and test organizations striving to deliver an objective evaluation despite very different ideas of what that looks like. The threat landscape is constantly evolving, and this makes evaluating a single product’s effectiveness at a single point in time problematic. Yet however imperfect and difficult it might be, Sophos is committed to improving every customer’s ability to make informed decisions about security solutions.
For a long time, we have encouraged organizations to seek independent validation of vendor-claimed product effectiveness, ideally from multiple sources. That’s why you’ll find detection performance, analyst reviews, and test results from independent vendors such as NSS Labs, SE Labs, MRG Effitas, and AV-Test on our website and in our marketing materials. Sophos proactively engages in public and commissioned tests and uses the results to improve product effectiveness.
Testing security products isn’t easy. There are few objective resources that can help to establish the connections between how attacks are performed in the real world and the capabilities of security products to prevent or detect them. Designing and executing a statistically relevant and objectively valuable test is extraordinarily difficult. There are many tools, frameworks, and sample sources available, but they require intensive curation, orchestration, and auditing to be used effectively in measurement. These challenges often lead to tests that measure security effectiveness across only a single threat vector, such as static file detections in isolation without consideration for methods of delivery. As a result, published independent tests are not holistic and disproportionately focus on a single dimension, such as file-based portable execution detection.
In 2008, Sophos became a founding member of AMTSO, as we believe that the way to improve testing experiences is to get all stakeholders — vendors, independent test organizations, customers, and analysts — to adopt working practices and test standards that better serve all our customers. In addition to our work with AMTSO, we are committing to uphold certain best practice principles whenever we engage in product testing.
Testing of security products should be fair, rigorous, transparent, and collaborative. Read the Sophos Third-Party Testing Principles here: www.sophos.com/bestpracticetesting
We understand the value of independent evaluations, yet sometimes tests do not provide a clear methodology or enable vendors to comment on results before they are published, which is disadvantageous for customers. With a commitment to transparency and a focus on the customer need, we can find the right path forward.
Sophos believes that customers and the industry will benefit if all security vendors and testing organizations:
- Embrace and contribute to the ongoing review and improvement of such projects as the AMTSO Testing Protocol Standard and MITRE’s ATT&CK Framework;
- Remove arbitrary or excessive restrictions on the use of security products for the purposes of comparative testing and publication of factual results;
- Actively participate in high quality independent tests, especially those that are aligned with the Sophos principles and AMTSO standards;
- Advocate for fair, rigorous, transparent, and collaborative testing that provides the answers prospective customers need to make better informed decisions about security products.
We are proud of the quality and effectiveness of our products – and you can put them to the test. We’ll be sharing more about how we intend to further improve everyone’s ability to test our security products shortly, so watch this space!