The Anti-Malware Testing Standards Organization (AMTSO) reaffirmed its support for independent product testing in the wake of disputes that surfaced at RSA Conference 2017 last week.
Two security vendors – Cylance and CrowdStrike – took issue with the practices of independent testing organizations, and it became a major discussion point among RSA attendees.
Sophos shares AMTSO’s position, which was mapped out in a press release:
Testing products in a fair and balanced way is very difficult. Product developers routinely make bold claims about the capabilities of their products. AMTSO supports the right of testers to put these claims to the test, to provide independent validation of their accuracy (or otherwise).
AMTSO said it was asked for an opinion on recent privately-commissioned anti-malware tests, and offered the following points:
- We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.
- We believe that any claims about what the results of tests show must be valid and accurate, and they must provide both data and evidence that the scenarios tested and the methodologies used do in fact match the resulting claims. In our opinion, test reports without this data and evidence should be rejected.
- We believe that tests that don’t give the tested product vendors an opportunity to engage and to comment on the approach or to validate their configuration are unfair.
- We believe that all comparative tests should follow our draft standards.
- We support the rights of a tester to run any test it wants to, and to test any available product without limitation, consistent with the AMTSO draft standards.
Sophos has long believed that independent testing is vital to the continued improvement of security technology. Sophos CTO Joe Levy acknowledged that while these tests are not perfect, they still have plenty of value.
“Methodologies can never be perfect, but the best testing houses will evolve them over time,” Levy said. “The worst will remain static and become increasingly irrelevant.”
There should be a partnership between the testing labs, end users and vendors, he added. Testing labs should under-promise and over-deliver, and work with vendors to configure environments correctly. End users should continuously be clear on the specific items they want to see reviewed, and vendors should make it easy for their products to be scrutinized.
For more meaningful and accurate testing, Levy suggested the following:
- There needs to be transparency (sharing methodologies and revealing all numbers) and consistency, and all vendors should be subjected to the same tests.
- Vendors should not try to hide from tests, and they should probably think twice before threatening litigation against testing labs, their partners, or other vendors. Such stunts make vendors look dishonest, and ultimately harm end users.
- Testing Labs should think twice about commissioned reports and what they could do to perceptions about their objectivity.
- End users should look at multiple testing sourcing rather than trusting just one. They should also endeavor to do their own testing where practical.
Simon Reed, VP of SophosLabs, said security product testing is hard, and third-party testing organizations need to focus on more depth and better testing than more different tests.
“Third-party testers need to acknowledge that some of their tests are focused on only parts of the malware attack chain and thus emphasize certain technologies over others,” Reed said. But in the bigger picture, he said, “Independent tests help us improve our quality. If third-party testers didn’t publish results, I would still want to be in at least half of them purely for an independent quality-control point of view.”
Reed added that third-party testers must clearly indicate in their reports which vendors assisted in the tests and which did not. Vendors included without approval should be able to make a short statement in the report on why they believe their inclusion wasn’t necessary, he said.