What makes for truly independent security product testing?

Security vendors Cylance and CrowdStrike have accused some independent security testers of bias and unfairness, and it was a big discussion point among those who attended RSA Conference 2017 last week.

For its part, Sophos believes in independent testing as a way to make products better. In light of the RSA controversy, the company reaffirmed its views in a Sophos Blog post. Sophos CTO Joe Levy acknowledged that while these tests are not perfect, they’re still valuable:

Methodologies can never be perfect, but the best testing houses will evolve them over time. The worst will remain static and become increasingly irrelevant.

This isn’t the first time independent testers have come under scrutiny. Some accuse them of running tests that are static and out of alignment with real-world scenarios, leading to faulty results that can unfairly damage a vendor’s reputation. Others decry what they see as a pay-to-play atmosphere that leads to bias.

Naked Security reached out to security practitioners for their view on the subject. Most said the tests remain important, but that a lack of consistent standards and other problems diminish their value. In other words, there’s plenty of room for improvement.

In emails and on social media, they outlined ways to ensure such testing is as ironclad and useful as possible.

Cylance and CrowdStrike go to war

To recap the recent sequence of events:

• CrowdStrike filed for a restraining order and injunction in a federal court a couple of weeks ago to keep NSS Labs from releasing the results of a recent NSS Advanced Endpoint Protection (AEP) group test. Last Monday, the first day of RSA, the court shot CrowdStrike down.
• Cylance has long decried the methods of two certification labs – AV-Comparatives and MRG Effitas – when conducting endpoint protection tests. Right before the start of RSA, Cylance provided Salted Hash blogger Steve Ragan with the results of a new comparison test they commissioned from AV-TEST. The move rekindled debate over the merits of independent testing vs vendor-commissioned tests.

In response to the debate, the Anti-Malware Testing Standards Organization (AMTSO) reaffirmed its support for independent product testing in a statement on its website. It said, among other things:

Testing products in a fair and balanced way is very difficult. Product developers routinely make bold claims about the capabilities of their products. AMTSO supports the right of testers to put these claims to the test, to provide independent validation of their accuracy (or otherwise).

AMTSO acknowledged problems need to be addressed, and offered the following points:

We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.

We believe that any claims about what the results of tests show must be valid and accurate, and they must provide both data and evidence that the scenarios tested and the methodologies used do in fact match the resulting claims. In our opinion, test reports without this data and evidence should be rejected.

We believe that tests that don’t give the tested product vendors an opportunity to engage and to comment on the approach or to validate their configuration are unfair.

We believe that all comparative tests should follow our draft standards.

We support the rights of a tester to run any test it wants to, and to test any available product without limitation, consistent with the AMTSO draft standards.

More consistent standards needed

Lawrence M Walsh, CEO and chief analyst at New York-based business strategy firm the 2112 Group, said independent third-party testing by media and/or labs is an important part of the security industry, but that the market needs objective reviews to provide customers with validation or relative expectations for how products will perform when deployed. He said:

The problem is we, as an industry, lack a common testing baseline. We need a set of common standards that everyone can expect labs to follow. This doesn’t necessarily hamstring testers, as they can improve upon the standards and add their own secret sauce. A standard should set minimum expectations, and that’s something sorely needed in security product testing.

At the same time, he said, vendors shouldn’t be allowed to dictate how the standards are written:

ICSA Labs collaborates with vendors on its testing standards, though it remains the final arbiter of the test criteria. That always bothered me, because I felt the vendors could influence the criteria to narrow the parameters to avoid truly horrific results. I don’t think vendors can be excluded from the standards process, but they can’t have control or too much say in it, either. ICSA Labs did a good job separating the two, but can we say the same for all? Testing should reflect real-world conditions, not just ideal circumstances under controlled conditions.

Context is key

Gal Shpantzer, a Washington DC-based information security and risk management practitioner, said that when it comes to measuring the effectiveness of security technology, context is critical. He said testers – whether they’re independent or in-house – should ask specific questions:

You should understand what you’re protecting, from what, and in what context. Windows XP in the user LAN vs Windows Server 2012 in the datacenter are two very different things, for example. Also, what is your anti-malware agent doing for you relating to abuse of signed and trusted programs like PowerShell? Are you monitoring DNS in addition to malware agent output? Are you properly configuring your mail server to not allow obvious crud like .js files? Are you segmenting your network so that one laptop or server catching fire doesn’t mean instant inferno with multiple victims?

When we obsessively focus on whether this APT123 malware is or isn’t caught by the unicorn agent on our endpoints, we’re doing it wrong.

Ask yourself: is the test truly independent?

Joshua Marpet, SVP of compliance and managed services for an organization based in Washington DC, said companies and users need to take a hard look at what qualifies as an independent test. In many cases, he said instances where a test is advertised as independent turn out to be false. He said:

Nobody is doing anything independent these days in information security. Nobody is performing any of the actual independent models that I am aware of. I would love to know if anybody is. I haven’t looked at NSS in a while but they used to be as close as you could get to honestly independent.

Conclusions

Security practitioners will always have differing opinions on how independent tests should be carried out.  Sophos CTO Joe Levy suggested the following ways in which testing can be made more meaningful and effective:

• There needs to be transparency (sharing methodologies and revealing all numbers) and consistency, and all vendors should be subjected to the same tests.
• Vendors should not try to hide from tests, and they should probably think twice before threatening litigation against testing labs, their partners, or other vendors. Such stunts make vendors look dishonest, and ultimately harms end users.
• Testing Labs should think twice about commissioned reports and what they could do to perceptions about their objectivity.
• End users should look at multiple testing sourcing rather that trusting just one. They should also endeavor to do their own testing where practical.