Skip to content
Naked Security Naked Security

What makes for truly independent security product testing?

It seems there's room for improvement when it comes to independent testing - but what are your thoughts on this?

Security vendors Cylance and CrowdStrike have accused some independent security testers of bias and unfairness, and it was a big discussion point among those who attended RSA Conference 2017 last week.

For its part, Sophos believes in independent testing as a way to make products better. In light of the RSA controversy, the company reaffirmed its views in a Sophos Blog post. Sophos CTO Joe Levy acknowledged that while these tests are not perfect, they’re still valuable:

Methodologies can never be perfect, but the best testing houses will evolve them over time. The worst will remain static and become increasingly irrelevant.

This isn’t the first time independent testers have come under scrutiny. Some accuse them of running tests that are static and out of alignment with real-world scenarios, leading to faulty results that can unfairly damage a vendor’s reputation. Others decry what they see as a pay-to-play atmosphere that leads to bias.

Naked Security reached out to security practitioners for their view on the subject. Most said the tests remain important, but that a lack of consistent standards and other problems diminish their value. In other words, there’s plenty of room for improvement.

In emails and on social media, they outlined ways to ensure such testing is as ironclad and useful as possible.

Cylance and CrowdStrike go to war

To recap the recent sequence of events:

  • CrowdStrike filed for a restraining order and injunction in a federal court a couple of weeks ago to keep NSS Labs from releasing the results of a recent NSS Advanced Endpoint Protection (AEP) group test. Last Monday, the first day of RSA, the court shot CrowdStrike down.
  • Cylance has long decried the methods of two certification labs – AV-Comparatives and MRG Effitas – when conducting endpoint protection tests. Right before the start of RSA, Cylance provided Salted Hash blogger Steve Ragan with the results of a new comparison test they commissioned from AV-TEST. The move rekindled debate over the merits of independent testing vs vendor-commissioned tests.

In response to the debate, the Anti-Malware Testing Standards Organization (AMTSO) reaffirmed its support for independent product testing in a statement on its website. It said, among other things:

Testing products in a fair and balanced way is very difficult. Product developers routinely make bold claims about the capabilities of their products. AMTSO supports the right of testers to put these claims to the test, to provide independent validation of their accuracy (or otherwise).

AMTSO acknowledged problems need to be addressed, and offered the following points:

We reject turning off product capabilities while comparing the capabilities of products in real-world use, as we believe that this introduces bias in the results.

We believe that any claims about what the results of tests show must be valid and accurate, and they must provide both data and evidence that the scenarios tested and the methodologies used do in fact match the resulting claims. In our opinion, test reports without this data and evidence should be rejected.

We believe that tests that don’t give the tested product vendors an opportunity to engage and to comment on the approach or to validate their configuration are unfair.

We believe that all comparative tests should follow our draft standards.

We support the rights of a tester to run any test it wants to, and to test any available product without limitation, consistent with the AMTSO draft standards.

More consistent standards needed

Lawrence M Walsh, CEO and chief analyst at New York-based business strategy firm the 2112 Group, said independent third-party testing by media and/or labs is an important part of the security industry, but that the market needs objective reviews to provide customers with validation or relative expectations for how products will perform when deployed. He said:

The problem is we, as an industry, lack a common testing baseline. We need a set of common standards that everyone can expect labs to follow. This doesn’t necessarily hamstring testers, as they can improve upon the standards and add their own secret sauce. A standard should set minimum expectations, and that’s something sorely needed in security product testing.

At the same time, he said, vendors shouldn’t be allowed to dictate how the standards are written:

ICSA Labs collaborates with vendors on its testing standards, though it remains the final arbiter of the test criteria. That always bothered me, because I felt the vendors could influence the criteria to narrow the parameters to avoid truly horrific results. I don’t think vendors can be excluded from the standards process, but they can’t have control or too much say in it, either. ICSA Labs did a good job separating the two, but can we say the same for all? Testing should reflect real-world conditions, not just ideal circumstances under controlled conditions.

Context is key

Gal Shpantzer, a Washington DC-based information security and risk management practitioner, said that when it comes to measuring the effectiveness of security technology, context is critical. He said testers – whether they’re independent or in-house – should ask specific questions:

You should understand what you’re protecting, from what, and in what context. Windows XP in the user LAN vs Windows Server 2012 in the datacenter are two very different things, for example. Also, what is your anti-malware agent doing for you relating to abuse of signed and trusted programs like PowerShell? Are you monitoring DNS in addition to malware agent output? Are you properly configuring your mail server to not allow obvious crud like .js files? Are you segmenting your network so that one laptop or server catching fire doesn’t mean instant inferno with multiple victims?
He added:
When we obsessively focus on whether this APT123 malware is or isn’t caught by the unicorn agent on our endpoints, we’re doing it wrong.

Ask yourself: is the test truly independent?

Joshua Marpet, SVP of compliance and managed services for an organization based in Washington DC, said companies and users need to take a hard look at what qualifies as an independent test. In many cases, he said instances where a test is advertised as independent turn out to be false. He said:

Nobody is doing anything independent these days in information security. Nobody is performing any of the actual independent models that I am aware of. I would love to know if anybody is. I haven’t looked at NSS in a while but they used to be as close as you could get to honestly independent.


Security practitioners will always have differing opinions on how independent tests should be carried out.  Sophos CTO Joe Levy suggested the following ways in which testing can be made more meaningful and effective:

  • There needs to be transparency (sharing methodologies and revealing all numbers) and consistency, and all vendors should be subjected to the same tests.
  • Vendors should not try to hide from tests, and they should probably think twice before threatening litigation against testing labs, their partners, or other vendors. Such stunts make vendors look dishonest, and ultimately harms end users.
  • Testing Labs should think twice about commissioned reports and what they could do to perceptions about their objectivity.
  • End users should look at multiple testing sourcing rather that trusting just one. They should also endeavor to do their own testing where practical.


“•End users should look at multiple testing sourcing rather that trusting just one. They should also endeavor to do their own testing where practical.”
We use the Virus Total website frequently. With regular use it does shed light on who is detecting fresh malware more often that others. But – I must add, there are other variables and features that can make some tools preferred over others.


Having worked at ICSA Labs for 18 years, and having authored many versions of its criteria testing standards, I can tell you that the sentence, “ICSA Labs used to let the vendors write the standards,” is patently untrue. While I enjoy your blog (usually), it is irresponsible to suggest that anyone other than ICSA Labs writes its criteria testing standards. It would be appreciated if nakedsecurity by Sophos would not misrepresent ICSA Labs by quoting someone who doesn’t work at the labs, who has never worked at the labs, and who is in no position to know what we do. As much as I like his last name, for him to suggest and for you to post that, “vendors could [avoid doing poorly in testing]” by having controlled the criteria testing standards development is also total bologna. The best example to refute this is the network IPS testing program. About 20 vendors attended the first few meetings where we collected input on criteria requirements. Some wanted this, some wanted that. In the end I wrote a set of testable requirements for what a network IPS ought to do. Of the 20 or so vendors, only 10 thought they could meet those requirements. Did you get that? Half thought their solution wasn’t ready and couldn’t meet the requirements. Of the 10 that participated in testing – all of which assured me repeatedly (as they are wont to do) – that they could meet the requirements, only 2 or 3 actually attained certification initially. Only one of those met the requirements while maintaining its claimed rated throughput. How about a more recent example? In advanced threat defense (ATD) testing today there are 10 or so vendor solutions in testing. 5 are certified. If we are so controlled by vendors who pay for testing and who allegedly control our testing requirement writing, how can this be? Back to the quote. I was surprised that there were actually two true statements about ICSA Labs in the quote, which I guess proves that even a blind pig can find an acorn now and then. First is that we at ICSA Labs “ha[ve] an idea going into the test” where we are trying to go with the testing requirements and second is that we collect security vendor input to “shape the parameters”. But that’s not the end of it. We also collect input from any interested stakeholder (e.g., those of you reading this). One final comment about the suggestion that ICSA Labs is somehow remotely controlled by vendors. Think about a group of 40 or 50 ATA, APT, or ATD (as we call it) security vendors all in a room. Imagine the dynamics of that for a second. Some are quiet and prefer to tell me in private what they think, others are outspoken, several are in conflict with one another. The point is that while all their input is valuable, it is not often harmonious. By making one vendor happy, three others could be annoyed. ICSA Labs has been navigating and steering through criteria development waters like that for a quarter of a century. I, for one, am proud of the standards development and security testing we have done on behalf of enterprises. It is they that stand to benefit from ICSA Labs’ testing. Take a look at ICSA Labs’ ATD testing results (for example) and see if you think there is any value to that testing. Then, perhaps, why not write about that?


Hi, Jack. Good to hear from you.

I checked back with Larry Walsh and showed him your comment. Upon reflection, he said he could have explained his position better. He clarified some points and I’ve adjusted the article accordingly. Please have a look and, if you still see a problem, do let me know.

Bill Brenner


Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to get the latest updates in your inbox.
Which categories are you interested in?
You’re now subscribed!