Thoughts on comparative testing

For months, Cylance has sought to dazzle audiences with its “Unbelievable” demonstration, staging well-choreographed battles against other IT security vendors, including Sophos. The exhibition ends with Cylance delivering near-perfect scores while everyone else (predictably) shows lackluster results. Yet when the playing field is leveled, and Cylance’s product comes under real scrutiny, the company cries foul, puts the fear of lawsuits into the minds of its partners, and accuses others of “smoke and mirrors” tactics.

At a recent Cylance presentation during an industry event in Las Vegas, one Sophos customer (from Chicago) in the audience asked to see how the Sophos product was configured for Cylance’s “Unbelievable” demo. On reviewing the settings, the customer discovered that key (and default) protection settings had been disabled. When the customer insisted that Cylance enable the proper default configuration and re-run the test, Sophos beat Cylance. The same behavior has been reported by multiple other vendors, including the disabling of everything other than hash lookups – an unfair test to say the least.

After seeing these demos and hearing numerous similar stories, we instructed our technical team to evaluate the Cylance claims so that we could test their validity. We focused on making the comparison fair, factual, and balanced using default and vendor-recommended settings. Sophos didn’t cherry pick or manipulate malware for the test.

Upon completing the testing, Sophos posted a video on YouTube outlining the results and showing real screen captures of the side-by-side testing. In fact, we deliberately went out of our way to be as transparent as possible.

After seeing the video online, Cylance contacted the reseller who provided access to the CylancePROTECT product, citing license compliance concerns and threatening “retribution” if the reseller involved did not demand that Sophos withdraw the video immediately; this left the reseller in fear of a lawsuit.

Given the importance of our partner relationships, at the request of the reseller and as a courtesy to them we chose to take down the video while we consider the best vehicle to provide the market with fair comparisons of Sophos’ and Cylance’s products. Again, to be very clear: the only reason we elected to take the video down was because the reseller was concerned about threats and pressure from Cylance, not because we believed the video was somehow inaccurate. If Cylance was to agree to stop pressuring or threatening the reseller, we’d be happy to re-post the video for all to see. In the meantime, if you’d like to hear the facts behind the video, just reach out to a Sophos partner or a Sophos sales representative.

Cylance itself has acquired access to many other vendors’ products, including Sophos, and has been using them in its own competitive testing in public demos, in violation of end user licenses. In fact, Cylance just renewed its licenses for Sophos products through one of our partners. When Cylance acquires our software we don’t threaten the reseller. Note that despite our efforts, to date, Cylance has been unwilling to allow us to license its products.

If you aspire to be a channel friendly IT security vendor, it’s not a good idea to bully your partners. Sophos is arguably the most channel-centric security vendor, as evidenced by sweeping the security-related CRN ARC awards for two consecutive years and counting. And we certainly didn’t get there by intimidating our partners.

As with any industry, IT security vendors sometimes make aggressive claims about what their products can do, and compare them to competitive products. Sometimes they are accurate and independently verifiable, sometimes less so. At Sophos, we have a long history of steadfastly adhering to one of our core company values – authenticity. It’s one of the key reasons we have built a base of more than 200,000 customers and over 20,000 reselling partners, and enjoy one of the highest customer satisfaction and renewal rates in the industry.

Sophos has not been contacted directly by Cylance to refute the results of our test. In the interest of fair play, we would welcome constructive conversation with Cylance to discuss our testing method and configurations used. If Cylance believes the configuration settings were somehow incorrect, we would be happy to reconfigure and rerun the test.

Cylance could also help the market assess the effectiveness of its product by participating in industry third-party tests. Cylance remains absent from virtually all public independent third-party tests (e.g., AV-TEST, AV Comparatives, SE Labs, NSS Labs, etc.). The one exception we are aware of was in December, 2015, when Cylance competed in AV-TEST’s independent analysis. Those test results were as follows:

• On protection against certain types of malware, Cylance scored 5.5 out of 6. Sophos scored a perfect 6 out of 6
• On performance Cylance scored 4 out of 6. Cylance had the second worst performance of all 11 vendors in the test. Sophos scored 5 out of 6 on the same test
• And on the usability test, Cylance again scored 4 out of 6 with an alarming 26 false positives (the entire group average was 3, Sophos had 1). Sophos scored a 5.5 out of 6 on the test.

More recently (June, 2016), Sophos asked a reputable third-party tester, MRG Effitas, to run an independent comparative test using live, in-the-wild samples representing zero-day malware. This test used malware that was minutes old, not days or weeks old, and that hadn’t yet circulated widely, if at all. The results: Sophos blocked 97 percent while CylancePROTECT blocked 91 percent.

We believe that customers should be wary of vendors who fail to participate in public tests because there is no way to hold them publicly accountable for their marketing claims.

Despite Cylance’s claims to the contrary, our industry has evolved significantly over the years. The days of just using hashes as the primary mechanism for blocking malware have long since passed. Any leader in this industry has stayed current by successfully adapting to the threats at hand and continuously innovating. Industry leaders do not rely on one piece of technology but an ever-evolving set for complete protection.

Effective cybersecurity solutions are needed now more than ever. At Sophos that’s our passion. And as a representative of Sophos, it’s nice to be part of an organization that acts in good faith and values innovation, truth, and authenticity.

UPDATE: I mentioned an event in Chicago, but have corrected to clarify that the event was in Las Vegas and the customer was from Chicago.