Bad as Cloudbleed is, there’s no evidence attackers exploited it before the patch was deployed. But since the vulnerability was triggered more than 1.2m times from 6,500 sites, Cloudflare is taking no chances: the company has tapped an outside company, Veracode, to scour its code.
CEO Matthew Prince pledged the external review as he set out a detailed update after 12 days of investigation. That update includes a synopsis of how the vulnerability was created and who faced the most risk. He said Cloudflare continues to work with Google and others to eliminate all leaked data from memory:
We’ve successfully removed more than 80,000 unique cached pages. That underestimates the total number because we’ve requested search engines purge and re-crawl entire sites in some instances.
Cloudbleed is a serious vulnerability in Cloudflare’s internet infrastructure that Google Project Zero researcher Tavis Ormandy discovered in mid-February. It turned out that a single character in Cloudflare’s code caused the problem. In its initial blog post on the matter, Cloudflare said the issue stemmed from its decision to use a new HTML parser called cf-html.
Defining the trigger
In his update, Prince said Cloudbleed was triggered when a page with two characteristics was requested through Cloudflare’s network. The two characteristics were:
- The HTML on the page needed to be broken in a specific way; and
- A particular set of Cloudflare features needed to be turned on for the page in question.
He said:
When a page for a particular customer is being parsed it is stored in memory on one of the servers that is a part of our infrastructure. Contents of the other customers’ requests are also in adjacent portions of memory on Cloudflare’s servers. The bug caused the parser, when it encountered unterminated attribute at the end of a page, to not stop when it reached the end of the portion of memory for the particular page being parsed. Instead, the parser continued to read from adjacent memory, which contained data from other customers’ requests.
How Cloudflare customers can protect themselves
In a previous article on Cloudbleed, Naked Security offered tips Cloudflare customers can use to protect themselves from future exploits. The guidance was provided by Ryan Lackey, a well-known industry professional and former Cloudflare employee. He suggested the following:
- Change passwords. Doing so will improve security from both this potential compromise and many other, far more likely security issues.
- Use this incident to improve response plans. The situation presents a prime opportunity for users to put their incident handling process to the test. Lackey suggests companies and individuals discuss the specific impact to their application and what response makes the most sense.
- Invalidate authentication credentials for mobile applications and other machine-to-machine communications such as IoT devices. This forces users to re-enroll apps and devices if they used Cloudflare as an infrastructure provider. It may not be as effective as having everyone change their passwords, Lackey wrote, but it’s still a useful exercise.
- Review what this means from a compliance perspective. Lackey said that if an application or website is on Cloudflare and is subject to industry or national regulation, Cloudbleed may count as a reportable incident. “Your security and compliance teams should evaluate. Obviously, full compliance with applicable regulations is an essential part of security,” he said.