Cloudbleed is a serious vulnerability in Cloudflare’s Internet infrastructure that Google Project Zero researcher Tavis Ormandy discovered in mid-February. Much has been made of its severity, and rightly so. But there’s another part of the story.
Though not perfect, many industry experts believe the incident was handled well by all sides, and is an example other companies can follow if they someday find themselves in Cloudflare’s position.
There have been some points of contention along the way. Some believe Ormandy jumped the gun and announced the vulnerability before the date he had worked out with Cloudflare, throwing the company into an unnecessary scramble.
But industry experts also praise Ormandy for finding this proverbial needle in a haystack, and Cloudflare for patching the vulnerability with lightening speed. Cloudflare is also getting credit for its honest, detailed public response.
Cloudbleed defined
Ormandy contacted Cloudflare to report a vulnerability in its edge servers on Feb. 17. It turned out that a single character in Cloudflare’s code caused the problem. In its blog post, Cloudflare said the issue stemmed from its decision to use a new HTML parser called cf-html.
From the Cloudflare blog:
It turned out that in some unusual circumstances, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines. We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response.
Ormandy also laid out the details in this advisory. He said:
This situation was unusual, PII was actively being downloaded by crawlers and users during normal usage, they just didn’t understand what they were seeing. Seconds mattered here, emails to support on a Friday evening were not going to cut it. I don’t have any Cloudflare contacts, so reached out for an urgent contact on twitter, and quickly reached the right people. After I explained the situation, Cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour.
Despite (or perhaps because) Tavis started his advisory “It took every ounce of strength not to call this issue ‘cloudbleed'” the flaw quickly received the same branding treatment given to such previous blockbuster vulnerabilities as Heartbleed and Shellshock. It got a catchy name and logo.
A rush to go public?
On the surface, this researcher-to-vendor collaboration went well. But in recent days, some in the security industry have suggested that Ormandy announced the bug too soon – specifically, sooner than the window he had originally worked out with Cloudflare.
When a researcher works with a vendor to mitigate a vulnerability, a window between discovery and public announcement is typically worked out so the affected organization has time to properly close the security hole and make sure customers are adequately protected.
Sources close to Cloudflare say that Ormandy went public earlier than promised, sending Cloudflare into a scramble to complete its investigation and communicate with customers.
Ormandy did not return requests for comment.
Misplaced rage
Wim Remes, CEO and principal consultant at NRJ Security, said criticism toward Ormandy is misplaced. In a conversation on Facebook Messenger Friday, he said the social media echo chamber was distorting matters.
“You’re either for or against Tavis Ormandy, you’re ok with Cloudflare’s approach or you aren’t, and so on,” he said. “It doesn’t really matter.”
In a blog post, he described this as misplaced rage. If companies using third-party service providers like Cloudflare took the time to understand what they were paying for and did more on their end to ensure security, the issues described above wouldn’t matter. He wrote:
I think I’ve been repeating the same mantra to companies for at least a decade: You outsource process and function, but never responsibility. If you include third-party services in your product, no matter what they are, you need to go beyond having the supplier fill in a 400-question SIG questionnaire. You have to actually freaking test that component as if it is a pacemaker that your mother will get implanted. Third-party components remain your responsibility!
Since it’s a responsibility companies don’t seem to take seriously, people should simply be thankful to Ormandy and Cloudflare for getting Cloudbleed sorted out, Remes said.
Defensive measures
Though Cloudflare dealt swiftly with Cloudbleed, there’s still concern about any potential damage done. Ryan Lackey, a well-known industry professional and former Cloudflare employee, mapped out the risks in his blog post:
While Cloudflare’s service was rapidly patched to eliminate this bug, data was leaking constantly before this point — for months. Some of this data was cached publicly in search engines such as Google, and is being removed. Other data might exist in other caches and services throughout the Internet, and obviously it is impossible to coordinate deletion across all of these locations. There is always the potential someone malicious discovered this vulnerability independently and before Tavis, and may have been actively exploiting it, but there is no evidence to support this theory. Unfortunately, it is also difficult to conclusively disprove.
With that in mind, Lackey suggested site owners and administrators who use Cloudflare take the following steps:
- Change your passwords. “While this is on all probability not necessary (it is unlikely your passwords were exposed in this incident), it will absolutely improve your security from both this potential compromise and many other, far more likely security issues,” he said.
- Use this incident to improve response plans. The situation presents a prime opportunity for users to put their incident handling process to the test. Lackey suggests companies and individuals discuss the specific impact to their application and what response makes the most sense.
- Invalidate authentication credentials for mobile applications and other machine-to-machine communications such as IoT devices. This forces users to re-enroll apps and devices if they used Cloudflare as an infrastructure provider. It may not be as effective as having everyone change their passwords, Lackey wrote, but it’s still a useful exercise.
- Review what this means from a compliance perspective. Lackey said that if an application or website is on Cloudflare and is subject to industry or national regulation, Cloudbleed may count as a reportable incident. “Your security and compliance teams should evaluate. Obviously, full compliance with applicable regulations is an essential part of security,” he said.
