An extremely serious vulnerability lay undiscovered at the heart of much of The Cloud for seven years.
The vulnerability (CVE-2015-7835), which affects the Xen hypervisor software used by Cloud hosting companies like Amazon Web Services, is so serious that it was widely patched under embargo before being disclosed on 29 October 2015.
It was discovered by 栾尚聪 (好风) of Alibaba and affects Xen software from version 3.4 onwards running on the x86 architecture.
If you’re running Xen and haven’t already patched your systems, do so now.
Xen hypervisor software allows a ‘host’ server to be sub-divided into a number of smaller, easily managed virtual ‘guest’ servers.
Although they share hardware and some software, the guests behave as entirely independent servers that are isolated from each other and their host.
Virtualisation has become incredibly popular in IT departments and data centres around the world and it’s a key underpinning technology for Cloud infrastructure and services.
The vulnerability within Xen allows an attacker who’s running a virtualised guest server to reliably access the host machine’s memory and take over the entire host system, something you might be forgiven for missing if you couldn’t stay awake all the way through the rather dry description in the advisory:
The code to validate level 2 page table entries is bypassed when certain conditions are satisfied. This means that a PV guest can create writeable mappings using super page mappings.
Such writeable mappings can violate Xen intended invariants for pages which Xen is supposed to keep read-only.
This is possible even if the "allowsuperpage" command line option is not used.
Malicious PV guest administrators can escalate privilege so as to control the whole system.
This most recent bug is at least the third such escape bug found in Xen this year, following the announcement of the VENOM vulnerability in May and a similar flaw disclosed in July that, for reasons unknown, wasn’t saddled with a super-villain moniker.
So how did something so serious go undiscovered for so long in something so critical?
One perspective is provided by the security team at QubesOS (a security-focused operating system that relies on Xen) in their 29 October 2015 security bulletin.
... this is subtle bug, because there is no buggy code that could be spotted immediately. The bug emerges only if one looks at a bigger picture of logic flows ...
In other words it wasn’t obvious, but that doesn’t mean they’re letting Xen off the hook though:
On the other hand, it is really shocking that such a bug has been lurking in the core of the hypervisor for so many years ...
Specifically, it worries us that, in the last 7 years (i.e. all the time when the bug was sitting there having a good time) so much engineering and development effort has been put into adding all sorts of new features and whatnots, yet no serious effort to improve Xen security effectively.
Ian Jackson, a long-time open source veteran and a member of the Xen Project Security Team provides a response on the Xen Project blog.
He explains why he thinks some people have the impression that Xen is buggier than other similar products:
Unlike almost all corporations, and even most Free Software projects, the Xen Project properly discloses, via an advisory, every vulnerability discovered in supported configurations.
... For researchers developing new analysis techniques, Xen is a prime target. A significant proportion of the reports to security@xenproject are the result of applying new scanning techniques to our codebase. So our existing code is being audited, with a focus on the areas and techniques likely to discover the most troublesome bugs.
More interesting than that though is his honest appraisal of the state of computer security and what he sees as our collective attitude to it:
The general state of computer security in almost all systems is very poor. The reason for this is quite simple: we all put up with it. We, collectively, choose convenience and functionality: both when we decide which software to run for ourselves, and when we decide what contributions to make to the projects we care about. For almost all software there is much stronger pressure (from all sides) to add features, than to improve security.
Ultimately, of course, a Free Software project like Xen is what the whole community makes it. In the project as a whole we get a lot more submissions of new functionality than we get submissions aimed at improving the security.
In other words, if we want better computer security then it necessarily comes at the expense of something else (typically, something shiny.)
Sentiments that apply as much to the entire Internet of Things and websites that still harbour SQL injection vulnerabilities in 2015 as they do to Xen.