Apache “Optionsbleed” vulnerability – what you need to know

Remember Heartbleed?

That was a weird sort of bug, based on a feature in OpenSSL called “heartbeat”, whereby a visitor to your server can send it a short message, such as HELLO, and then wait a bit for the same short message to come back, thus proving that the connection is still alive.

The Heartbleed vulnerability was that you could sneakily tell the server to reply with more data than you originally sent in, and instead of ignoring your malformed request, the server would send back your data…

…plus whatever was lying around nearby in memory, even if that was personal data such as browsing history from someone else’s web session, or private data such as encryption keys from the web server itself.

No need for authenticated sessions, remotely injected executable commands, guessed passwords, or any other sort of sneakily labyrinthine sequence of hacking steps.

Crooks could just ask a server the same old innocent-looking question over and over again, keep track of the data that it let slip every time, and dig through it later on, looking for interesting stuff.

Well, something similar has happened again.

This time, the bug isn’t in OpenSSL, but in a program called httpd, probably better known as the Apache Web Server, and officially called the Apache HTTP Server Project.

The vulnerability has been dubbed OptionsBleed, because the bug is triggered by making HTTP OPTIONS requests.

If you know a bit about HTTP, you’ve probably heard of GET and POST requests, typically used to fetch data or web pages or to upload files or filled-in forms.

But there are numerous other HTTP request types, known officially as methods, including:

Not all web servers support all the many official methods, so there’s a special method to help you out:

By using OPTIONS you can avoid hammering a web server with requests that are never going to work, thus avoiding frustration at your end of the connection, and saving the server from wasted effort at the other.

For example:

  $ wget -S --method=OPTIONS https://my.example/index.html
  HTTP request sent, awaiting response... 
    HTTP/1.1 200 OK
    Allow: OPTIONS, TRACE, GET, HEAD, POST
    Public: OPTIONS, TRACE, GET, HEAD, POST
    Content-Length: 0
    Date: Tue, 19 Sep 2017 14:09:26 GMT
  $

Here, you can see the methods that the my.example server accepts, listed in the Allow: header.

Harmless enough, you might think – and that’s how it ought to be.

But software and security researcher Hanno Böck found otherwise.

He set out to measure what OPTIONS were supported by the Alexa Top 1,000,000 websites, asking each of them in turn..

Böck spotted that a small but noticeable fraction of the servers sent back garbled or corrupted-looking responses, such as:

  Allow: ,GET,,,POST,OPTIONS,HEAD,,
  Allow: POST,OPTIONS,,HEAD,:09:44 GMT
  Allow: GET,HEAD,OPTIONS,=write HTTP/1.0,HEAD,,HEAD,POST,,HEAD,TRACE

One that Böck lists looked curiously suggestive of a “bleed-type” data leak:

  Allow: GET,HEAD,OPTIONS,,HEAD,,HEAD,,HEAD,, 
    HEAD,,HEAD,,HEAD,,HEAD,POST,,HEAD,, HEAD,!DOCTYPE 
    html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"

Not only are many of the options suspiciously and weirdly repeated, but a fragment of what looks like a web page – content expected in an HTTP reply body, not in an OPTIONS response, which typically has no body content at all – has crept into the data sent back from the server.

What went wrong?

Böck noticed leaked data in some of the garbled replies that strongly suggested that Apache servers were involved. (Some of the leaked data apparently looked like Apache-specific configuration settings.)

With the help of Apache developer Jacob Champion, they got to the bottom of it, and the bug is intriguingly arcane – a strong reminder of how far-reaching the effects of a minor-sounding bug can be.

First, some background.

Apache servers can be configured by putting files called .htaccess into the directory tree of content that is stored on the server.

Each .htaccess file sets configuration options for the directory it’s in and all the others below it, unless their settings are overridden by another .htaccess file lower down, and so on.

Although this has the negative effect of scattering Apache’s configuration settings all across the server’s disk, it has the positive effect of safely letting one server, and one directory tree, serve many different websites (known in the jargon as virtual hosts) at the same time.

Whether the virtual hosts represent multiple departments inside the same organisation, or separate companies buying into a shared web hosting service, each customer can be given their own directory subtree.

The subtrees can be locked down to safe defaults at the top of the tree; each customer can then reconfigure their own part of the server to be even stricter if they want.

Rather than needing one computer, or one virtual machine, and one running copy of httpd for each customer, the hosting company can split up one high-powered server safely – in theory, at least – between numerous websites and customers.

One of the settings you can configure in your .htaccess files – Apache calls these settings by the rather bureaucratic name directives – is known as Limits, which, as the name suggests, limits the available OPTIONS in the current directory tree.

Like this, for example:

  <Limits POST PATCH PUT DELETE>
    Deny from all
  </Limits>

If you have a part of your website’s directory tree that is only there to serve up files, this sort of restriction makes sense – it’s a good example of what’s known in the trade as “reducing your attack surface”, and you do it for much the same reasons that you don’t let everyone login as an administrator all the time.

But here’s the thing: if any of the HTTP methods you configure in a directive are inapplicable, the “Optionsbleed” bug is triggered.

By inapplicable, we mean either than the method has already been turned off in the global httpd configuration settings, or that the method doesn’t actually exist, for example because you typed DELLETE instead of DELETE.

You’d think that this shouldn’t matter, beyond perhaps a warning in a logfile somewhere – banning an option that is already banned or turning off a non-existent option is a waste of time, but shouldn’t make things worse.

However, it seems that trying to set a Limit that doesn’t exist (or doesn’t apply) causes Apache to free up memory that it no longer needs…

…but to carry on referring to that memory, perhaps even after it has been allocated to and used by another part of the Apache program.

This sort of bug has the self-explanatory name of use-after-free, and you can see how these bugs lead to data leakage vulnerabilities: you can easily end up using someone else’s data, perhaps even copying some of their personal or private information, and sending it out over the network by mistake.

So, as far as Böck and Champion could tell, a memory mismanagement bug, provoked when Apache processes an .htaccess file that is meant to improve security…

…can end up reducing security by leaking data later on when a completely different part of Apache processes an OPTIONS request.

Thus the name Optionsbleed.

How bad is it?

The good news is that the side-effects of the bug don’t seem to show up often, given that it requires the coincidence of an incorrectly-configured .htaccess file and an unluckily-timed OPTIONS request.

Indeed, Böck’s tests provoked just 466 “optionbleeds” from 1,000,000 different web servers. (Statistics suggest that about 400,000 of those would have been running Apache, for a bug-trigger rate of about 0.12% of vulnerable systems.)

Nevertheless, it’s important to remember that on a server that’s hosting many different domains for many virtual hosts in many different directory trees, one malevolent customer could provoke this bug by deliberately setting an invalid option in their own .htaccess, and then repeatedly visiting one of their own URLs to see what data might leak out.

The leaked data comes from the memory of the Apache server software, and could in theory include content from other customers, or from the server itself.

Similarly, a well-meaning customer could ruin it for everyone else by copying-and-pasting an .htaccess file from an example configuration, and leaving redundant options in there on the very reasonable grounds – as we mentioned above – that turning off options that are already off should be harmless.

What to do?

Fortunately, there’s a fix available: a patch for Optionsbleed is available from the Apache source code servers.

If you outsource your servers or your web hosting, ask your provider if they can apply the patch for you.

Unfortunately, at the time of writing [2017-09-19T17:00Z], Apache hasn’t yet published an formal advisory or an officially-updated Apache version – for the time being, you’ll need to apply the code changes yourself and rebuild your own updated version of httpd.

Of course, with no official approval yet from Apache, it’s hard to judge if the currently-available patch is indeed the best and safest way to squah this bug.

So, if you can’t or don’t want to patch immediately, we suggest you visit all your .htaccess files looking for settings that aren’t applicable (or are mis-spelled), even though you shouldn’t really have to remove settings that are legal and purposeful but just happen to be redundant in the current configuration.

Whichever route you choose, keep your eye out for Apache’s next official security update – the current patch may be replaced, improved, extended or superseded.