Skip to content
Naked Security Naked Security

Apache “Optionsbleed” vulnerability – what you need to know

Remember Heartbleed, where servers could be tricked into letting other people's data slip? "Optionsbleed" is an Apache bug that's similar.

Remember Heartbleed?

That was a weird sort of bug, based on a feature in OpenSSL called “heartbeat”, whereby a visitor to your server can send it a short message, such as HELLO, and then wait a bit for the same short message to come back, thus proving that the connection is still alive.

The Heartbleed vulnerability was that you could sneakily tell the server to reply with more data than you originally sent in, and instead of ignoring your malformed request, the server would send back your data…

…plus whatever was lying around nearby in memory, even if that was personal data such as browsing history from someone else’s web session, or private data such as encryption keys from the web server itself.

No need for authenticated sessions, remotely injected executable commands, guessed passwords, or any other sort of sneakily labyrinthine sequence of hacking steps.

Crooks could just ask a server the same old innocent-looking question over and over again, keep track of the data that it let slip every time, and dig through it later on, looking for interesting stuff.

Well, something similar has happened again.

This time, the bug isn’t in OpenSSL, but in a program called httpd, probably better known as the Apache Web Server, and officially called the Apache HTTP Server Project.

The vulnerability has been dubbed OptionsBleed, because the bug is triggered by making HTTP OPTIONS requests.

If you know a bit about HTTP, you’ve probably heard of GET and POST requests, typically used to fetch data or web pages or to upload files or filled-in forms.

But there are numerous other HTTP request types, known officially as methods, including:

  • PUT: add a new item to a web database.
  • PATCH: edit an existing item in a web database.
  • DELETE: delete the specified item.
  • HEAD: like GET, but send only the headers and omit the body.
  • TRACE: echo back what was uploaded for debugging purposes.

Not all web servers support all the many official methods, so there’s a special method to help you out:

  • OPTIONS: report the methods that are allowed for a specific web page.

By using OPTIONS you can avoid hammering a web server with requests that are never going to work, thus avoiding frustration at your end of the connection, and saving the server from wasted effort at the other.

For example:

  $ wget -S --method=OPTIONS https://my.example/index.html
  HTTP request sent, awaiting response... 
    HTTP/1.1 200 OK
    Allow: OPTIONS, TRACE, GET, HEAD, POST
    Public: OPTIONS, TRACE, GET, HEAD, POST
    Content-Length: 0
    Date: Tue, 19 Sep 2017 14:09:26 GMT
  $

Here, you can see the methods that the my.example server accepts, listed in the Allow: header.

Harmless enough, you might think – and that’s how it ought to be.

But software and security researcher Hanno Böck found otherwise.

He set out to measure what OPTIONS were supported by the Alexa Top 1,000,000 websites, asking each of them in turn..

Böck spotted that a small but noticeable fraction of the servers sent back garbled or corrupted-looking responses, such as:

  Allow: ,GET,,,POST,OPTIONS,HEAD,,
  Allow: POST,OPTIONS,,HEAD,:09:44 GMT
  Allow: GET,HEAD,OPTIONS,=write HTTP/1.0,HEAD,,HEAD,POST,,HEAD,TRACE

One that Böck lists looked curiously suggestive of a “bleed-type” data leak:

  Allow: GET,HEAD,OPTIONS,,HEAD,,HEAD,,HEAD,, 
    HEAD,,HEAD,,HEAD,,HEAD,POST,,HEAD,, HEAD,!DOCTYPE 
    html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"

Not only are many of the options suspiciously and weirdly repeated, but a fragment of what looks like a web page – content expected in an HTTP reply body, not in an OPTIONS response, which typically has no body content at all – has crept into the data sent back from the server.

What went wrong?

Böck noticed leaked data in some of the garbled replies that strongly suggested that Apache servers were involved. (Some of the leaked data apparently looked like Apache-specific configuration settings.)

With the help of Apache developer Jacob Champion, they got to the bottom of it, and the bug is intriguingly arcane – a strong reminder of how far-reaching the effects of a minor-sounding bug can be.

First, some background.

Apache servers can be configured by putting files called .htaccess into the directory tree of content that is stored on the server.

Each .htaccess file sets configuration options for the directory it’s in and all the others below it, unless their settings are overridden by another .htaccess file lower down, and so on.

Although this has the negative effect of scattering Apache’s configuration settings all across the server’s disk, it has the positive effect of safely letting one server, and one directory tree, serve many different websites (known in the jargon as virtual hosts) at the same time.

Whether the virtual hosts represent multiple departments inside the same organisation, or separate companies buying into a shared web hosting service, each customer can be given their own directory subtree.

The subtrees can be locked down to safe defaults at the top of the tree; each customer can then reconfigure their own part of the server to be even stricter if they want.

Rather than needing one computer, or one virtual machine, and one running copy of httpd for each customer, the hosting company can split up one high-powered server safely – in theory, at least – between numerous websites and customers.

One of the settings you can configure in your .htaccess files – Apache calls these settings by the rather bureaucratic name directives – is known as Limits, which, as the name suggests, limits the available OPTIONS in the current directory tree.

Like this, for example:

  <Limits POST PATCH PUT DELETE>
    Deny from all
  </Limits>

If you have a part of your website’s directory tree that is only there to serve up files, this sort of restriction makes sense – it’s a good example of what’s known in the trade as “reducing your attack surface”, and you do it for much the same reasons that you don’t let everyone login as an administrator all the time.

But here’s the thing: if any of the HTTP methods you configure in a directive are inapplicable, the “Optionsbleed” bug is triggered.

By inapplicable, we mean either than the method has already been turned off in the global httpd configuration settings, or that the method doesn’t actually exist, for example because you typed DELLETE instead of DELETE.

You’d think that this shouldn’t matter, beyond perhaps a warning in a logfile somewhere – banning an option that is already banned or turning off a non-existent option is a waste of time, but shouldn’t make things worse.

However, it seems that trying to set a Limit that doesn’t exist (or doesn’t apply) causes Apache to free up memory that it no longer needs…

…but to carry on referring to that memory, perhaps even after it has been allocated to and used by another part of the Apache program.

This sort of bug has the self-explanatory name of use-after-free, and you can see how these bugs lead to data leakage vulnerabilities: you can easily end up using someone else’s data, perhaps even copying some of their personal or private information, and sending it out over the network by mistake.

So, as far as Böck and Champion could tell, a memory mismanagement bug, provoked when Apache processes an .htaccess file that is meant to improve security…

…can end up reducing security by leaking data later on when a completely different part of Apache processes an OPTIONS request.

Thus the name Optionsbleed.

How bad is it?

The good news is that the side-effects of the bug don’t seem to show up often, given that it requires the coincidence of an incorrectly-configured .htaccess file and an unluckily-timed OPTIONS request.

Indeed, Böck’s tests provoked just 466 “optionbleeds” from 1,000,000 different web servers. (Statistics suggest that about 400,000 of those would have been running Apache, for a bug-trigger rate of about 0.12% of vulnerable systems.)

Nevertheless, it’s important to remember that on a server that’s hosting many different domains for many virtual hosts in many different directory trees, one malevolent customer could provoke this bug by deliberately setting an invalid option in their own .htaccess, and then repeatedly visiting one of their own URLs to see what data might leak out.

The leaked data comes from the memory of the Apache server software, and could in theory include content from other customers, or from the server itself.

Similarly, a well-meaning customer could ruin it for everyone else by copying-and-pasting an .htaccess file from an example configuration, and leaving redundant options in there on the very reasonable grounds – as we mentioned above – that turning off options that are already off should be harmless.

What to do?

Fortunately, there’s a fix available: a patch for Optionsbleed is available from the Apache source code servers.

If you outsource your servers or your web hosting, ask your provider if they can apply the patch for you.

Unfortunately, at the time of writing [2017-09-19T17:00Z], Apache hasn’t yet published an formal advisory or an officially-updated Apache version – for the time being, you’ll need to apply the code changes yourself and rebuild your own updated version of httpd.

Of course, with no official approval yet from Apache, it’s hard to judge if the currently-available patch is indeed the best and safest way to squah this bug.

So, if you can’t or don’t want to patch immediately, we suggest you visit all your .htaccess files looking for settings that aren’t applicable (or are mis-spelled), even though you shouldn’t really have to remove settings that are legal and purposeful but just happen to be redundant in the current configuration.

Whichever route you choose, keep your eye out for Apache’s next official security update – the current patch may be replaced, improved, extended or superseded.


21 Comments

Ouch.
(1) My web clients lack admin access to their sites (removing the malicious customer scenario), so this alleviated some urgency for me when it yielded a null result:
find [DocumentRoot] -name “.htaccess*” -exec grep -i limits {} \;

(2) Estimated 200,000 of 1M? I thought Apache’s share was larger than that.

Reply

If you take a look at the link “Statistics suggest” you’ll see that apache’s preeminence is over. Nginx is hot on its heels and, somehow, Microsoft is now far out in front (Azure maybe?)

Reply

Hmmm. I went and revisited the “Stats suggest” link myself and I have now changed my mind :-) My re-reading of the stats is that they suggest that Apache’s httpd is running on about 20% of all visible sites, but on 40% “of the top 1,000,000 sites”. Admittedly, those are Netcraft’s “top million”, not Alexa’s (the Alexa top million list no longer seems to exist), but I nevertheless decided to change my “200,000 out of 1M are probably vulnerable” assumption to “400,000 out of 1M”, and thus my bug-triggering rate from 466/200K to 466/400K, or 0.12%.

Lies, damn lies and statistics…so take the numbers with a pinch of ionic crystal lattice…but at least I have shown my working :-)

Reply

*DOH*
And here I’ve been telling people Dad taught me to read.

Well I suppose that IIS was bound to take the lead seeing as how much better it is at security, manageability, and reliability. :-/

Reply

Re-run your search, using limit instead of limits. There’s an error in the Apache directive sample.

It should read:

<Limits POST PATCH PUT DELETE>
Deny from all
</Limits>

Using Limits in a .htaccess file causes a 500 Internal Server Error.

Reply

It’s pretty much considered a best practice to disable the OPTIONS method altogether. My question is: if OPTIONS is disabled in httpd.conf as well as in .htaccess (which would trigger the bug for any other method) does it still bleed?

Reply

I am afraid I don’t know. What I do know (I saw it when testing) was a site that responded to an OPTIONS method request with an “HTTP Error 405 – Method Not Allowed”…

…followed by *exactly the same headers that you’d see if OPTIONS were permitted*, in order to let you know what methods you could have used.

In other words, with OPTIONS turned off, you still get an OPTIONS-type response with an Allow: list, but OPTIONS itself is not in the list.

I have no idea if that site was running Apache, but I certainly formed the opinion that the code paths were very likely to be the same – and thus that OPTIONS requests would always be processed, and only the error code returned would differ.

If that site was running Apache, and my inference is correct about the code paths being the same, then you might further infer that the answer to your question is yes. (It depends on whether .htaccess is consulted if there are already reasons to report an error, I guess.)

I don’t have the infrastructure to test this right now – anyone got a vulnerable test server on which they’ve been able to provoke this bug, and who is willing to try the suggested experiment? (Pair of Sophos Blue Screen of Death socks if you help us :-)

Reply

I just tested this on a fresh digitalocean server Apache/2.4.6 (CentOS) with OPTIONS disabled in httpd.conf as well as in the .htaccess, and yeah, with options disabled, it just treats it as a GET.

With options Enabled:
curl -XOPTIONS 165.227.217.211 -v
* Rebuilt URL to: 165.227.217.211/
* Trying 165.227.217.211…
* Connected to 165.227.217.211 (165.227.217.211) port 80 (#0)
> OPTIONS / HTTP/1.1
> Host: 165.227.217.211
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 20 Sep 2017 17:16:11 GMT
< Server: Apache/2.4.6 (CentOS)
< Allow: OPTIONS,GET,HEAD,POST
< Content-Length: 0
< Content-Type: httpd/unix-directory
OPTIONS / HTTP/1.1
> Host: 165.227.217.211
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 20 Sep 2017 17:28:38 GMT
< Server: Apache/2.4.6 (CentOS)
< Last-Modified: Thu, 16 Oct 2014 13:20:58 GMT
< ETag: "1321-5058a1e728280"
< Accept-Ranges: bytes
< Content-Length: 4897
< Content-Type: text/html; charset=UTF-8
<

With options disabled in httpd.conf and .htaccess:
curl -XOPTIONS 165.227.217.211 -v
* Rebuilt URL to: 165.227.217.211/
* Trying 165.227.217.211…
* Connected to 165.227.217.211 (165.227.217.211) port 80 (#0)
> OPTIONS / HTTP/1.1
> Host: 165.227.217.211
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 20 Sep 2017 17:28:38 GMT
< Server: Apache/2.4.6 (CentOS)
< Last-Modified: Thu, 16 Oct 2014 13:20:58 GMT
< ETag: "1321-5058a1e728280"
< Accept-Ranges: bytes
< Content-Length: 4897
< Content-Type: text/html; charset=UTF-8
<

Reply

While this seems a good test-case, it actually does not touch an interesting situation that Paul mentioned above. This illustrates that disabling OPTIONS returns “403 Forbidden” response. However, if a configuration could cause “405 Method Not Allowed” response, we expect to see all the allowed methods listed in the “Allow” header, even if OPTIONS is disabled, because OPTIONS is not the only method that can return “Allow” header. RFC2616 says, “An Allow header field MUST be present in a 405 (Method Not Allowed) response”. There are many other situations where an “Allow” header can be used to ask for or to convey the capabilities of the server on a resource. I have a feeling that the exploit is not really “Optionsbleed”, but it is “Allowbleed” instead.

Reply

I’ve tried this on some of my sites – of the three servers (hosts) I use, one gives a 405 and the other two treat it like an ordinary GET request (and download the html for the page requested). The first one could well be a Nginx server and the others are definitely Apache.

Reply

I will leave here the link of our 2014 paper, “Support for Various HTTP Methods on the Web” [link removed] for the reference. The issue is described in section 5.3 of the paper.

Reply

Should have known better, trying to put a link on a security blog!

Reply

Well, it is a publicly available pre-print paper, so searching for the title should land anyone to the full-text. However, I left the link here because this article seemed like one of the very few articles on this matter that went into details with enough background. I thought, the author might want to update the article to acknowledge readers that this issue was observed three years ago. I have seen some other security/tech news sites have done that.

Reply

Hmmm. Thanks for the history – looks like you guys stumbled across this bug back then without realising what you had found – one of the replies you list shows a not-dissimilar corruption to an example above:

Allow: GET, HEAD, POST, PUT, DELETEtext/html; charset=iso-8859-1

If I had been in your position I think I’d have reached a similar conclusion to you – that at least some servers were sloppy about constructing “Allow:” lists (one was all in lower case, another had spaces instead of commas and a third had a few rogue command characters), and that’s all.

I don’t think I’d have guessed at a use-after-free flaw. After all, your paper was to investigate the whole panoply of how servers supported various features, not to assess exactly how perfectly they had implemented the protocol of the OPTIONS method. Also, in your examples there is nothing that really looks like a *bleed* – at least, there is no leaked data that seems to have misplaced HTTP *body* data into the headers, just a few odd bytes and that Content-Type: header text in the Allow: list.

But nevertheless – I for one will never write off a rogue control character in a logfile as a “typo” or an inconsequential corruption again…bet you won’t, either :-)

Now here’s a question: how many apps or clients ever even use OPTIONS? How many use it and reply on the result in any purposeful way?

Reply

Your assessment is quite accurate. All credits go to Böck to realize that it was a software flaw, not just some copy-paste error in a config file. However, in our paper we did note various oddities, categorized them, illustrated them with hand-picked examples, and quantified them. In the process we did confirm from the relevant RFCs and other credible sources what is valid and what is invalid. Lower-case method name is one example which seems OK, but it is not a valid method name. On the other hand, we have observed in many instances method names were separated by multiple commas, which looks odd, but turns out to be perfectly valid.

I think we did observe pieces of HTML body in that header, not just the partial content-type as illustrated in the paper. However, we just hand-picked one example from each category of malformed header for illustration. In retrospect it seems very obvious that we should have noted that as a use-after-free memory bleed, but as you said, we didn’t really guess it, so did not pay much attention to its details. Besides, the paper topic was about analyzing usage statistics of various HTTP methods, hence, there was a limit how much we could have deviated from the topic. There is another story why we did this research in the first place, but I will leave that for some other time.

As far as your question about the usage potential of the OPTIONS method is concerned, it is actually being utilized more often these days. There are some applications that interact with REST APIs, where they do use OPTIONS to realize capabilities of the service. However, the most common use of OPTIONS happens every single day from almost every web browser behind the scene numerous times. Every time a web page makes a Cross-Origin Resource Sharing (CORS) request which does not fall in the “simple” category, the browser makes a “preflight” request which is essentially an OPTIONS request for the same URI, which is like asking for permission for the actual request. As our web services become more and more machine-friendly, we expect to see more utilization of niche HTTP methods, status codes, and headers.

Reply

Modern browsers use OPTIONS quite frequently to handle CORS requests (checking the remote site with a “safe” query to see if it allows cross-origin requests from a particular domain before making a GET/POST to that remote server). Possibly WebDAV… just off the top of my head… otherwise I think it’s pretty arcane, most things talking over HTTP already know the methods available to use based on API specs, an HTML form attribute, etc.

Reply

> @Wes: otherwise I think it’s pretty arcane, most things talking over HTTP already know the methods available to use based on API specs, an HTML form attribute, etc.

This is not how REST purists and people behind the HTTP think though. They talk about HATEOAS which is essentially following your nose starting from one well-known entry point using hypermedia and relations, without having any out-of-band knowledge about the service. The OPTIONS method can play a big role when a service architecture is designed for such interactions.

Reply

This is fixed in apache 2.4.27, you might want to update this post.

Reply

You sure? The patch I linked to above, which seems to relate to this issue (AFAICS it stops .htaccess settings from being processed if they’re already set, though I can’t see where the leaky free is dealt with, if at all) is dated 2017-09-08 but 2.4.27 came out back in July. Have I misread/misunderstood something here?

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe to get the latest updates in your inbox.
Which categories are you interested in?
You’re now subscribed!