Microsoft has fixed a bug that could have led to distributed denial of service (DDoS) attacks on its web server software.
The flaw lay in the way that Internet Information Server (IIS) processed requests sent using HTTP/2.
Ratified in 2015, HTTP/2 is an enhanced version of the original HTTP standard that includes better flow control and handles a wider variety of connections between clients and servers.
Flow control in HTTP/2 enables a client computer to describe how it wants to receive information from the sender so that it can work more efficiently.
For example, you might ask your browser to stream a high-bandwidth video, but then pause the video halfway through.
With HTTP/2, the browser can use flow control to pause the delivery and buffering of the video and concentrate on downloading something else that is suddenly more important, such as a real-time ticker update.
To manage flow control, HTTP/2 uses a feature known as a SETTINGS frame.
Clients can specify any number of SETTINGS frames, and this is the root of the problem that Microsoft found in IIS – too many frames can overload the server, maxing out CPU usage at 100%.
Microsoft reported:
In some situations, excessive settings can cause services to become unstable and may result in a temporary CPU usage spike until the connection timeout is reached and the connection is closed.
The flaw meant that attackers with a botnet of zombie computers, or hacktivists with a following of willing helpers, could have brought IIS servers – which as of January 2019 hosted 25% of all web domains, according to Netcraft – to their knees.
Microsoft fixed the problem by adding an option to limit the number of SETTINGS frames in an HTTP/2 request.
What to do?
To access this feature, customers can download the cumulative updates KB4487006, KB4487011, KB4487021, and KB4487029.
The fix allows administrators to set two parameters in the registry: Http2MaxSettingsPerFrame and Http2MaxSettingsPerMinute.
If the number of SETTINGS frames surpasses either of these two limits, IIS will kill the connection:
When appropriately set, [the] two limits together help to terminate the malicious connection violating those limits and form a threshold for legitimate connections.
Don’t forget, though, that these settings aren’t turned on by default, even after you install the update – a suitable registry tweak is needed to enable this DDoS mitigation.
Bill Justesen
Thanks for the heads up about the registry tweaks. When I performed a search for all four Knowledge Base (KB) articles, there wasn’t even a mention in any of them that there were still pro-active measures to be taken to mitigate the DDOS issue. If it weren’t for you Naked Security, I’d have thought I was protected. Seriously, thank you! (And personal thank you to Danny Bradbury for the article and digging up the full mitigation.)
In attempting to determine good numbers to use, I found a Reddit thread that discussed this issue. The point was made that the MaxSettingsPerFrame should be set between 7 (min) and 2796202 (max), and that a value of 7 would likely suffice for now, unless “you expect the HTTP/2 spec to support more settings frame parameters in the future, give some higher value that’s not high enough to cause the issue. If you’re going that route, you’ll want to be able to reliably reproduce the bug in a test environment so you actually understand the impact of the limit you’re setting.”
The Http2MaxSettingsPerMinute allows a minimum value of 7, but doesn’t apparently set a maximum. It was suggested to determine the maximum expected connections per minute, and then multiply that value by the MaxSettingsPerFrame. Since we’re a fairly small IIS shop, I can’t imagine we’d be hitting anywhere near 500 individual connections per minute as discussed in the thread. (I know that by reviewing the logs too.) I’ll just determine a good number and go from there. If anyone complains, then I’ll have to look and see if they are getting cut off or if there is some other issue.
Danny Bradbury
Thanks for the feedback, Bill, and I’m glad it helped. Thanks for the extra information, too.
Laurence Marks
The big problem will be that you set the parameters now. Three years from now, when your replacement has taken over and traffic has increased and server reliability becomes impacted. No one will be able to figure out why. If you’ve got a short memory, this could happen even if you stay on the job.
People write these protocols without ever thinking about unintended consequences.
Jeff Stebelton
As far as I can find, this only applies to Server 2016. That track with what you found?