Deutsche Telekom outage: Mirai botnet goes double-rogue

We’ve written about the Mirai DDoS botnet before.

Now it’s back in the news again, after apparently causing trouble for close to 1 in 20 users of Deutsche Telekom in Germany.

Here’s the story as we understand it.

First, some terminology.

A DDoS is a distributed denial of service attack, where crooks persuade or trick thousands of devices into simulataneously sending redundant internet traffic to a victim’s server to bog it down.

A botnet is a robot network: a collection of infected online devices, which could be laptops, servers, phones, routers, webcams, or any connected device that can run programs and send data across the internet.

Every so often, perhaps every few minutes, or even every few seconds, each robot in the network connects to a server controlled by cybercriminals to fetch instructions on what to do next.

Those instructions vary all the way from “send spam to all these addresses“, through “sneakily take pictures using the webcam and upload them”, to “start blasting this victim’s website with denial-of-service traffic”.

Zombies on the Internet of Things

Until recently, most bots, also commonly called zombies for obvious reasons, ran on regular computers, such as the desktop or laptop you’ve probably got at home.

But the recent Mirai botnet runs on so-called Internet of Things (IoT) devices such as routers, webcams and even printers.

Those might seem like unlikely tools for cybercrooks to use to attack other people, but infecting IoT devices for DDoS purposes turns out to work rather well, because:

• Many IoT devices are poorly secured, shipping straight from the manufacturer with security holes that make them easy to infect.
• Most IoT devices are powerful enough to flood the outbound link a home network connection with time-wasting network traffic, even though they have only the fraction of the computing power of the average laptop.
• Many IoT devices are designed to connect automatically (e.g. via Wi-Fi), so they end up with default configuration settings that are insecure and never get changed.

Mirai changed the game not only by using IoT devices as zombie attack bots instead of relying on desktops and laptops, but also by introducing a “go out looking for new zombies” feature.

Additionally, after a widely-publicised attack against cybersecurity journalist Brian Krebs, the source code of the Mirai malware was published so that anyone could have a go at running a botnet – known as being a botmaster or botherder.

In short, Mirai has both an “attack now” part that focuses traffic from an infected device onto some hapless victims server, and a “go looking” part that sprays out traffic from an infected device in the hunt for other insecure devices in the neighbourhood.

In other words, a crook who controls a Mirai botnet can use it to not only to mount today’s attack, but also to go out probing for additional IoT devices to co-opt into tomorrow’s attacks.

Probing the internet

Loosely speaking, there are three outcomes when botherders actively probe other devices to look for a security hole to exploit:

1. The device is vulnerable and thus ends up co-opted into the new botnet, and contributes to tomorrow’s problem.
2. The device is immune, and the probe fails.
3. The device neither complies nor resists, but instead misbehaves and crashes, effectively DoSsing the owner of the device.

(Outcome 3 above is why network hacking and penetration testing require explicit permission and careful planning. Actively probing for security holes can have unintended and dangerous side effects, so doing it without authorisation is quite rightly a criminal offence in many countries.)

The Deutsche Telekom problem

If the device being probed is a home router, and suffers outcome (3) above, the router will probably end up cut off from the internet, unable to pass traffic in either direction and possibly unable to reconnect to the internet until it is rebooted.

Of course, it will then only stay up until the next time it gets probed by chance, crashes again, goes offline, reboots, reconnects, and so on.

As far as we can tell, that’s what happened to many customers of Deutsche Telekom in Germany over the weekend.

According to a statement from the company, about 900,000 of the 20 million routers in use by its customers were prone to locking up when probed by a particular variant of the Mirai botnet.

Ironically, it seems as though the fact that the routers couldn’t be infected caused them to be affected instead, with about 4% of Deutsche Telekom customers knocked offline and prevented from reconnecting.

What to do?

According to Deutsche Telekom, the company has added various networking packet filtering rules in its core network, through which traffic to and from its customers passes.

By identifying and stripping out the traffic that can crash affected routers, the company hopes to reduce and perhaps even to eliminate the “you’ve been booted offline” problem, because the probe packets causing the crashes won’t reach their destinations.

So, rebooting your router now (even if you’ve tried this before without success) ought to allow it to reconnect and stay connected to the now-filtered network.

Of course, this is only a workaround; a full-blown fix will almost certainly involve a firmware update to your router.

We therefore suggest that you keep your eyes open for the next update to your router firmware.

If you don’t know where to look, try asking the vendor of the router, or your ISP if they supplied your router.