“Chrome considered harmful” – the Law of Unintended Consequences

An excellent article appeared last week on the APNIC blog.
Researched and written by Matthew Thomas of Verisign, the article is entitled Chromium’s impact on root DNS traffic, and it has raised some important issues amongst the Chromium browser development community relating to a feature in the browser code that’s known as the Intranet Redirect Detector.
To explain.
APNIC is the Asia Pacific Network Information Centre, headquartered in Brisbane, Australia, one of five internet number registries around the world.
These Regional Internet Registries (RIRs) look after global IP number allocations, maintain definitive internet domain name databases for their regions, and generally concern themselves with the health of the global internet.
As you can imagine, anything that upsets the balance of the internet – from spamming and cybercrime to misconfigured servers and badly-behaved network software – is of great concern to the RIRs.
The root DNS servers form the heart of the global Domain Name System, which automatically converts human-friendly server names such as nakedsecurity.sophos.com into network numbers that computers can use to send and receive traffic, such as 192.0.66.200 (that was our IP number when I looked it up today, as shown below).
As you can imagine, any unnecessary load on the root DNS servers could slow down internet access for all of us, by stretching out the time taken convert names to numbers, something that our browsers need to do all the time as we click from link to link online.
Chromium, as you almost certainly know, is a Google open-source project that produces the software at the core of many contemporary browsers, notably Google’s own Chrome Browser, which accounts for the majority of web traffic these days on laptops and mobile phones alike.
Chromium is also used in many other browsers, including Vivaldi, Brave and – recently, at least – Microsoft Edge. (Of today’s mainstream browsers, only Safari and Firefox aren’t based on a Chromium core.)
As you can imagine, any problematic code in Chromium could have an enormous global effect due to the prevalence of Chromium-based browsers in modern internet usage.
And last, but not least, the Intranet Redirect Detector is a “feature” added to the Chromium browser that is supposed to detect, and work around, a deceptive practice known as NXDOMAIN redirection (or, pejoratively, as NXDOMAIN hijacking) that is still used by some ISPs in some countries.

Domain redirection explained

Very generally speaking, DNS redirection is where an ISP or network provider doesn’t tell you the unadorned truth about a server name you want to locate online.
Sometimes, this sort of purposeful inaccuracy can be a good thing, if you’re aware that it might happen and if it’s done sensitively.
An obvious example is for the purposes of security filtering, where a network security device or cloud service deliberately redirects known bad domains, such as malware repositories, thus heading off potentially malicious traffic right at the DNS level.
A more controversial example of DNS redirection involves censorship, where a country’s government demands that ISPs kill off traffic to all sorts of sites that the authorities regard as “dangerous”, not only if those sites might lead to malware or other unexceptionably nasty content, but also if those sites present facts and opinions that the state wants to hide from its citizens.
But NXDOMAIN redirection is another thing altogether.
Simply put, a DNS lookup for a server name that doesn’t exist at all, and therefore can’t be resolved, is supposed to come back with a DNS error 3, known as NXDOMAIN, short for non-exsistent domain.
Here is some Lua script code that we used to do two DNS lookups and retreive the low-level results, using Google’s free and uncensored DNS server located at IP address 8.8.8.8.
First, we looked up a DNS record of type A (short for IP address) for nakedsecurity.sophos.com and received a chain of two answers:

   > dns = require 'dns.resolver'
   > lookup = dns.new('8.8.8.8')
   > lookup:resolveRaw('nakedsecurity.sophos.com') -- returns list of answers
                                                   -- if domain resolves
   results = {
      questions = {
         1 = {
            type = "A"
            name = "nakedsecurity.sophos.com"
         }
      }
      answers = {
         1 = {
            type = "CNAME"
            name = "nakedsecurity.sophos.com"
            content = "news-sophos.go-vip.net"
            TTL = 60
         }
         2 = {
            type = "A"
            name = "news-sophos.go-vip.net"
            content = "192.0.66.200"
            TTL = 60
         }
   }

The first answer is a CNAME (short for canonical name), which says that nakedsecurity.sophos.com is actually an alias for the server where our content is stored, namely news-sophos.go-vip.net, so the IP number of the go-vip.net site is actually the one we want.
The second answer then chases down the A record of the relevant go-vip.net server, giving us the answer 192.0.66.200, which is where we need to connect to access Naked Security’s content.

TTL, by the way, is short for time to live, and denotes how long, in seconds, we can assume the reply is correct before we need to look it up again. A browser, for example, might need to fetch 10 or 20 different items from the news-sophos.go-vip.net server, such as stylesheeets, JavaScripts, images and HTML files, just to display one web page. By telling our computer that it can rely on the original DNS reply for up to a minute before checking again, load on the DNS system is greatly reduced. But with a suitably small TTL value, stale DNS entries don’t last long and soon get corrected automatically. TTLs typically vary from 60 seconds for cloud services that need to adapt quickly to changes in availability, to several hours for static servers that handle little traffic.
Second, we asked about a made-up name that doesn’t exist:

   > dns = require 'dns.resolver'
   > lookup = dns.new('8.8.8.8')
   > lookup:resolveRaw('iurwcnurewcnurc.example')  -- function returns nil,error,errno
                                                   -- if not resolved
   results = nil, "NXDOMAIN", 3

As you can see, the DNS server quickly and definitely told us, “There’s no such site, so there’s no point in trying to access it.”
Censorship, of course, could be achieved by keeping a list of domains you don’t like and deliberately, if dishonestly, “downgrading” their true answers to NXDOMAIN errors, thus making it look as though the sites don’t exist at all.
But it’s hard to imagine why you might want to perform this trick the other way around and pretend that a non-existent domain really does exist.
However, many years ago a number of ISPs began to do just that, treating all NXDOMAIN replies as missed opportunities for advertising.

NXDOMAIN hijacking

If you mistyped a domain name, for example, an ISP could use a DNS hijack to send you to one of its own portals instead, and offer you a range of alternative webpages or products when you got there.
If the ISP figured that the domain name you were interested in might actually be worth some money, and therefore that you might be checking to see if it was in use, then it could pop up a page offering to sell you the domain right away.
Instead of “downgrading” a real site to non-existent status, as a censorship-happy government might do, NXDOMAIN hijacks go the other way, “upgrading” domains that don’t exist so that they appear to be there after all.
The problem, of course, is that this makes it impossible to tell if a domain really exists or not, because every server name magically works, even when it shouldn’t.
Sadly, there aren’t any globally-agreed internet regulations that the RIRs or ICANN – the Internet Corporation for Assigned Names and Numbers – can use to stamp out this practice, but for more than a decade, ICANN’s official opinion has been that:

[We] strongly [discourage] the use of DNS redirection, wildcards, synthesized responses and any other form of NXDOMAIN substitution.

Indeed, ICANN criticises NXDOMAIN hijacking as producing a poor user experience, because it means you can no longer easily tell whether a domain really exists or not, and as being a potential abuse of power, given that end users who want to resolve domain names to their own servers have to pay for each domain individually.

Problems for browsers

Interestingly, the creators of the original Chromium code faced a problem of their own with NXDOMAIN trickery, namely that they wanted their new browser’s address bar to be a search engine at the same time.
Instead of typing in a URL, you could just type in a word to search for, and Google – whose primary business, after all, is online search and advertising – would treat it as a search term.
However, many companies use unadorned domain names internally, so that staff who are in the office (or, in coronavirus times, are connected via VPN) – can use single-word names to reach internal corporate servers.
Chromium, therefore, checks to see if any “search term” in the address bar also works as a domain name, so that it won’t accidentally stop you accessing servers that do exist on your intranet.
But if you’re using an ISP with DNS servers that do NXDOMAIN hijacking, then every domain name you try will seem to exist, and Chromium won’t usefully be able to distinguish between search terms and domain names.
In other words, domain hijacking behaviour seen as a “feature” by ISPs turned out to get in the way of the autosearch “feature” that Google wanted to implement in the Chromium code.
So, as Matthew Thomas explained in the research we referenced at the start, the Chromium programmers created yet another “feature” to help them decide whether to enable their autosearch “feature”, and coded it up in a file called intranet_redirect_detector.c.
What this code does is to try three randomly-generated domain names when the browser starts up, as our own local DNS logs show here:

[Chrome run 1]
DNS/query: type=A name=boyhfqrconuryg source=127.0.0.1
DNS/query: type=A name=bhephhtodjgghex source=127.0.0.1
DNS/query: type=A name=akomcbscjhqzt source=127.0.0.1
[Chrome run 2]
DNS/query: type=A name=sgfagnzojxpdou source=127.0.0.1
DNS/query: type=A name=mbwuvkac source=127.0.0.1
DNS/query: type=A name=xaswmuofxyda source=127.0.0.1
{Chrome run 3]
DNS/query: type=A name=bjgqwiedftoskk source=127.0.0.1
DNS/query: type=A name=wmrwezznxjxmma source=127.0.0.1
DNS/query: type=A name=skwqumuedmf source=127.0.0.1

If two (or all) of these unlikely domains resolve, and come back with the same IP number, Chromium infers that NXDOMAIN hijacking is taking place.
In other words, if the intranet_redirect_detector triggers, then the browser assumes it can’t rely on DNS lookups to tell the difference between words you probably meant to search for, and servers you probably wanted to browse to.

Needless lookups considered harmful

As cheeky as the Chromium code might seem, three random DNS lookups every time you open your browser doesn’t sound like a lot.
After all, many people don’t even restart their browser once a day, preferring to keep apps open as long as they can, and rarely logging off or shutting down their computers.
But, as Thomas points out, on a network where there is no NXDOMAIN hijacking, those random DNS lookups all have to be handled by the root DNS servers.
The made-up domains names don’t exist, and therefore won’t be in anyone’s DNS caches, and will always require your computer’s DNS resolver to check right back to the mother ship – one of the 13 root DNS servers – for an answer that annoyingly serves no purpose except to help Chromium configure its autosearch “feature”.
And given that Chromium’s code is running on the majority of laptops and phones out there – that’s an absolute majority, in the sense of “running more than 50% of all laptops and phones in the world”, not merely “running on more devices than any other browser code” – then it turns out that those randomised test lookups are giving the root DNS servers a huge amount of extra work to do…
…none of which has anything to do with finding the location of any real servers, which is the main purpose of DNS.
Even worse, as NXDOMAIN hijacking becomes less common, thanks to pressure on ISPs not to do it, Chromium’s pressure on the root DNS servers will increase, because more and more of the random intranet_redirect_detector lookups will end up making it all the way to those 13 root servers, only to produce errors.

What to do?

The good news is that the Chromium development list seems to have accepted that this autosearch detection feature has triggered what you might call the Law of Unintended Consquences, and represents bad publicity for Chromium and Chrome alike.
Of course, just dropping this the “feature” entirely from the Chromium code would probably be the quickest solution…
…but that itself would very likely trigger the Law of Unintended Consequences once more by introducing confusion amongst users who have been relying on the feature, probably without even realising it.
We therefore suspect that the solution will have more than one part, perhaps including:

1. Make the NXDOMAIN connection process less demanding on the root servers, by finding a way to perform it less frequently.
2. Make the autosearch detection feature optional but leave it on by default and provide a way for users who care about DNS traffic to turn it off.
3. If problems persist, turn the feature off by default but provide a way for those who want it to turn it back on.

Avoiding the Law of Unintended Consequences in cybersecurity is very tricky…
…but having a community that is willing to listen and react to well-reasoned articles like the one from Matthew Thomas is an important part of its solution!