The Dark Web: anarchy, law, freedom and anonymity

The Deep Web, the bit of the World Wide Web that’s not indexed by search engines like Google and Bing, is of intense interest to people who want to avoid government spies and law enforcement.

It is a lawless cyber-frontier with similarities to the Old West; it is intrinsically neither good nor bad but it holds particular attraction for pioneers because its resources haven’t been fully explored, and it holds particular attraction for criminals because they can get away with doing things there that they can’t do elsewhere.

Like the Old West when it was everything west of the Mississippi river, the Deep Web appears to be larger than the territory that’s already been settled but genuine outposts of activity are probably quite sparse and widely separated.

For the most part, the fact that sites in the Deep Web don’t appear in Google results is a reflection of Google’s commercial priorities and indexing methods rather than anything sinister.

Most of the Deep Web is dark for the same reason that the Old West was dark – it hasn’t been worth anyone’s while to install good lighting.

A small corner of the Deep Web is really dark though – so dark that it’s called the Dark Web – because fundamental things like who you are and where you or the website you’re using is located are a secret.

This is the domain created by tools like Tor and I2P that provide ways to interact that are difficult to discover, and are anonymous and untraceable.

The Dark Web is many things to many people – it can be a safe haven and a secure communication channel for spies, citizens, journalists and whistle-blowers for example – but for the worst criminals it is the safest place to conduct their business online.

It attracts people who want to engage in things like robbery to order, sex trafficking, arms trafficking, terrorism and distributing child pornography.

How to combat those people is a huge challenge for law enforcement but, just as it was in Oregon and the Black Hills, their arrival is inevitable.

A report published by the Global Comission on Internet Governance entitled The Impact of the Dark Web on Internet Governance and Cyber Security proposes six areas that law enforcement efforts should focus on.

The paper is written by a couple of heavy hitters; Michael Chertoff has been a judge in the US Court of Appeals and was secretary of the US Department of Homeland Security from 2005 to 2009. Co-author Tobby Simon is president of The Synergia Foundation, a think tank focused on geo-economics and geo-security.

Although their report is deliberately focused on the negative impact of the Dark Web they offer an even-handed view, describing it in non-judgemental terms and detailing both its capacity to save lives and the extreme criminality that finds safe haven in it.

Their words on anonymity are worth repeating at length, to put their later recommendations in context:

Like any technology, anonymity can be used for both good and bad purposes. Many people do not want the things they say online to be connected with their offline identities. They may be concerned about political or economic retribution, harassment or even threats to their lives...

For these individuals and the organizations that support them, secure anonymity is critical. It may literally save lives. While the undesired effects of Tor must be recognized, the complexities and varied situations should make us suspicious of sweeping imperatives. Policies should be crafted to specific contexts.

Their recommendations to security agencies all focus on monitoring and reconnaissance, reflecting the fact that the signature problem of the Dark Web at this stage is still just finding out what’s going on (rather than who is doing it.)

Their recommendations, with my interpretation of what they meant, are listed below:

• Mapping the hidden services directory

  Like the regular Web of old there isn’t a great way to find out what’s on the Dark Web, although that is slowly changing.

  Finding out what’s actually out there is naturally a first priority and that means collating the information contained in the distributed hash tables that act as domain databases on networks like Tor. The paper suggests doing this by becoming part of the network: “it is possible to deploy nodes in the DHT to monitor requests coming from a given domain.”

• Customer data monitoring

  This is about security agencies monitoring customer data from the regular web to spot ‘non-standard domains’. The paper states that “This can be done without intruding on the user’s privacy as only the destinations of the web requests need to be monitored and not who is connecting to them.”

  The source for this data is not discussed but if you’re reading this you probably don’t need telling that your every day web activities are open to a great deal of logging and storage (legitimate and otherwise). The FBI, for example, has had specialist tools for doing this going back to Carnivore in 1997 and before.

• Social site monitoring

  Regular websites like Pastebin are used to exchange information about new Dark Web sites and monitoring them could be a way to discover what’s new in the darkness.

  The paper doesn’t mention Facebook in this context but elsewhere it quotes Robert Hannigan, director of GCHQ, who said that tech giants like Facebook, Twitter and WhatsApp have become the “command-and-control networks of choice for terrorist and criminals.” I can’t imagine they’d be left out of any monitoring.

• Hidden service monitoring

  Because of the volatile nature of some hidden services, the paper’s authors regard it as “essential to get a snapshot of every new site as soon as it is spotted” simply because it might not be around for very long or may disappear and reappear under a new name.

• Semantic analysis

  Once a hidden service has been discovered it needs to be understood and properly categorised.

  Since the number of sites is likely to be enormous, the discovery, analysis and categorisation has to be automatic. To be properly understood, sites will need to be analysed in terms of the concepts and relationships they represent and not simply the words they contain.

• Marketplace profiling

  Profiling transactions made on the Dark Web will allow agencies to build up a picture of what goods are exchanged and by which sellers and buyers.

  You might not be able to tell who a buyer is but you may be able to understand what activities a particular anonymous buyer or site has been involved in.

The report does not mention Memex, the Deep Web search tools created by DARPA that were revealed this week. Memex is mapping sex trafficking sites on the Deep Web with some success and it is perhaps a practical example of how mapping and semantic analysis might be done.

The paper provides an interesting insight into how the content of the Dark Web, not just the technology, makes life difficult for law enforcement:

...crawling the clear internet is usually an operation involving the retrieval of resources related to a site, this is not recommended in the Dark Web. There is the possibility of automatically downloading content such as child pornography, the simple possession of which is considered illegal in most countries.

The report is measured in its description of the Dark Web – to my eyes it does not seek to demonise, only describe the criminality that is the concern of the report.

The recommendations are logical but anyone who has read about the spying methods revealed by Edward Snowden will read them coloured by that knowledge.

In particular, it’s difficult to read the suggestions about customer data and social site monitoring without thinking of the NSA and its secret dragnet surveillance.

That thought is chilling – methods matter.

The paper exposes our most modern dilemma, writ large again; I want personal freedom, I want encryption, I don’t want dragnet surveillance but I don’t want safe havens for paedophiles or sex traffickers either and I don’t want to have to make an absolute trade between any of those things.

The Dark Web isn’t intrinsically bad, nor criminal, but it’s folly to deny that alongside its ability to protect privacy and save lives, it is a magnet for extreme criminality.

As police forces get better at prosecuting cross-border crime on the regular web, it’s likely to become an ever more popular place for criminals to do business too.

Some people will react to that challenge by hiding their head in the sand, insisting everything is fine and claiming that people like me are just demonising the Dark Web.

History will brush them aside.

Any notion that the Dark Web will be allowed to flourish as a vast, lawless space without a serious challenge is fantasy.

The rest of us have to grapple with reality – what balance of surveillance and darkness are we prepared to tolerate on the web?