Though the European Union’s General Data Protection Regulation (GDPR) is designed to push companies toward better information security, it comes with potentially unintended consequences.
As has happened with other security regulations, the language of the GDPR leaves room to disrupt how companies do business.
Right to explanation
One example is GDPR’s Right to Explanation, which will affect algorithms that make decisions based on user behaviors. Under the provision, a user can ask for an explanation about an algorithmic decision made about them.
On the surface, that sounds like a terrific idea. Who doesn’t want to know what companies are collecting about them and why? Who hasn’t been annoyed at some point about ads that cater to their interests to the point of creepiness?
The problem, some believe, is that it’s impractical – impossible, even – to explain every decision made by algorithms, including those at the heart of many security programs. In other words, it opens up a can of non-ethical issues and can cause as much trouble for security as it’s meant to solve.
Impractical and impossible
Nick Wallace, a Brussels-based senior policy analyst with data policy think tank Center for Data Innovation, explained the problem as he sees it in an article he wrote in January for TechZone360. Among other things, he wrote, the right of explanation will do little to help consumers and will hobble the development and use of machine learning/artificial intelligence tools by holding developers to a ridiculously disruptive standard:
Those who drafted the GDPR do not seem to understand that it is often not practical or even possible, to explain all decisions made by algorithms. For example, autonomous vehicles are controlled by a multitude of algorithms that make many kinds of decisions. It is possible to log these actions, but it would take hours of work by highly paid data scientists to render them intelligible. Often, the challenge of explaining an algorithmic decision comes not from the complexity of the algorithm, but the difficulty of giving meaning to the data it draws on.
He wrote that the regulation should be amended before the right to explanation piece stifles artificial intelligence programs and their myriad social and economic benefits.
The case of PredPol
Sophos CTO Joe Levy said he’s been thinking hard about the topic ever since reading Evgeny Morozov’s To Save Everything, Click Here a few years ago.
In that book, Morozov describes the crime prognostication system known as PredPol and the problems some see in how it calculates where and when crime will occur using algorithm software similar to what Facebook and Amazon use to fashion ads to individuals.
Privacy advocates have raised concern that the information used to build the algorithms is biased against minorities.
A Washington Post article last year painted a picture of those privacy concerns:
Some police departments have hailed PredPol and other systems as instrumental in reducing crime, focusing scarce resources on trouble spots and individuals and replacing officers’ hunches and potential biases with hard data. But privacy and racial justice groups say there is little evidence the technologies work and note the formulas powering the systems are largely a secret. They are concerned the practice could unfairly concentrate enforcement in communities of color by relying on racially skewed policing data. And they worry that officers who expect a theft or burglary is about to happen may be more likely to treat the people they encounter as potential criminals.
In that excerpt, we see both sides of the argument around GDPR’s right of explanation provision.
On one side, the right of explanation is about protecting privacy. On the other side, it’s about a regulation messing with machine learning systems that are proving effective in building better security and crime-prevention programs.
Rewrite GDPR provision or play a longer game?
It’s reasonable to argue that some algorithms collect data that can violate a person’s privacy while having no impact on the problems they were originally set up to solve.
It may even be reasonable to suggest the GDPR right of explanation provision be rewritten. One area where it could offer greater consideration is in offering better intellectual property protections to the inventors of the algorithms, Levy noted.
But it’s also fair for someone to say GDPR isn’t the problem, or even the algorithms for that matter. Rather, the problem is in the humans who set the algorithms. Shouldn’t the fix be happening there, during the development process?
Cathy O’Neil, a former Wall Street worker who quit after the last economic crash, joined the Occupy Wall Street movement and now publishes the mathbabe blog, suggested the answer is yes in her book Weapons of Math Destruction. A Wall Street Journal article about the book sums up the essence with this headline: Algorithms Aren’t Biased, But the People Who Write Them May Be.
In our own image
Levy believes we’re on path to a future where asking an algorithm why it reached a certain conclusion will be much like asking people about their judgments or tastes. The answer might resemble “just because” as algorithms become more like the minds that create them. He said:
As we entrust algorithms to produce more complex classifications and predictions, we’ll be asking them to venture out of the light of objective truths into the shadows of subjectivity. Questions are bound to arise about the ‘fairness’ of the training sets, about the social and political leanings of the company that produced the algorithms, and whether any of the values of individual programmers had any influence.
As we allow them to learn in the wild (rather than in controlled development environments), we’ll need to worry about them being corrupted by mischievous or malicious actors. Levy said:
It begins to sound a lot like raising kids.
O’Neil says one long-term fix is to create more transparency. For example, she wrote, consumers should get an alert whenever their information is used on mathematical profiles. Consumers deserve to understand how scores like those used by car insurers are calculated. If the information used to create such scores is wrong, she wrote, people deserve an opportunity to flag and fix it. Of course, this would create new costs to businesses using and creating algorithms as they will have to separate real disputes from fraudulent disputes. But, Levy noted with a smile, “There are sure to be algorithms that could help with that.”
Analysts Bryce Goodman and Seth Flaxman wrote in their paper, European Union regulations on algorithmic decision-making and a right to explanation, that despite the problems right of explanation presents, there are also opportunities to make better algorithms. They optimistically wrote:
While this law will pose large challenges for industry, it highlights opportunities for computer scientists to take the lead in designing algorithms and evaluation frameworks which avoid discrimination and enable explanation.
Since we’re still a year away from GDPR taking effect, it’ll be some time before people on either side of the data-collecting argument get to see if they are right or wrong in their current concerns.
Meantime, we should all continue to keep working to ensure they collect information in the fairest, most accurate way possible.
