Oops, Google said on Tuesday: you know that domain administrator’s tool to reset passwords in the G Suite enterprise product? The one we implemented back in 2005, as in, 14 years ago?
We goofed, Google said. The company’s been storing copies of unhashed passwords – as in, plaintext, unencrypted passwords – all this time.
From a blog post written by Google vice president of engineering Suzanne Frey:
We made an error when implementing this functionality back in 2005: The admin console stored a copy of the unhashed password. This practice did not live up to our standards.
Only a small number of enterprise customers were affected, she said, though Google hasn’t put a number on it. People using the free, consumer version weren’t affected. Google’s notified a subset of its enterprise G Suite customers that some of their passwords were stored in plaintext in its encrypted internal systems.
Frey said that no harm came of it, as far as Google can ascertain, and it’s since been fixed:
To be clear, these passwords remained in our secure encrypted infrastructure. This issue has been fixed and we have seen no evidence of improper access to or misuse of the affected passwords.
How it’s supposed to work
The way Google typically handles passwords is by scrambling them with a hashing algorithm so humans can’t read them. It then stores hashed passwords along with their usernames. Then, both usernames and hashed passwords are encrypted before being saved to disk.
The next time a user tries to sign in, Google again scrambles their password with the same hashing algorithm. If the result matches the stored string, Google knows you must have typed the correct password, so your sign-in can proceed.
As Frey explained, the beauty of hashing is that it’s one-way: scrambling a password is easy, but it’s nearly impossible to unscramble it. So, if someone gets your scrambled password, they won’t be able to backtrack to your real password. Presuming, that is, that it’s also been salted. A salt is a random string added to a password before it’s cryptographically hashed.
The salt isn’t a secret. It’s just there to make sure that two people with the same password get different hashes. That stops hackers from using rainbow tables of pre-computed hashes to crack passwords, and from cross-checking hash frequency against password popularity. (In a database of unsalted hashes, the hash that occurs most frequently is likely to be the hashed version of the notoriously popular “123456”, for example.)
The downside of that one-way password hashing street is that you’re out of luck if you forget your password: Google can’t help you out by unscrambling your password for you. What it can do is to reset your password to a temporary password, make it valid only for one-time use, and then require you to pick a new one.
That’s the way it should work, anyway, though we’ve seen plenty of cases where forgetful users get emailed their plaintext password: an indication that their passwords are being stored, in plaintext, unsalted and unhashed.
Goodbye, handy dandy password recovery tool
To avoid storing passwords in plaintext, and to still be able to help out users who’ve forgotten their passwords, Google in 2005 introduced a tool for password setting and recovery to G Suite.
The tool, located in the admin console, enables admins to upload or manually set user passwords for their company’s users. Google’s intent behind introducing the tool was to help with onboarding of new users, such as when a new employee needs an account on the first day they start work, and also for account recovery.
That tool was apparently the component that stored plaintext passwords.
But wait, there’s more.
Google says that when it was troubleshooting its new G Suite customer sign-up flows, it discovered that starting in January 2019, it also inadvertently stored a subset of unhashed passwords inside its network.
This time, the passwords were supposedly there for at most 14 days.
That plaintext glitch has also been fixed. And like the other glitch, this second one apparently didn’t lead to anybody getting at the passwords, either.
Sorry, Google said: we’ll try to ensure this is an isolated incident. That presumably means “isolated” as in “it only happened twice.”
It’s not just Google
Unfortunately, Google isn’t the only technology company, large or small, that’s guilty of “isolated incidents” involving the storage of plaintext passwords.
Facebook, for example, admitted in April 2019 that it accidentally logged millions of Instagram passwords inside its network, in what feels like a similar sort of blunder to Google’s.
The moral of the story is that tech behemoths like Google and Facebook sometimes screw up and store passwords in plaintext, which makes it a pretty good bet that any other smaller online service that employs less slick technology and far fewer security engineers might very well slip up and do the same, be it by mistake or because they don’t know any better.
By the way, two-factor authentication (2FA) is a good way to save your bacon. Slather it on everywhere you can: 2FA, or U2F (Universal 2nd Factor) security keys, mean that a password alone isn’t enough for crooks to raid your account.
anthonymaw
Is it 14 years or 14 days ? The article cites both. perhaps a typographical error ?
Paul Ducklin
I’ve reworded that bit for clarity.
From Google’s blog post, it seems there were two incidents. In one, there was a user-facing tool dating back to 2005 that stored plaintext passwords, so in that case, the plaintext lay around for up 14 years.
In the other incident, a system tool dating to January 2019 also leaked plaintext passwords, but apparently the plaintext only lay around in that case for up to 14 days.
The fact that there were 14 units of time on each occasion is just a coincidence.
Magyver
So Lisa, no harm came of it, as far as Google can ascertain – hmmm…
I bet the 14 Eyes Countries would disagree, but then again these guys don’t leave tracks.
Thanks Lisa!
Anonymous
14 YEARS NO WONDER I DON’T HAVE AN IPHONE OR SAMSUNG
Anonymous
Glad that is fixed. Now if I can just keep my coworkers from writing down their passwords on sticky notes and leaving them stuck to their monitors.