We just received a warning from Twitter admitting that the company had made a serious security blunder: it had been storing unencrypted copies of passwords.
You read that correctly: plaintext passwords, saved to disk.
What an announcement to have to make on World Password Day!
Any regular reader of Naked Security will know that plaintext passwords are an enormous no-no.
It’s OK to have passwords in memory temporarily when verifying them at login time, but that’s all you should ever do with raw passwords.
You shouldn’t write passwords to a temporary file that you intend to delete later – your program could crash before it cleans up properly, or the disk where you wrote the data could be unmounted before you’ve finished.
You shouldn’t even keep passwords in virtual memory that the operating system might page out into the system swapfile, lest the passwords get flipped out to disk when the system is heavily loaded.
On Windows, you can use the VirtualLock() system function to keep memory blocks “locked into” physical RAM, thus preventing them from getting paged out to the swapfile.
You really, certainly, absolutely shouldn’t let unencrypted passwords get saved anywhere that’s supposed to be permanent…
…and that most definitely means taking great care that you DON’T WRITE PASSWORDS INTO LOGFILES BY MISTAKE.
Unfortunately, that’s what Twitter realised it’s been doing, thus its mea culpa warning today.
The good news is that the password databases actively used by Twitter to patrol logins were implemented securely.
Twitter uses bcrypt, an algorithm that performs what’s known as salt-hash-stretch to turn passwords into cryptographic checksums that you can later use to verify that a password held temporarily in memory was supplied correctly.
That’s because a password hashing procedure as bcrypt lets you work forwards from a supplied password to match it against the stored password hash, but not to work backwards from the hash to recover the original password.
Even if crooks hack in and steal the password file, they can’t just read the passwords out of the purloined database.
Instead, they have to try every possible password (or try a “dictionary” of likely passwords) against each user’s hash and see which passwords they were lucky enough to guess correctly.
The bad news here, of course, is that the safety provided by using bcrypt for password verification was undone by writing plaintext passwords to the system logs.
If the crooks went for the logs instead of the password database, they could extract the logged passwords directly, with no cracking or “dictionary attacks” required.
What to do?
Twitter claims that it has now “fixed the bug” and that its investigation “shows no indication of breach or misuse by anyone”.
Twitter therefore suggests merely that you “consider changing your password”.
We’ll go one step further and urge you to change your password – after all, Twitter isn’t saying how long it’s been logging passwords, or how many it collected by mistake along the way, so there’s no way to judge how far and wide any saved passwords might have been replicated by now.
We also suggest that you start using Twitter’s two-factor authentication system, also known as login verification, if you aren’t already.
This means you need to supply a single-use code that’s sent to, or calculated by, your mobile phone when you login – your password isn’t enough on its own.
Should you stop using Twitter altogether on account of this carelessness?
That’s up to you, of course, but we suggest that changing your password promptly is enough.
Curtis K
I have 2fa enabled :)
mike@gmail.com
That’s great and all, but if your 1st factor is compromised, no you don’t have two-factor, you are back to one.
Kasun
Things are coming out before 25th May. Expect more.
Bryan
Don’t forget to bring a towel.
Steve
The link to the Microsoft page on the VirtualLock function is incorrect; it goes to the page for VirtualProtectFromApp. The VirtualLock page is available via that page by clicking a link on the left side of the page for VirtualLock, for anyone who visits the site before NS corrects the link.
Paul Ducklin
Don’t know how that happened – coulda sworn I tested that link…
…anyway, thanks for the comment. I’ve fixed it now.
john
is there any ways that someone else can get those passwords though???
i yes than i think its not safe to use Twitter.
Paul Ducklin
“Never say never.” The only way to be 100% sure that someone else doesn’t stumble across your stored passwords by mistake…is not to store them in the first place.
However, in this case, Twitter seems confident that the logs weren’t unlawfully accessed – and if you change your password now then the old password becomes useless anyway, whether it was stolen or not.
security
There are different requirements for secure password storage for devices (as PIN), desktop applications (for example password managers) and web applications. We’re talking about webapps here, so I’m not quite sure why VirtualLock is mentioned. If the attacker escalates his rights to be high enough to read swap files/direct access memory – then webapp owner got much bigger problem.
Moreover, please also keep in mind that proper memory management is in most cases servers’ implementation responsibility. It’s server which processes sent requests (including the requests which contain login credentials). Remember old good Heartbleed? :)
Speaking of Windows (in the context of password storage in desktop apps) – DPAPI is worth to take a look.
Paul Ducklin
Why *wouldn’t* you prevent passwords from being swapped to disk unnecessarily, given that it’s easy to do? If an attacker does get in, and does get admin access (or if an existing sysadmin goes rogue), why make it easier than ever for the attacker to get a list of recently-used passwords by simply grepping the swapfile?
By your argument, you wouldn’t bother with forward secrecy for HTTP connections over TLS, because “if the attacker gets the private key you’re in plenty of trouble anyway”.
security
I’ve never said I wouldn’t. What I’m trying to say, is that the article describes problem in web application. Then let’s stick to recommendation on webapp level (as: store passwords in encrypted form, make sure passwords won’t leak, etc.).
I’m a fan of hardening and increasing security on every possible area (hardening your webapp, server, OS) but only with full awareness of what you are really doing.
Trying to create “secure” memory management process from the webapp perspective (and all my comments are being written from that perspective, ‘cos this article is about web application) – is just bad advice.
It is a person who is responsible for choosing the infrastructure (the place where webapp will be hosted) should take into consideration stuff like: does chosen OS offers swap partition encryption, does webserver securely manages memory, etc.
Do you know any web-programming language which allows memory management? VirtualLock() is not in .NET API (both CLR and JVM are memory-managed runtimes). The reasons are simple:
1. Higher level of abstraction – the less you have to care about, the less buggy you code will be
2. If my webapp would have privileges to access memory, then anyone who will be able to compromise me on webapp-level will be able to get that access too (principle of least privilege?)
Awareness also means, that you are able to estimate the severity of some adversary actions. I’d appreciate not putting words about TLS into my mouth, ‘cos that example is completely different.
Even if you’re in trouble, because of private key leakage – forward secrecy fulfills its task, as the attacker is still not able to decrypt old ciphertexts. In you example – if attacker already gained so high privileges that he can access swap files – then game is already over (‘cos there are plenty of other ways he can steal other user passwords).
Paul Ducklin
You’ve got a lot of advice there…
…but I still say, “Take every step you can can to prevent plaintext passwords getting written to disk.”
This thing about “oh, it’s only a web app so don’t burden me with security considerations beyond my web application world” is a cop-out.
(You make a big play about how you should store passwords encrypted and not allow them to leak. Then you say that if the passwords leak into system files, that’s OK, because the system files are protected above and beyond that.)
Nobody_Holme
What i read here was “take an end-to-end holistic view of security” (great) and then “but this advice about one step to take was bad and shouldn’t have been here because it’s not relevant” (bad).
Duck’s advice is good, as anyone who knows well enough to have a wide view will either already be doing it or can integrate the idea into the rest of their security, while people who are regrettably narrowly focused will keep their view narrow, but will at least look at making sure they prevent one source of leaks. (they’re reading NS and are either understaffed, undertrained, or underfinanced, and its an easy step, they’re going to take it).
You could have taken Duck’s point and built a useful bit of learning on it for readers, or been confrontational and started an argument to put doubt in peoples’ minds about the value of a sound security principle. Which did you do?
security
Building people security awareness requires consistency and context. So when the article mentions all good steps which should be taken into consideration (e.g. part about bcrypt is written in very clear and educating way) and then out of nowhere some advice about VirtualLock() appears – then it creates confusion. Please try to read this advice from your reader’s perspective, who’s trying to learn how to write secure web applications. As you recommended – they hash their passwords, make sure that those password aren’t leaked in logs and then they stuck at VirtualLock() advice. They won’t find VirtualLock() in their web programming language, so they will read how it works and will try to find some similar solution for their platform. We will end up in situations where web developer (based on your advice) will try to directly interact with memory, instead of just leave it to the webserver. That can bring more damage than advantages.
So just please, for the sake of your readers, please just add to that VirtualLock advice a sentence which sounds like: “Do you know that password leakage is a serious problem not only in web applications? Passwords in virtual memory that the operating system might page out into the system swapfile…”
And then everyone will be happy :)
Paul Ducklin
You’re the one who is injecting complexity by worrying about how complex it might be to call VirtualLock() from your Java app, or from C#, or whatever. Perhaps the real problem here is that people who develop web applications often think like that, and assume that because they’re a web application, security is someone else’s problem. “I can’t manage memory securely in my super-high-level web framework, so it is unreasonable to expect me to care”, sort of thing. But if you know how the lower level details work, you can ask the question of the vendor that made your web coding framework: “What did you guys do about left-over data?”
Can’t you just take that sidebar as exactly what it is… a small sidebar that gives an optional hint about one way that you can control the swappability of memory on Windows if you wish, with a link that gives some hints on where to look next?
Once you know about the low-level functionality of VirtualLock(), it’s really not hard to fan out from there with a few clicks to a lot of other MSDN advice about the security and safety of data in memory, and why you should care.
security
Actually, it’s you who’s making stuff more complex, by breaking important security role: principle of least privilege.
Talking about memory management in web application context is directly suggesting that you should break that rule (and give the web application access to taking care of stuff which should be taken care of on lower lever mechanisms – and what’s important – separately from the web app).
If adding one more sentence to your article – which will get rid of all of the confusion is too much for you – then sorry, I don’t like to waste more time on trying to convince you that VirtualLock advice leads to confusion. Basically it’s not your opinion vs mine. It’s your opinion vs principle of least privilege rule. The problem in this discussion is that you’re blindly trying to defend your text – even I said I agreed with it. Taking care of swap files, memory – all of that is important. But saying about that in web app context creates confusion. I never said you should not take care of memory management. I said, that talking about it in web app context will confuse your readers who will take your advice for granted and might try to implement it on web app level (which will make more damage then good).
As the author you should be more open to readers advice – especially those one who points out that some parts of your text are unclear and can be improved. I even told you how this text can be improved – just adding one line that VirtualLock/memory management should be taken care in the context beyond the web app. Cheers!
Paul Ducklin
As for “avoiding leakage to disk” from, say, C#, what about the SecureString class? To quote MSDN: “The value of a SecureString object is pinned in memory, may use a protection mechanism, such as encryption, provided by the underlying operating system, can be modified until your application marks it as read-only, and can be deleted from computer memory either by your application calling the Dispose method or by the .NET Framework garbage collector.” It’s not perfect, but it at least shows willing.