Remember the Gentoo data breach story last week?
Someone broke into the Linux distro’s GitHub repository, took it over completely by kicking out all the Gentoo developers, infected the source code by implanting malicious commands (rm ‑rf) all over the place, added a racist slur, and generally brought a week of woe to the world of Gentoo.
In case you’re wondering, rm ‑rf is Unix/Linux system command language for remove files (rm) recursively (‑r), which means “including any subdirectories”, and forcibly (‑f), which means that the user won’t see any warnings or prompts. The Windows equivalent is DEL /S /F /Q, a command you often regret almost immediately after you hit [Enter].
Fortunately, Gentoo’s GitHub repository wasn’t the primary source for Gentoo code, and few, if any, Gentoo users were relying on it for software updates.
Phew.
Other good news is that the stolen GitHub account is back under Gentoo’s control now; the hacked files have all been identified and removed; and Gentoo has learned (and, at the same time, taught the rest of us) three main lessons.
Lesson 1. A prompt notification goes a long way.
At first, Gentoo knew merely that something bad had happened – it was locked out of its own GitHub account, which was a bit of a giveaway – but not how or why.
Nevertheless, the organisation didn’t beat around the bush in preparing a breach notification message, and it didn’t waste time trying to work a marketing spin into its initial report.
As a result, the issue got widespread attention and community help right away.
Many commercial organisations could learn from this – trying to disguise bad news as if it were nothing of the sort often ends up sounding confusing at best, and devious at worst.
Lesson 2. Pick a proper password.
Gentoo’s final summary of the incident says:
The attacker gained access to a password of an organization administrator. Evidence collected suggests a password scheme where disclosure on one site made it easy to guess passwords for unrelated webpages.
In other words, the user whose password was guessed had fallen into the trap of using different but nevertheless obviously related passwords on multiple sites.
It’s an easy thing to do – pick a core password (for example, pASS\/\/orD) and then use some easily-derived additional text each time you need a new password, for example like this:
pASS\/\/orD-FB pASS\/\/orD-TW pASS\/\/orD-G+ pASS\/\/orD-Y!
Technically, this means you are complying with the rule that says, “One site, one password – never use the same password on different sites.”
But if I were to figure out, or even just to guess, that -Y! in the last password was meant to denote Yahoo!, it would be an easy jump to try suffixes like -FB, -TW and -G+ for Facebook, Twitter and Google Plus respectively.
Don’t use a core password with tweaks or suffixes for each site – the crooks will figure out your pattern sooner or later.
Use a password manager and let it choose a totally different password for each site.
3. 2FA is your friend.
Apparently, Gentoo didn’t insist on two-factor authentication (2FA) before the breach.
It does now!
2FA, also known as two-step verification, usually means you have to put in your regular username and password and then follow it up by typing in a one-time code that works only for the session you are trying to set up.
Those one-time codes generally come either from an app on your phone, or via an SMS or other text message sent by the service provider.
2FA isn’t perfect, but it does make things harder for the crooks, because they can’t just steal or guess your password – they typically need your phone (and its unlock code), too.
What to do?
The Gentoo breach turned out to have a root cause that wasn’t about malware attacks, phishing emails, social engineering calls, exploits, zero-days, or any other technological trickery.
This story is a straightforward reminder that cybersecurity basics matter – and that making it very slightly less convenient for legitimate users to login every time makes it very much harder for crooks to login at any time.
If you’re asked to trade a tiny bit of personal convenience for a lot of extra cybersecurity for your company…
…take one for the team!
John-Paul Caldwell
One of the issues with ‘Nix users is that sudo is needed every time anything is done, having to type your password so much makes it annoying to have any complex passwords.
Over security makes things less secure because people get lazy. Changing passwords every 30 days makes people write down their passwords making things way less secure and adding text at the end of a password is AWFUL *TISK TISK writer* advise because passwords just get hashed password1, password2, password3 and substitution does not make for a more complex password, Length does.
The best advise I can give for passwords is pick three words and a number
3catsdogspig, cupsunflowermirror0 and you will have long secure ass passwords that are easy to remember.
Paul Ducklin
Your advice is well-meant, but it’s not enough – if you ask people to “pick three words” then you are inviting them, by accident or by design, to be *very non-random indeed*.
A decent dictionary, for example the New Oxford American Dictionary, contains about 350,000 words. Let’s guess that about 100,000 (105) of those are suitable as “words to use in a multiword password”, so that your password generator offers a total password space of 105×105×105×10. That comes to 1016, which is close to 253, or 53 bits’ worth of password.
But [a] you don’t know 100,000 words (no one does, not even Susie Dent) and [b] of the 10,000 or so words you do know, you are likely to choose from far fewer than that and [c] you are, in any case, more likely to choose some combinations than others – as you did when you wrote “catsdogspig” instead than “endocrineindigentwoomera”.
Anyway, different people remember things differently, so the “always pick multiple words” approach (the technique promoted by Her Majesty’s Government in the UK, for example) is, in my opinion, a flawed suggestion because it prescribes randomness through order – a contradiction in terms.
Seriously, please watch the video in the article above – it’s just a tiny bit over 2′ long but tries to touch on all of this stuff…
…and then get a password manager :-)
John C
Yes, using multiple words does narrow the population of possible combinations, if the hacker knows that is the method used, knows how many words you’re using, and tries every conceivable prefix or suffix. In other words, this method might not be the very best way to develop strong passwords, but it’s a very good one. If you are the target of a focused attack by someone who has stolen the hashed passwords, they will eventually find your password if they are willing to dedicate the resources to do so. Most attacks involve finding the low-hanging fruit of short, easily-guessed or reused passwords. In most cases, a “very good” password is more than adequate. I might choose to use a truly random password for my email account and banking accounts, but the method detailed above is fine for almost anything else, in my opinion.
Paul Ducklin
It’s almost as though you didn’t watch the video :-)
s31064
I’m glad you don’t run our security…
s31064
I’ve used a password manager for a very long time; so long that I don’t remember ever being without it (except for the days working on DEC mini-systems, of course). Since watching that video when it first appeared, I’ve stopped using my password manager to simply store my passwords, URLs and notes about a site/system, I now have it generate all my passwords as 40 character (some of the systems I use can’t handle 58, let alone 64) random passwords, like S#e5|UHUORbtcU@h8@kc*3!s’:%*Q`p+Xbf=X+$= Don’t get me wrong, this really screws up being able to login to an online account if I’m not at my own PC and don’t have my phone, but I find the peace of mind well worth it.
Epic_Null
I haden’t thought about it before, but I would agree that using sudo all the time does make using a complex password a bit unruly… perhaps it may not be a bad time to discuss extending the timeout period, and reminding people to use `sudo -k` when they finish.
Paul Ducklin
Or “sudo su -” :-)
(At least you end up at a root prompt, which kind of focuses your mind on the power you have appropriated.)
Epic_Null
A root prompt does have a lot of power, and it is a tool I do use, but it’s a tool I find is often inappropriate – usually its a minority of commands that I’m looking to run as root, while the majority I’d prefer to run as my current user. A root prompt also typically has a different environment, including a different home directory, which can cause unexpected behavior (for example, trying to copy a document to my home directory and having it land in the root’s home directory).
Jim Gersetich
I agree with your main point that the plethora of sudo commands needed pushes people into lax security.
However, I also agree with Paul that using words can cause its own issues.
What I propose instead is one or both of two tricks that should keep most pirates at bay:
One is to purposely misspell the words. You can even use a mental rule, like “all my e’s become u’s, and all my u’s become e’s”.
The other is to capitalize the words in a non-standard way, like always capitalize the last letter of each word (instead of or in addition to the first).
Also, don’t just use one digit. Use more, and for systems that will take it, also use the corresponding shifted version of that key elsewhere in the password. So, if you have a 1 in the PW, put a ! in a different part of the password. You can remember the key (1) and also where you put them (like, 1 between the first and second word and ! between the second and third).
I do agree with Paul’s statement to get a password manager, but that may not be feasible for Linux or Unix systems. So, the above can turn your passphrase into a much stronger mental password algorithm. Just don’t tell anybody your algorithm. ;)
Willem
KeepassXC will do excellent work as a Passwordmanager for MacOS, Linux or Unix. It is based on Keepass and very well maintained. So it’s possible to increase your security measures by using a passwordmanager.
Max
Still kinda hard to utilize a pw manager if you don’t have a graphical user interface on your linux machine.
Paul Ducklin
There are plenty of command-line password managers around. If you are hard enough to run Linux without X then you are either running Slackware or should be switching to OpenBSD… and the command line holds no fears for you :-)
Daniel Reinhardt
Spelling mistake:
files have all been identified and removed; and Gento has learned (and, at the same time, taught the
Paul Ducklin
Fixed, thanks.
Foo
Passwords in general are bad. If I need to use one it’s 32-64 chars using “makepasswd –chars=32”. Doesn’t choose ambiguous digits either so typing them in if necessary isn’t impossible. I do sync passwords but the syncing service for web stuff and I use requires a yubikey as a second factor wherever possible (an actual physical key, almost like a USB stick). Google supports it, so does Facebook if you’re into that kind of thing. Anything I ssh into requires ssh keys installed in addition to a password.
[URL removed]
The second factor is my yubikey. Any Linux machine can be made to require one with pam-yubico and it works even for remote logins. The how-to to set it up is not hard. I also have the “fail2ban” daemon and a few other running so failed logins will result in a ban for that machine. GitHub supports yubikey too. Kinda incomprehensible Gentoo devs weren’t using one. I wouldn’t judge all Linux users by one careless person. Linux has a ton of very good security features including mandatory access controls (selinux), most of which aren’t actually that hard to set if you’re not afraid of a little documentation.
Richard
Not that it makes it much better, but “Gentoo wasn’t using any form of two-factor authentication (2FA) before the breach” is inaccurate. 2FA on github was optional and some (many?) of us were using it. I know that I was. The owner of the compromised account sadly was not. 2FA is now mandatory. We have also had a talk with the developer in question (and another who admitted to also having weak passwords) about using randomly generated passwords and using things like lastpass.
We had much higher security requirements for our other infrastructure. SSH public keys are mandatory for access to core infrastructure and doing commits to the main tree while gitolite mandated PGP signed commits. The github organization was added because some developers and users wanted an easier way to receive/contribute changes. Unfortunately, it was not subject to the same hardening that the core infrastructure was.
Audits are being done of our security elsewhere to catch any potential issues. My own PGP keys got flagged by an audit because one, I had commit privileges with two keys (one of which was never used to commit to Gentoo) and two, I didn’t use an expiration date. I removed the non gentoo.org key from our internal infrastructure. I will set an expiration date on the Gentoo key now that the issue has been raised after I am certain that I am not pushing a public key where I messed up in some non-reversible way.
I had planned to switch to using OpenPGP smartcard(s) before the breach, but I had not done it yet. I switched to one the day after the breach. I have switched to U2F-based 2FA on my github account. Others are advocating that everyone adopt smartcards for U2F and OpenPGP. I believe that it is quite likely that the Gentoo Foundation would pay for them if developers were to request financial support in getting them.
I hope that we are being sufficiently transparent about our security practices and what we are doing to correct oversights that the github organization breach either exposed or gave us incentive to find. We definitely could have and should have done more to force proper habits, but we are now doing that.
Lastly, I am neither one of the people with the authority to force secure practices on other Gentoo developers nor am I one of the people who messed up with weak passwords and no 2FA. The idea that someone didn’t use 2FA was as much a surprise to me as it was to anyone. I am in favor of forcing as many secure practices on ourselves as we can.
Paul Ducklin
You’re right… I’ll reword my comments to make them clearer – the word “any” doesn’t really belong in there because it implies that 2FA was forbidden entirely, not that it wasn’t enforced for everyone :-)
Thanks for coming here to comment – it’s much appreciated to hear a real-life story as you have told it.
Paul Ducklin
Have a look now – a small change to my text but I think it’s clearer…
Laurence Marks
> “Don’t use a core password with tweaks or suffixes for each site – the crooks will figure out your pattern sooner or later.”
Great advice, particularly if you are a prominent persona. Not sure someone would bother trying to figure this out if you were just Joe Smith on the street. How many minutes do you think they could afford to spend on each individual guessing at patterns and trying them out? And what value could they get from access to one’s Facebook or Google Plus account anyway? The action would be at -BoA or -CMB.
Obviously one shouldn’t use the same password everywhere. It is easy to write a script that tries a single leaked password on all the major banks, etc. But the -FB, -Y! suffix defeats that.
Anonymous
Sophos needs to put this same level attention towards their own Linux-based products. Feature parity across OSes is almost non-existent.
Paul Ducklin
I am not 100% sure what you mean here – we’re writing about passwords and 2FA in the development process, not about how product managers pick which features to develop.
Your words come across as though the fact that our Windows product has more features (e.g. Intercept X, CryptoGuard, WipeGuard) implies that we have a problem with managing passwords and access control amongst our Linux coders compared to Windows…
I’ll pass on your comment… but I think you are making it in reference to the wrong issue, and in the wrong forum,
Josh
It’s not an underscore it’s a dash. rm -rf
Gerhard Lemmer
> Obviously one shouldn’t use the same password everywhere. It is easy to write a script that tries a single leaked password on all the major banks, etc. But the -FB, -Y! suffix defeats that.
The suffix **does not** “defeat that”. It merely somewhat reduces the risk. It’s easy to modify the script you mention to attempt a large variety of tweaks (including 2-letter suffixes derived from the site) w/o a large additional amount of overhead. The script could also be modified to try to automatically identify your tweak from the leaked password. Doing this may not be very cost-effective for an attacker attacking only your password if he’s already looked at your password and didn’t identify any obvious tweak, but the most likely way your password got leaked in the first place is if it got leaked together with multiple other users. Getting one more access = probably not worth the bother. Getting 10000 more accesses from a somewhat smarter script (since you’re trying 1000000 accounts and tweaking passwords is so common) = profit!