Datadog bitten by data breach, kills all passwords

Hackers took a bite out of Software-as-a-Service (SaaS) platform Datadog, breaching multiple servers on Friday.

Datadog is one big dog: its customer list includes the likes of Spotify, PBS, Slashdot, Samsung, Imgur, Coursera, The New York Times, and Ziff Davis.

Fortunately, it’s buried those tasty password bones pretty deep: it uses a unique salt with bcrypt, a strong cryptographic algorithm used for storing passwords that takes a long time for thieves to chew through, even if a crook drags a database away to attack offline.

Still, Datadog’s playing it safe, telling customers to change all login credentials immediately.

Chief Security Officer Andrew Becherer said in a security notice that Datadog’s invalidated all stored passwords.

To help customers tell if the emails they’ve received about the breach are legitimate, he listed the emails sent out: one’s a password reset notice sent to all users with a stored password (though Google Auth and SAML users weren’t affected).

The other email was a security notice sent to all admin users, instructing them to rotate/revoke credentials stored in Datadog.

If any users suspect they’re being phished, they can directly reset new passwords at Datadog’s site: https://app.datadoghq.com.

The breach hit “a handful” of production servers, Becherer said, including a database that stores user credentials.

In addition, one user reported unsuccessful attempts to use the Amazon Web Services (AWS) he or she shares with Datadog. So again, “to err on the side of caution,” Datadog’s recommending the immediate revocation of any and all credentials shared with it.

Datadog’s strongly encouraging AWS users to employ Identity and Access Management Role Delegation, which prevents the sharing of security credentials between accounts.

If any organizations have Datadog agents running on their servers, there’s no need to worry, the company says: those agents weren’t affected by this breach.

From the security notice:

[The agents] were designed to never receive any data or code from our servers. They are also isolated from our own infrastructure, only ever communicating outbound from your instances to us via HTTPS. Our agents do not send local credentials to Datadog servers for storage.

Datadog’s currently up and running. It rebuilt all the systems it identified as having been compromised, along with additional, unspecified infrastructure, and has mitigated known vulnerabilities.

The company’s brought in incident response and forensics experts and is promising a post-mortem after it pieces together the attack. Expect the forensics to continue “well into next week,” it says.

What to do?

• Tried and tested password storage systems such as bcrypt, scrypt and PBKDF2 reduce the rate at which crooks can crack stolen passwords by a factor you can choose, and can scale up every year as computers get faster. If you’re storing passwords, do it this way, and don’t try to “knit your own crypto.”
• Storing passwords securely isn’t a replacement for protecting your authentication database against hackers. It’s a second layer of defense in depth. Your best outcome is not to get breached in the first place.
• Even if the hashes takes 100,000 times longer to crack with bcrypt, poorly chosen passwords will still be cracked first. Don’t be the low-hanging fruit: learn How to Pick a Proper Password.

(No video? Watch on YouTube. No audio? Click on the [CC] icon for subtitles.)