What’s worse than a world-readable copy of your customers’ data that any hackers with time on their hands (or even just a bit lof luck) might stumble across?
How about a world-readable set of customer data that’s also world writable?
That way, once crooks have downloaded the entire trove, they can snoop through the stolen data, make a bunch of subtle (or even not-so-subtle) changes…
…and upload the alterations back to your server.
That way, they get a chance to make history, not merely to snoop on it.
According to a recent report from cybersecurity company UpGuard, that’s just what its researchers found earlier this month at a Canadian robotics company.
The robotics company’s ill-secured server not only exposed 150GB of private data from automotive companies including Ford, GM, Tesla and Toyota, but also leaked personal information about its own staff, including scans of passports and driving licences.
The unsecured data even included a non-disclosure agreement (NDA) form from trendy US automotive firm Tesla.
There’s no suggestion that the NDA had been signed – and many NDAs are, in any case, a matter of public record, because it’s not the existence of the NDA itself that’s secret – but it’s a wry irony in any case.
How were the backups organised?
Apparently, the afflicted company, Level One Robotics, is a user of the astonishingly useful open source software rsync.
Simply put, rsync is a remote file and directory copying tool that makes it both easy and efficient to keep two copies of your data synchronised, even if they’re on opposite sides of the world.
Traditional copying tools typically duplicate files blindly from A to B, even if B already has the needed files – this can waste enormous amounts of time and network bandwidth.
Traditional backup tools add a bit more intelligence by only copying files that have changed since last time, but usually copy an entire file even if only a few bytes in it have changed.
But rsync goes much further than that, and automatically.
When synchronising, client A and server B work in tandem through the files at each end of the link and efficiently:
- Agree which new files need to be copied from A to B.
- Agree which files on B are no longer needed and should be deleted.
- Figure out how to update files on B to match the files on A by modifying only the parts of the files that have changed.
For example, in a text file that has had extra paragraphs added at the end, rsync will copy across only the new lines, thus avoiding copying the bulk of the file that already exists at the other side of the link.
Likewise, if the middle few slides in a presentation have been deleted but the rest of the document is the same, rsync will figure that out and transmit the instruction “delete the middle 20% of the target file” instead of re-transmitting the 80% of the file that hasn’t changed.
The rsync tool is part of the Ph.D. dissertation that made Australian computer scientist Andrew Tridgell, better known as “tridge”, into Dr. Andrew Tridgell. Tridge is held in iconic regard in the open source community for creating Samba, a free reimplementation of Microsoft’s SMB networking protocols (now standardised as CIFS). Samba made it easy to introduce Linux servers into networks that were dominated by Microsoft products. You should read tridge’s Ph.D. – it’s not oppressively long, unlike some doctoral papers out there, and Andrew writes really well.
What went wrong?
The problem, if that is the right word, with rsync is that it’s enormously powerful – and that makes it prone to disastrous misconfiguration, by accident or design.
You need to be really careful about who’s allowed to connect to your rsync servers, and what sort of commands they’re allowed to send.
For example, to leech your entire backup, all I need to do is start with an empty directory and tell your server, “Please get me synchronised” – rsync will quickly figure out that I need a copy of every file, and that’s what I’ll get, without needing to know anything about your directory structure to start with.
Of course, if you run the same command in the other direction, you’re telling the server to “sync” itself with an empty directory, so it will immediately delete everything at the other end in order to make the source and target consist of an identical collection of zero files.
Alternatively, to make a small and subtle series of unauthorised hacks to your files, I can rsync your entire data set to my computer, find and edit just the files I want to mess with, and then run rsync once again to push back my alterations – automatically and super-efficiently.
Unfortunately, Level One Robotics hadn’t been cybervigilant enough at controlling who could get at its data.
Its rsync server, with access to more than 150GB of private and personal data, ought to have been accessible only to carefully selected servers on the company’s own network.
But the rsync server was accessible directly from the internet, findable by anyone prepared to set out and look.
The good news here is that UpGuard reported this potentially disastrous misconfiguration to Level One, and Level One fixed the problem pretty quickly – before the story was made public.
What to do?
If you’ve got servers that are only supposed to be accessible to selected people from selected computers on selected networks…
…then for goodness’ sake check to make sure that your servers don’t show up where they aren’t supposed to.
If you’re new to network scanning, which is where you deliberately go looking – with permission, of course – for what’s visible where on your own network, we recommend starting off with Nmap.
Nmap is free, powerful, has 20 years of history behind it, and you might as well run it against your own network to see what’s what – you might not be scanning your servers, but you can be sure the crooks are!