Imagine being the employee who had to send this rather melancholic message:
First of all, we want to offer our apologies for any inconvenience.
Unfortunately, an ex administrator has deleted all customer data and wiped most servers. Because of this, we took the necessary steps to temporarily take our network offline.We have been working hard to recover the data
but this was not possible for all data that was lost.After investigation it looks like we didn’t lose any important data.
Our network and hosting services will be back this week with security updates. Current customers who are still interested in our services will receive compensation for their services. If clients have important data please contact us at support@verelox.com. We will try our best with our technical team to recover you data.
You can read the message in full, with many far more cheerful updates, here. It was posted after Verelox, a Netherlands hosting company, last week suffered what at first blush looked like a rampage from one of those ex-employees from Hell.
In this case, it was an ex-IT admin who, Verelox initially thought, had walked in and wiped out all customer data and most servers.
Fortunately, further investigation found that Verelox didn’t lose any important data. Its most recent updates said that all its systems were back up and running – including for customers in the US, France, Canada and the Netherlands – and that it’s currently working on getting up a new site and control panel.
According to posts on forums including Hacker News, FreeVPS and LowEndTalk, this all started on Thursday.
Clients were left in the dark, with no heads-up from Verelox about what was happening or when it would be fixed. Well, except an initial message that it was undergoing maintenance and would be back “soon!”
Of course, it’s pretty easy to see why customers wouldn’t have gotten a heads-up about the “maintenance.” It had lost all its customer data, after all. Contact customers to give them updates, you say? OK. Ummm …. how?
Was this really one of those rogue ex-IT employee stories? Those are the monster stories that IT execs must tell each other when they sit around the campfire. Like, the one about the ex-IT director who got sued for allegedly setting himself up with a fake email account the day before he left and then using it to hack the company for more than two years.
Then too, there was Yovan Garcia, who was fined $318,661.70 after a California court found him guilty of padding his work hours, hacking the company’s servers to steal data on customers, demolishing the servers in the process, defacing the website, ripping off the proprietary software, and setting up a rival business running on that ripped-off program.
So many IT ex-employee monster stories! But to put some perspective around Verelox’s own nightmare ex-IT admin, it helps to take a look at another recent story that brings to mind Hanlon’s razor. To wit:
Never attribute to malice that which is adequately explained by stupidity.
…or, come to think of it, never attribute to villainy what can be explained by incompetence, misunderstanding, or the fact that the employer company didn’t make backups.
This second story comes to us courtesy of Quartz, and it’s about “a young developer who made a huge, embarrassing mistake.”
As the Junior Software Developer for a day explained it on Reddit, on the first day on the job, following company documentation, they managed to accidentally destroy the company’s production database, was instantly fired, and as of nine days ago, was being threatened with legal action.
how screwed am i?
The IT world has flocked to support the young Padawan, with Redditors like this one pointing to examples of where similar catastrophes – say, this December 2012 SNAFU at Amazon – have been triggered by non-villains:
Does the employer of the young, terrified Padawan suck at life? Well, it’s hard to say, just as it’s impossible to figure out whether Verelox sucks at life. But while we can’t ascertain whether such companies (and others) that suffer catastrophic data loss at the hands of employees are themselves sucking at life, it’s fair to assume they’re sucking at the information security principle of least privilege.
From a Hacker News commenter who likened Verelox’s tale to that of another hosting site:
Good cautionary tale for segregation of credentials and proper user key management.
Who should lawyer up? The ex-employees who trigger data mudslides, on purpose or on “OOPS”!!? Whoever left the system open to the extent that the fired Junior Software Developer could actually get at, and annihilate, a live production database? Whoever wrote the manual he or she was following? Whoever failed to set up a backup plan for a live production database?
Readers: your thoughts, please!