NPM update changes critical Linux filesystem permissions, breaks everything

You’ve probably heard of JavaScript.
It was invented as a programming language, for use in web pages, that would run not on your web server but inside every web visitor’s browser.
Unlike traditional program downloads, with a pop-up dialog and an “Are you sure” button, JavaScript is shovelled straight into your browser and, by default, runs automatically without asking.
Website features we take for granted these days – pull-down menus, pop-up forms, animated image transitions, clickable icons, and much more – are all achieved thanks to browser-side JavaScript.
Of course, this means that even though JavaScript is a fully-featured, general-purpose language, it can’t be used for just any old thing while you are browsing.
The power of JavaScript is carefully limited by the browser, to reduce the rather obvious risks of running program code that arrived in a web page from “out there somewhere”.
For example, JavaScript programs in your browser generally can’t reach out into other tabs, can’t start or stop other programs, can’t access files on your hard disk, can’t read the registry, can’t scan the network, can’t sniff around in memory.

But JavaScript became so well-known and widely-loved – as coding languages go, it is clean, expressive and powerful – that it has turned into a server-side programming language, too.
Thanks to a project called Node.js, JavaScript has now joined popular server-side languages such as PHP, Perl, Python, Ruby and Java (which is unrelated to JavaScript, by the way) as a coding platform for building complex systems.
Unlike browser JavaScript, Node.js is augmented by add-on toolkits to do just about anything you can think of: manage processes, run servers, read local files and databases, control the network, perform cryptographic calculations, transcode images and videos, recognise faces, you name it.
The careful restrictions imposed on JavaScript inside the browser are unnecessary – indeed, are a hindrance – when you’re writing a full-blown application using Node.js.

More power means more complexity

All these add-on capabilities come with a price: complexity.
For example, let’s say you want to use Node.js to program face recognition into your website’s login system, and you decide to use the ready-made library called facenet, a “deep convolutional network designed by Google, trained to solve face verification, recognition and clustering problem with efficiently at scale.” [Sic. Let’s hope the code has fewer errors per line than that short quote.]
Well, facenet itself needs a bunch of additional add-ons, namely: @types/ndarray, argparse, blessed, blessed-contrib, brolog, canvas, chinese-whispers, glob, mkdirp, printf, python-bridge, rimraf, tar, and update-notifier.
So far, so good, but these dependencies have their own needs: chinese-whispers, for example, needs jsnetworkx, knuth-shuffle and numjs.
And so it goes: jsnetworkx needs babel-runtime, lodash, through and tiny-sprintf; and babel-runtime in turn needs regenerator-runtime.
The good news is that when you adopt Node.js for programming, you also get NPM, the Node Package Manager, which sorts out all these dependencies for you.
Automatically. From all over the internet.
The bad news, of course, is that when you adopt Node.js for programming, you also get NPM, the Node Package Manager, which sorts out all these dependencies for you.
Automatically. From all over the internet.
You may find you can write a five-line JavaScript program that is elegantly simple, but only if your Node Package Manager drags in tens or even hundreds of thousands of lines of other people’s software.
Automatically. From all over the internet.
And keeps it updated, automatically, from all over the internet.

Awash in other people’s code

Simply put, your graceful five-line JavaScript program, behind the scenes, is a hodge-podge of directories and files awash with other people’s code that you couldn’t easily keep track of yourself even if you wanted to, all of it kept afloat automatically by a package management toolkit that is about as complex as the auto-updating system built into the operating system itself, yet not integrated with it or even built in collaboration with it.
(If the previous paragraph seems rather long and breathless, please treat that as a metaphor.)
Perhaps unsurprisingly, then, a recent update to the Node Package Manager introduced a bug that caused it to interfere with the operating system, by incorrectly changing the file permissions of a raft of important system directories that should have been left well alone.
In other words, you had to give NPM operating system superpowers to keep your Node.js world in order; while doing so, NPM mis-used these superpowers to throw your operating system into disarray by locking the system itself out of numerous mission-critical files:

I found that a selection of directories in / were owned by a non-root user after running sudo npm and many binaries in /usr/bin stopped working as their permissions were changed. People experiencing this bug will likely have to fully reinstall their system due to this update.

What to do?

1. Keep backups that make a meaningful rollback easy. NPM has caused reliability disasters before, and given its vaguely anarchical nature, will cause them again.
2. Skip the 5.7.0 version of NPM. NPM version 5.7.1 is supposed to fix this bug.
3. Don’t autoupdate production or development servers. Prove the latest update in testing first.
4. Remember that simple software can be immensely complex. Keep that in mind when making time for testing (see 3).

There’s an adage that’s been applied to many “breakthroughs” in software engineering in the past few decades:

I’ve got a programming problem! I know, I’ll solve it using X. Oh, dear… now I’ve got two problems.

Technologies like Node.js and NPM are both a blessing (because they let you do complex things quickly), and a curse (because they solve the problem of “quickly”, not the problem of “complex”).
Plan accordingly, because those who cannot remember the past are condemned to repeat it.