It’s not often that a single software bug can bring an entire industry to a virtual standstill, but it happened this week – and experts finally found an unlikely culprit.
The problem began on Monday 22 September when reports emerged of a problem with Macs running Avid software.
Avid is an editing suite that production companies use to put movies and TV programs together. A few days ago, movie editors started reporting that Mac Pros running Avid software were crashing. If users tried to restart their machines, they wouldn’t reboot.
Here’s one tweet from Shane Ross, staff editor at Prometheus Entertainment, as the situation broke:
Can anyone confirm issues with Macs running Avid (mainly reports of Mac Pro Trash Cans) shutting down, but not coming back up? This is being widely reported as a problem as of this afternoon (9-23-2019) all over Los Angeles.
— SHANE ROSS aka Mr. Snippy-snip! 🎞⚔🎞 (@comebackshane) September 24, 2019
And here’s Michael Kamens, assistant editor on Modern Family, complaining of the same thing:
Happened to our whole team at modern family. Our vendor is moviola. Its all systems there and other vendors. Its an OS issue. Key is if you see “var” folder in the mac HD icon. Lots of misinformation but hopefullh more info to come over the next day
— Michael Kamens (@TheMBrand) September 24, 2019
Imagine how you’d be feeling if you were working on something with a deadline of hours, like a news segment.
Props to Avid, which was all over this problem from the beginning, dropping everything to work out the issue, in a perfect example of how to handle a technical issue properly. The company even put up a video:
What was going on? Was it a virus? Was it another crazy attack from hackers upset about a movie that they didn’t like?
Yesterday, we found out. Google did it.
The problem wasn’t with Avid, or with macOS, but with the Chrome browser. Google’s latest Chrome update had borked the system with a bug.
When Mac users install Chrome, they’re not just getting the browser. Google also installs another module under the hood called Keystone. It’s an update manager that regularly checks to see if there are new versions of Google programs and updates them behind the scenes. Doesn’t that make you feel safe? Well, it does, until it goes wrong. The latest version of Keystone was broken.
According to a Google post explaining the incident, Chrome damaged the file system on macOS machines. It said little more than that, other than providing some command line code to fix the issue. A Chrome bug report shed more light on the matter, though.
Chrome removed a symbolic link (symlink), which is a shortcut to a linked object. The system treats the symlink as the linked object. Keystone removed the /var symlink, which threw the affected Macs into disarray. Several online commentators have already labelled the bug a ‘varsectomy’. Geddit?
Macs are supposed to prevent programs from tinkering with the system by default, using a projective measure called System Integrity Protection (SIP). Also known as rootless, it’s a feature introduced in the El Capitan version of macOS that protects system-owned files from alteration. It even protects them from sudo, which is the Linux command that people use when they’re doing dangerous stuff on the system and need to escalate their privileges.
SIP is switched on by default, but programs wanting deep access to graphics cards, like, say, a movie editing program, often need it turned off. That’s why Avid users were so vulnerable to the issue, but it also affects pre-El Capitan versions of macOS that didn’t have SIP installed.
This just goes to show how much trouble a simple software bug can cause. The problem wasn’t with Avid or macOS at all, but with a completely different third-party app.
Google was still working on a patch at the time of writing. In the meantime, follow the instructions to fix the problem.
This is one situation where the classic question “have you tried turning it off and on again” most definitely would not have been advisable.
Paul M
It’s “wringing their hands” not “ringing their hair”.
Paul Ducklin
I couldn’t figure out if that was an accidental typo or a deliberate play on words…
…so I just removed the sentence instead :-)
Bryan
My guess was an autocorrect faux pas.
Bryan
The ‘varsectomy’ thing made me laugh out loud.
We dodged this in our small production shop; only one Avid machine (on Windows). Three Macs and one Winbox run Adobe Premier instead.
OSX/MacOS’s root home directory is under /var/; it’s the only OS I know of to do so. Although I expect lacking /var/lib and /var/spool (even /var/log) is a bigger hindrance to booting properly.
Deviation from main topic of discourse:
Does anyone have a link to a resource explaining why Apple decided to change things like that? They deprecated /etc/fstab. I’ve never coded an OS, but it seems fstab is a solid paradigm–what would drive a deviation from that?
final bonus, Pareidolia in title photo, lower right:
Was this image chosen for depicting a frustrated editor awaiting system reboot?
:,)
Anonymous
The cynic (and long term Pro Tools abuser) in me asks if “deep access to graphics cards” requires turning SIP off, why did not this bug turn up in every video editing and video game software? :-\