Rift keels over after Oculus forgets to renew security certificate

Somebody screwed up at Oculus on Wednesday, when an expired security certificate caused all Rift virtual reality headsets to keel over.
It was first called out on Reddit when a user said his machine decided to update, never restarted, and gave an error message that read “Can’t reach Oculus Runtime Service.”

The problem turned out to be an expired security certificate that Oculus failed to update along with the Rift software, the company confirmed on its forum. Oculus co-founder and head of Rift Nate Mitchell also confirmed the headset issue on Twitter:

We're aware of an issue affecting Rift on PC, and we're working on resolving now. Stay tuned.

— Nate Mitchell (@natemitchell) March 7, 2018

Unfortunately, updating the certificate turned out to be a bit of a sticky wicket, according to the company, which Facebook bought for $2 billion in March 2014.

Unfortunately, pushing the [update] out to affected users has some added complexity, as the expired cert blocks our standard software update path.

The expired certificate was used by OculusAppFramework.dll: a dynamic link library (DLL) in Rift’s Runtime Services. The certificate expired on 7 March 2018.
A DLL is a library full of code and data that can be used by more than one program. Microsoft requires code libraries to be signed so that a program using a library can check it’s using the genuine one rather than a malicious interloper.
The certificates used to sign code eventually expire so they need to be replaced from time to time, although the replacement normally happens before the expiry date so nobody notices.
Reddit user TrefoilHat spelled out the difficulty of certificate management with a hypothetical scenario that readers working in software engineering might recognize:

…this is exactly the kind of problem people just assume will be figured out later. A developer or release manager generated the signature (and went through the whole validation process), maybe stuck a note in a spreadsheet/JIRA ticket/whatever, and moved on. Maybe that person is no longer at Oculus. Maybe they’re in a different role. Maybe there are super-tight controls now, but that one key slipped through the cracks (just like that neighbor’s key you vaguely remember…did you give it back, or not….hmmm…it’s not where you expected it, so maybe you did give it back?)

Code signing is a very good idea indeed but it isn’t perfect. Naked Security’s Paul Ducklin took a dive into the security certificate ecosystem recently and noted that there’s a lot that can go wrong besides “Oops, forgot to renew it”, not least:

• Crooks can steal certificates and start signing malicious code with a vendor’s official seal.
• Certificate Authorities (CA) can rogue or get sloppy, undermining the chain of trust that vouches for the vendor’s certificate.

Regardless of how it happened at Oculus, Rift’s glitch left VR developers, gamers and other users fuming. They were left in the dark for hours, they complained, and what’s up with the lack of 24/7 support?
It was all ironed out as of midnight on Thursday, Mitchell said. You can find the fix here. He apologized, thanked users for their patience, and offered Oculus store credit as recompense for the downtime:

Rift is back online as of ~12am. This was a mistake on our end, and we apologize. Folks impacted by today's downtime will be provided with an Oculus store credit. More details to follow soon. Thanks again for everyone's patience as we worked through this one.

— Nate Mitchell (@natemitchell) March 8, 2018