Updates. Shortly after publishing this article we were able to fetch Firefox 66.0.4, which claims to fix this issue by repairing a broken certificate chain. [2019-05-05T22:15Z] In the early hours of Tuesday morning, UK time, we received an update notification from Tor and were able to fetch Tor Browser 8.0.9. Don’t forget to switch your “signatures required” setting back to true (see below for how) if you turned off signature checking as a temporary workaround. [2019-05-07T00:15Z] Firefox 66.0.5 followed a few days later to tidy up the installation of the fix for some users where 66.0.4 hadn’t resolved the issue. [2019-05-09T10:50Z] Mozilla has published an explainer that sets out what went wrong, how it was fixed, and what changes are planned to improve signature verification resilience in future. [2019-05-10T12:30Z]
It’s a long weekend here in the UK, so the atmosphere is relaxed…
…except, we suspect, for any British members of the Mozilla Firefox programming squad.
Mozilla is currently stuck in the middle of a cybersecurity blunder involving digital signatures.
The bug reports we’ve seen so far don’t give much more detail than “expired intermediate certificate” problems, but the symptoms are obvious, especially for Tor users.
We didn’t get hit by this bug immediately – we were off the grid yesterday and left our computing kit at home. (Nothing Bear Gryllsy, you understand – we took ourselves off to Bristol on Brunel’s famous Great Western Railway to visit a bicycle show but left our mobile phone behind entirely by mistake.)
But today, not long after firing up the Tor Browser, which is a special version of Firefox with numerous privacy-centric settings turned on and baked into the build, we received a worrying popup warning.
According to the Tor Browser program, one of our browser add-ons could no longer be trusted and had been turned off – the alert didn’t say which one, just that some sort of cybersecurity concern had suddenly arisen.
We were online to look into a couple of untrusted sites, and we’d already started digging around when the warning popped up, which increased our sense of disquiet.
After all, we were already in the middle of various HTTP sessions; we’d been interacting with the sites we wanted to investigate; and we weren’t aware of having allowed those sites to install any new addons.
What had changed?
A trip to the special URL about:addons (Tools → Add-ons from the menu bar) and a click on the Unsupported tab revealed the following:
NoScript could not be verified for use in Tor Browser and has been disabled.
Ow! Ouch! Owowowowowowowowow!
NoScript is an important security addon that’s officially trusted by Tor, as well as being installed by millions of other regular browser users, including – to judge from comments on this site – a significant number of Naked Security readers.
Had the NoScript repository been hacked? Was a bogus NoScript addon circulating amongst the Tor community? Was there some sort of Firefox vulnerability that allowed rogue sites to sneak bogus addons into your browser without popping up any sort of “Are you sure” or “Do you want to do this” dialog?
We assumed that this was just the sort of cybertreachery that Mozilla’s 2016 “addon signing” feature was meant to catch, and so we took the warning seriously.
Back at Firefox version 44 in early 2016 (we’re currently at 66 – updates come out every 42 days, or 6 weeks), Mozilla decided to stop allowing unsigned addons in the browser, effectively appointing itself as the custodian of addons, in the same sort of way that Google decides who gets into the Play Store.
Two questions immediately came to mind:
- What had caused this apparently hacked version of NoScript to show up now, and where had it come from?
- Given the importance of NoScript in the Tor Browser’s default protections, was it still safe to have Tor open at all?
A quick search of NoScript’s own web page, plus a minute or two on the various social media channels, revealed the reason, but not the explanation:
All users who're complaining about @noscript being suddenly disabled, here's what happening: @mozilla has probably forgot to handle certificate expiration for all add-ons signing, and couldn't figure out yet how to fix this đ€Ź https://t.co/3MP8htn8hT
— Giorgio Maone đ @ma1@todon.eu (@ma1) May 4, 2019
NoScript hadn’t changed and its digital signature was still valid and unexpired…
…but Firefox no longer trusted it, and so Tor Browser wouldn’t (indeed, for most users, couldn’t) load it any more.
The bug is somewhere in Mozilla’s signature verification, not in the addon itself – and the bug seems to affect the validation of every addon in pretty much every version of Firefox.
Indeed, ten or fifteen minutes after Tor scared us, our running copy of Firefox decided that its addons were no longer safe and killed them too. (We only use one third-party addon, a screenshotting tool, but reports suggests that any and all addons that you have will simply get killed off.)
What to do?
Mozilla has pushed out a temporary patch, referred to as a hotfix, but it only works if you have Mozilla’s Studies feature turned on.
Studies is a bit of a euphemism – what it really means is “let Mozilla collect data from your browser, as well as push out test code that’s not yet part of the main release.”
It’s turned on by default, but we – and probably many of our Firefox-using readers, too – have turned it off, on the grounds that the easiest way to ensure that the data that’s collected about you never leaks is simply not to let it get collected in the first place.
And there’s no way to get hotfixes or temporary patches delivered by means of Studies if you have Firefox’s data collection option turned off.
To check if you have Studies activated, and to enable it in order to get the hotfix if you wish, go to Preferences → Privacy & Security → Firefox Data Collection and Use:
An interesting – though hardly unexpected – irony for Tor Browser users is that Studies is not just off by default in the Tor build, but actually omitted entirely on the grounds that Tor users never want to be tracked.
So Tor users can’t get the hotfix and need to turn off “addon signing” altogether instead.
Go to the special “don’t try this at home” page about:config, find the option xpinstall.signatures.required and flip it from true to false:
Here’s what the about:addons page will show, if you have the buggy version of Tor, with “addon signing” turned on (the default setting, above) and turned off (below):
Quick fix
In short:
- If you use Tor Browser, turn off
xpinstall.signatures.requiredtemporarily. Remember to turn it back on when the official fix for this bug comes out. - If you use regular Firefox, check whether you have Studies enabled if you want the hotfix. If you like to have data collection disabled, remember to turn it back off when the official fix for this bug comes out.
Anonymous
You can also “roll back” FireFox to an earlier version by using their “ESR” download.
Fixed FF for me by dropping it back to version 60 something instead of the 66 it was on.
Paul Ducklin
If you have the ESR then you should be able to use set xpinstall.signatures.required to false succesfully…
…but now 66.0.4 is out then you could just upgrade to that instead (may happen automatically but you can force the issue by doing Firefox → About Firefox from the menu bar).
For those not familiar with ESR, it’s short for Extended Support Release and it’s basically a Firefox build that gets only security updates and bug fixes for several months while the regular version ploughs ahead with new features as well. ESR then gets all the missed features in big lumps, just under once a year. When a new ESR comes out, the old one still exists for a further two releases so that even the tardiest and most fearful change control committee ought to have time to make up its mind.
Anonymous
I use an ESR version and I can confirm it blocked NoScript. Changing the config did work, but you’ll have to remember to change it back.
It’s odd that it seemed to be NoScript in particular that was affected. The Firefox error and help page was also misleading and it suggested installing a newer version of the addon (which failed again because of the signing error).
Brian
Might it be apropos to advise against installing/upgrading any add-ons until after fixed? Or verify the hash of the certificate used to sign any addon installs or updates? Perhaps especially when on shared networks?
Paul Ducklin
Youâd hope that an addon downloaded from Mozillaâs official repository would have passed its digital signature test before being accepted in the first place, and that the repository hasnât been hacked…
…but the real problem here was that existing addons, including ones that there was no reason to distrust and that were there for security purposes (e.g. NoScript), stopped working – apparently mid-session while you were actually interacting with a website.
Wilbur
>> apparently mid-session while you were actually interacting with a website.
Absolutely – I had just gone to the Adobe site for the first time in several years and as soon as the page loaded the “Add-Ons Disabled” notice popped up. My first reaction was instant confusion over how Adobe – or any other site – could have done that. Later, a little web searching revealed the source of the problem along with the hotfix of turning on “Studies”. I had previously disabled it, so I re-enabled and within a half hour or so the problem was patched. I disabled “Studies” again immediately after verifying Add-Ons were working and everything continued working. So “Studies” didn’t need to be left enabled for the hotfix to work, although that’s academic now that the new release is out.
Alastair Millar
Annoyingly, even after 66.0.4 was released, it didn’t download automatically, and the ‘about’ page declined to recognise that it was available – I ended up having to do a manual download from the website to get it.
Paul Ducklin
There might be a random delay before the update is made visible to different users and regions – to prevent download surges.
If so, then in cases like this where people will want to update sooner rather than later, the irony is that they will do exactly what you did and manually fetch the much larger installer instead :-)
Thanks for the comment – the bottom line is that if you click âAbout Firefoxâ and see 66.0.3 as your version, then an update *is* available even if the âAboutâ dialog doesnât say so.
Arlyne Alcaraz
I’ll wait for Mozilla to finish their fixing about add ons problem, so far I’m happy & satisfied using Firefox Mozilla browser except that I lost my theme add instigating. More power Mozilla Firefox. đ
Arlyne Alcaraz
I only lost my theme add ons not instigating, wrong clicked. â
Les Bugs
“in order to get the hotfix if you wish, go to Preferences â Privacy & Security â Firefox Data Collection and Use” – Beware it is Optionsâ Privacy & Security , NOT Preferences â Privacy & Security
Paul Ducklin
Ah, on the Mac, application preferences are called Preferences rather than Options :-)
But you are right, if you canât find Preferences then Options is where you want to go.
For all that I recognise the value that the community gets from Studies, Iâd generally suggest turning it off – after all, when they say Studies, itâs *you* who are the subject of the study…
jescott418@comcast.net
Complete and utter fail for Mozilla and Firefox team to let a expiring key fail. Surely they are given ample warning of impending doom and what the effects of this would be? Pretty hard to defend Mozilla on this and for me its the last straw. If you can’t get the day to day stuff done then why should I trust your other initiatives?
Paul Ducklin
To be fair, they did fix it pretty quickly, there was a half-decent workaround, and your exposure to danger wasnât too bad (though it did freak me out for a minute or two).
So itâs a bad look for Mozilla – they decided they ought to appoint themselves Kings and Queens of Addons and then dropped the ball…
…in a very visible way.
But if this is enough to drive you away from Mozilla youâll probably have to switch browser every month as each vendor in turn fixes a bunch of RCE bugs and the occasional 0-day, and youâll never be able to trust Google at all considering how malware prone its own âaddon communityâ is (thinking of Play Store here). So if you distrust Mozilla as a whole on account of this, that kind of rules out Chrome and Blink…
…over to WebKit browsers, and that could be said to be âtaintedâ by Apple, and so on.
But, yes, bad look for Moz.
Vince Medlock
This was truly the last straw. My computer belongs to *me*, not to Mozilla! *I* decide which software runs on it, not Mozilla. Fortunately, with just the tiniest bit of know-how, this ill-advised central-control scheme can be turned off on my OS of choice. I’m now using Waterfox, and I don’t really envision ever returning to Firefox voluntarily so long as this massive cockup is even possible.
Paul Ducklin
So what youâre saying is that you hate Firefox because it gets to control what you can run…
…and then you say that itâs really easy to turn that control off so you can run what you want?
I get your point about Mozilla âknowing betterâ and then screwing up, but your rant seems like much ado about nothing to me.
With the knowhow you insist you ought to be allowed to exercise, you are able to exercise it…
Vince Medlock
My point was that users shouldn’t *have* to know how to do an end-run around the programmer in order to use *their own machine* as they see fit.
And “hate” is a really strong word that I didn’t use. Extreme “disappointment” and “distrust” would be more accurate.
Paul Ducklin
You didnât use the word âhateâ – I did, because I found your comment needlessly vitriolic and, to be honest, hateful.
Seriously, you think that Mozilla making its own addon store be âsigned onlyâ by default literally âstops you doing what you want with your computerâ? Did someone force you to use Firefox or prevent you getting its source code and do what you want with it? Mozilla screwed up and fixed it – bu you make it sound as though there was some kind of malevolence in the mix.
(Whenever you are in a shop and see Chromebooks or iPhones, you must go apoplectic! Honestly, for all that this is a bad look for Mozilla… letâs get a sense of proportion here. It was a worrying inconvenience but it was not something that tried to âstop you using your computer as you see fitâ, and itâs mostly sorted out. If you think Mozilla is incompetent then switch to another browser.)
Vince Medlock
You are, of course, entitled to your opinion. As I am to mine. But please refrain from putting words in my mouth. I am more than capable of doing that on my own.
Yes, Mozilla has reached into my computer and prevented me from doing what I want with *my own equipment that I own*. This is more than bad optics for Mozilla. If anyone is actually paying attention, this is the end of Mozilla.
Mozilla hasn’t fixed anything yet. They’ve forced everyone to open up worse security holes in the browser (benefiting Mozilla’s data collection efforts, BTW). They’ve made the situation even worse by doubling down on a bad decision that was predictably disastrous.
J
You did exactly what you wanted with your machine. You trusted a source other than yourself, and installed a program that you yourself did not code. That choice sometimes has consequences. If you want total control of your computer, write your own programs or realize that using software that others create comes with risk. You’re free to choose how much risk is too much.
Vince Medlock
And I’ve done exactly that. The risk of continuing to use Firefox under the current paradigm is too great. The risk that they will take control of my machine and do something that I don’t want, as they did on Friday, is unacceptable.
Paul Ducklin
I think you made that point already – you switched to Waterfox because you think that Firefox “took over” your computer. The fact that you could freely switch to a new browser (ironically, one that is directly derived from Firefox) is an interesting form of “taking over your computer”… but good luck with your journey.
Vince Medlock
And I also never said it was “easy.” I said it was possible with some technical skill and willingness to use CL tools.
Paul Ducklin
You made it sound pretty easy to me – âfortunately, with just the tiniest bit of know-how, this ill-advised central-control scheme can be turned offâ were your words.
I think we can be kind enough to assume that the average user could acquire âthe tiniest bit of knowhowâ without too much of an intellectual battle…
Vince Medlock
If you truly believe it’s that easy, go ahead and manually edit the settings files. Linux will allow it. Mac will allow it if you bypass a few safety precautions. I have no idea if Windows allows it.
Tony P.
FYI, on my W10 PC and Firefox 66.0.3, to find the Firefox Data Collection and Use section, it was under:
Tools â Options â Privacy & Security â Firefox Data Collection and Use:
Paul Ducklin
I donât want to sound like a Mac fanbuoy, but perhaps I am… Preferences is a macOS standard; as you say they are Options on other platforms. Thanks for the note.
Steve T
Has this now been fixed? Last night my Tor browser updated to a newer version .. (8.0.9 I think .. not at my computer at the moment..)
Thanks for the heads up on this.. greatly appreciated!
Paul Ducklin
Yes, as mentioned in the update box at the top, 8.0.9 is supposed to fix this (and seems to do so) so that the âcheck signaturesâ setting can be toggled back to true.
Steve Tate
Sorry Paul.. didn’t see that.. thanks again for an awesome service.
Paul Ducklin
No worries – glad you got the update!
HackerJack
For me it’s clear: The US LEA and NSA and SpyIndustry compromised Mozilla and TorProject to push all users to some action / reaction and to kill the TOR network and any system to avoid Java and Java Script. With this initiative, the US SpyGoverment tries to take over Oracle (Java) and Mozilla (70% of worldwide browser) and the TOR security. The TOR project is already sponsored and owend by US military / gov. With this simple certificate trick they do the last step in overtaking most of the worldwide security infrastructure.
Paul Ducklin
That sounds extremely unlikely – especially the Java bit, given how few users even have browser Java these days (can you even install Java into the Tor Browser?).
Cassandra
Is it being excessively “tin-hat”ish to wonder whether it was a genuine cock-up (by a well respected company not renowned for major cock-ups) or something deliberate by someone to cause certain people (possibly a very small group) to temporarily change their security settings for just long enough for something to happen?