LISTEN NOW
Click-and-drag on the soundwaves below to skip to any point. You can also listen directly on Soundcloud.
With Doug Aamoth and Paul Ducklin. Intro and outro music by Edith Mudge.
You can listen to us on Soundcloud, Apple Podcasts, Google Podcasts, Spotify, Stitcher and anywhere that good podcasts are found. Or just drop the URL of our RSS feed into your favourite podcatcher.
READ THE TRANSCRIPT
DOUG AAMOTH. Log4Shell, Log4Shell, Log4Shell, and Apple security updates.
All that and more on the Naked Security Podcast.
Welcome to the podcast, everybody: I am Doug; he is Paul.
[IRONIC VOICE] And, Paul, what a boring, mundane, uneventful week in the world of cybersecurity!
PAUL DUCKLIN. I’ll tell you what, Doug: I’ve actually been waiting all weekend for the Fun Fact, because I could do with some of that right now.
DOUG. Great, because I’ve got a perfect Fun Fact for you.
You could call it the “ultimate beta test”: before the first powered flight, brothers Orville and Wilbur Wright had made more than 700 successful test flights in an unpowered glider.
We’ll talk more about the Wright brothers later in the show.
DUCK. Cool.
That’s a great bit of history, isn’t it?
Because they really did everything [CHUCKLES] “right” from a science and engineering point of view.
They didn’t know how to fly, because nobody had done it before (powered flight, that is), but they didn’t just make it up as they went along – they followed a solid process.
DOUG. Yes, Sir!
So, we shift gears to a not so solid process: Log4Shell…
DUCK. Or maybe a process so solid…
DOUG. A little TOO solid! [LAUGHS]
DUCK. …so heavyweight that it’s sinking us all like stones.
OK, exaggerating for effect there, Doug, but not by much.
DOUG. Lots of talk about this bug, and lots to unpack.
What is the bug? And I guess my secondary question: is this really a bug, or a feature that has been exploited?
DUCK. All good questions.
And just for anybody who’s missed out, we are talking about a bug that goes by the name “Log4Shell”; you’ll also see the text “Log4J”, which is a Java programming library from Apache that means “Logging For Java”; and you will see it referred to by its official name CVE-2021-44228.
As you say, Doug, “Is it a bug or a feature?”
This Log4J programming library: it’s very, very popular; it’s been around for years.
It’s made by Apache, the same people who make the httpd web server.
And if you write Java software, which, as you know, is very popular in the back-end business logic marketplace, logging is really important.
Because logging is how you do auditing, compliance, all sorts of stuff.
So, clearly, you want a really convenient, [GOES OVER THE TOP] super-powerful, extraordinarily excellent logging system, and Log4J is supposed to be one of those.
Sadly, it has a feature that allows the data you want to log to include what are called in the jargon “metacharacters”, special characters.
Dollar sign, squiggly brackets, some special commands, squiggly brackets. [IN US ENGLISH. SQUIGGLY BRACKETS ARE ALSO KNOWN AS BRACES]
And when it writes that log data, it doesn’t write ${...} – it converts that into something “more useful”, Doug, and sadly, one of the “more useful” conversions it can do…
…and this is by design – like you say, this is a feature that was supposed to be there all along…
…[WRY LAUGH] one of the things you can do, in the data you want to log: you can put a special command that says, “Look, I don’t really know what data you’re supposed to log at this point; here is a network URL that you can connect to and look it up.”
What could possibly go wrong, Doug?
DOUG. {GASP OF INCREDULITY] Why?
So, someone like me, who understands the concept of logging but may not understand completely how loggers work…
…I was under the impression that, whatever I give you, RECORD THAT THING.
Log what I say; don’t let me put other stuff in there for you to interpret.
DUCK. Oh, it gets worse, Doug!
When you call back to me, I can say, “Look, I don’t actually know the data, but I’ve got a little Java program here that I’ll send you that can figure it out for you. Why don’t you run it and see what it says?”
DOUG. [INDIGNANT] I mean to say…
DUCK. I kid you not! That is the feature. [LAUGHS]
DOUG. I was going to say, “They’re trying to be helpful.”
But in their zeal to be helpful, it’s turned into the fact that you could just run arbitrary code.
The logee is able to control the logger, which shouldn’t be the case, right?
DUCK. It is, as you say, “Logee controls logger.”
They send a name, or a string of digits or whatever they’re transmitting, that is essentially a secret “mini-program” for the other end to run.
So I think that’s why this thing is just being abused so crazily.
It’s not like a traditional exploit.
We’ve talked on the podcast before about buffer overflows, use-after-free, bugs like that.
Those are vulnerabilities, but to exploit them, what you’re doing is you’re poking a stick into the server until it crashes.
And then, as it crumbles around you, you’re guiding your way carefully through the rubble and coming out the other end in control.
You can see why that’s quite difficult, and exploits like that can take months to perfect, and they may be super-unreliable, even at the end of it.
But Log4Shell is “by design”.
The system is supposed to let you say, “Here’s a URL: connect out to it; download the program that you find there; and, regardless of what it is, kindly execute it for me. Thank you.”
DOUG. Has it always been this way, and we just discovered it?
Or is did something happen in the past month or so that created a vulnerability whereby you can have code run like this?
DUCK. From what I’ve heard, it was a feature request to the Apache Log4J team many years ago – eight years ago, or something.
Someone said, “Hey, it would be really handy if I could do a directory services lookup from the log. Why don’t you put it in?”
And it is just that, lately, people figured out, “Hey, you can do cool stuff with this.”
From what I’ve heard, some of the kids on Minecraft…
BOTH. [LAUGHTER]
DUCK. It’s not funny, yet it is funny.
Some of the kids on Minecraft figured, “Hey, how cool is this?”
So word got out, and proof-of-concept stuff appeared, and notes on Twitter arrived.
So it turned into a zero day, a very visible zero day.
Apache were fairly quick to a-patch it [LAUGHTER], but Pandora’s knowledge was out of the jar by that time, and it was too late to get it back in.
DOUG. OK, so the big picture right now: what is happening across the internet?
Because a lot of companies are using this logging library.
What are the criminals using this vulnerability for… do we call it a vulnerability? Or is it a feature?
DUCK. I think we do have a vulnerability here.
The news is, and SophosLabs is tracking this as vigorously as it can, that there’s an awful lot of smoke/smog/steam/mist, which is not helping…
…but in amongst that, there are some very definite fires burning, and it seems that one of the main things that active cybercriminals are doing, as you can imagine – we’ve spoken about this recently – is cryptomining.
I’m assuming because, if it’s a server, and it’s a business-logic server, it might be quite powerful!
DOUG. [LAUGHS] Uh-huh!
DUCK.So it seems there’s quite a bit of cryptomining going on.
And, as we mentioned before, that can be expensive.
If you’ve got a server that typically doesn’t do much, and it’s in the cloud, and it’s currently costing you $5 a month…
…you do not want to wake up to find that it is now costing you $5000 a month so that some crook can skim off the rewards from cryptocurrency transactions.
Another thing that crooks are doing, and maybe we can come back to this later on, is they’re using the vulnerability to exfiltrate data.
They poke in code that reads something interesting that they suspect is stored on the server, and send it back to themselves: they’re using the bug for data stealing.
So that seems to be the main things that are happening right now: cryptomining; data stealing; and implanting general zombie code that can be instructed later on to go and download something else.
DOUG. And it sounds like it’s widespread enough that it could be causing enough havoc, and diverting enough attention, that it could possibly cause a smokescreen for other criminal activity while people are dealing with this.
So I guess our advice would be that you should deal with this, but you should also keep an eye on other things, because everyone’s distracted right now.
DUCK. Yes!
As you said at the very top of the show, Doug, “Log4Shell, Log4Shell, Log4Shell… and Apple patches.”
I’ll give the game away in advance: the Apple patches actually have nothing to do with Log4Shell, but they do have everything to do with a whole number of other vulnerabilities, apparently including jailbreaking the kernel.
Which is implant-secret-spyware kind of stuff.
So you are quite right.
But I don’t know whether the crooks are *actively* figuring, “Hey, this is a great smokescreen.”
I think the problem there is don’t they don’t need to.
DOUG. So, if I’m a company, and this is very widespread, at what point can I assume I’m not affected by this?
Is it, “I only have a public facing server and there’s Java code on it”?
Or does it extend beyond something like that?
DUCK. That’s one of the most important questions to know the answer to.
Because I have seen some responses – obviously, they’re flying around the internet, what companies have said to their customers – using words to the effect of, “Don’t panic, we’re golden. All our public-facing servers are non-Java.”
That was perhaps what a lot of companies assumed at first.
They figured, “I’ve got IIS from Microsoft; I’ve got Apache httpd, which is not written in Java, it’s written in C; I’ve got nginx; I’ve got all these servers; none of them are written in Java. Phew!”
And the problem is that, because this is associated with logging untrusted user-supplied data, the bug can be triggered on a server that is not directly connected to the internet.
All that has to happen is user connects to your web server fills in a web form where they put in username, address phone number: “Dollar, squiggly brackets”, etc.,
DOUG. [LAUGHS]
DUCK. They send it to your web server…
…your web server is going to package up that data.
Some of it, the web server might log itself; the rest of it, almost certainly in many companies, is going to get bundled up and passed off deeper into the network, sent off to your back-end servers, your business logic servers.
So the real problem is that it’s not just your public facing servers that you need to worry about.
So what that means, really, is that you can’t just think about defence-in-depth in this case.
You have to think about defence-in-breadth – you need to scan your whole estate.
And, at the first cut, all you really need to do is look for files called log4j*.jar, where JAR is a Java archive.
That’s Step One.
And, Step Two, I recommend this: make yourself a nice, strong cup of tea.
[CHUCKLES] Prepare yourself for an unpleasant surprise.
DOUG. [LAUGHS]
DUCK. Because I suspect that many organisations are going to find that they have a lot more copies of Log4J than they ever expected.
DOUG. That’s my next question: if I’m a home user, should I be searching my own machine for this?
Does this affect home users as well?
DUCK. There’s no reason not to.
We’ve got some instructions on how to do it on Naked Security – some files you can look for.
However, a lot of people at home don’t consider themselves IT experts, and for that reason have gone out to a service provider for the family web site, or for the discussion forum that they use for the sports club they belong to.
You probably want to go to those providers and find out whether the stuff that you’ve got there, content about your family, and about your customers if you have any, is safe and sound.
I’d recommend not emailing them first – go to their website and see if you can find a statement.
A lot of companies have already managed to do the right thing, and they have statements where they list all their products.
DOUG. Very good!
Let’s talk about the patch a bit.
Does the patch work?
And, I guess more importantly, what do you do if you can’t just roll out a patch to a billion machines in the next 15 minutes?
DUCK. The good news is that if you update to Log4J 2.15.0, then it protects you from yourself and from your logs.
I wrote a quite detailed article – but user-friendly, it’s not too technical – on Naked Security that charts my own journey through building a deliberately vulnerable system, applying the patches, and validating for myself that they actually worked.
The other thing that I did, in case you can’t apply the patch, is that there are various ways of altering the runtime configuration of a system you already have, without changing any of the code.
You just change the commands you use to run it, or the configuration settings that you use when it starts.
What those mitigations do is they set what are called Java system properties that force the logging library into a state that says, “No outside lookups allowed.”
And I validated those mitigations as well, and I show how you can apply that fix and check that the bug stops working.
So yes, there is a patch; it’s surprisingly simple; and if you haven’t started, then you’re a bit late to the party.
DOUG. [LAUGHS]
DUCK. The problem is, with this threat, that’s so broad, and so easy to exploit, and so easy to obfuscate by using all sorts of different characters in the magic dollar-squiggly-bracket stuff…
..,that really you have to be part of the solution, not part of the problem.
You can throw technology at it, but you do need also to go and find where you’ve got this thing and fix it.
DOUG. The next is a somewhat related question, given that many medium-to-large-size organisations are leveraging a ton of different servers, whether they be in the cloud or different technologies from different vendors…
…does this start to feel kind of like a supply-chain issue, because of lot of the third-party services that a company might be leveraging could have this Java exploit?
DUCK. Yes, I think you can consider this a supply-chain attack.
When I scanned my laptop just for fun, just to see what was there, I found apps that I could barely remember I had; that I had no idea used Java; and, even if I did, I would not have on any magic list that said they included the Log4J code.
So in that sense, it is a supply-chain attack.
And I guess that’s why it’s important, whether you’re a small, medium or large business, if you can publish a short, sharp, clear document, easily accessible on your website.
That is a great comfort to your users, because what it means is that you’re saying, “I know the bill of materials in my products, and I have found out whether they have the vulnerable code or not, and I’m prepared to disclose that to you simply, clearly and in plain English, without lots of lots of gobbledygook.”
DOUG. All right, we’ve done some good work here.
It may not be over, but let’s put it to rest for now.
We’ve got two articles up on Naked Security, and we’ve got another technical article on Sophos News.
Plenty of things for the good people to read, and ingest, and put to use.
So we will leave you with that and take a quick break for This Week in Tech History – a little palate cleanser, if you will.
We talked about the Wright brothers earlier in the show, and on 17 December 1903, Orville and Wilbur Wright made their first successful powered airplane flights: four test flights were made that day.
The first, by Orville, lasted about 12 seconds, and the fourth, by Wilbur, lasted a record 59 seconds.
Now, the brothers couldn’t find an automobile engine both light and powerful enough to propel their airplane, so they had one fabricated in their bicycle shop by their shop mechanic, Charlie Taylor.
Taylor built it in just six weeks.
So we hear a lot about the Wright brothers, but not a whole lot about Charlie Taylor, one of the true heroes of flight!
DUCK. Yes, that’s a very funky story, isn’t it?
“We can’t get an internal combustion engine with the right characteristics. The automobile guys aren’t interested in this. They don’t even know what we need. We’ll start from scratch.”
DOUG. Yeah, build our own engine, no problem!
Well, let us now get back to cybersecurity and *not* talk about Log4Shell.
We’ll talk about a whole slew of Apple security updates without a mention of Log4Shell, so what’s going on with thsse updates, Paul?
DUCK. Last night after I finished my magnum opus about how to understand, assess, patch and verify your response to this Log4Shell vulnerability…
…to my surprise, I found there was an update for my Mac, and I thought, “Ahaha, it can only be, surely….
DOUG. [LAUGHS] Aha!
DUCK. And of course, like you said earlier, could there be a smokescreen?
Could this be a distraction?
What if we focus on Log4Shell and we forget everything else?
So, the first thing I did is that I set the download going – it was 1GB or something- and then I jumped onto the the Apple security bulletins to find out, “What did they fix?”
So I set my Firefox search to 44228 and I went through all the bulletins waiting for that to pop up. (That’s the CVE number for Log4Shell.)
Nothing!
So I thought, “Well, that’s interesting, because the bulletins are quite long,” and it turns out there are loads of other things that were fixed.
DOUG. Some of these look pretty serious.
We’ve got kernel level remote code execution.
DUCK. Yes, I think that’s the big one.
And from what I’ve read elsewhere (I haven’t checked the veracity of this yet), apparently this was essentially a jailbreaking bug.
That’s basically where someone doesn’t just break out of one app, but takes over the operating system completely.
Now there are hackers out there love to do that to their own iPhones, because they think it’s a great idea – we do *not* recommend it, particularly for business phones, because there’s an awful lot that can go wrong….
…but as a lot of our listeners will know, the problem with jailbreaking or iPhone “rootkittery” is if somebody can do it without you realising, then it’s the ultimate way to implant spyware, isn’t it?
NSO Group, Pegasus, etc., etc.
Those are the kind of vulnerabilities that could lead to full device takeover, and are very definitely things that you want to patch on your mobile phone.
DOUG. Yes!
And then tracking flaws, that sounds bad.
DUCK. Apparently, there was a way that you could make tracking suppression features not work properly,
It doesn’t sound terribly severe.
But if you’re thinking, “Well, the good thing about my Mac, or my iPhone, or my iPad, at least Apple have put tracking limitation features in”, that’s a bad thing to think if they’re not working properly.
DOUG. Yes, another bad sounding one.
Malware detection bypasses?
DUCK. This, as far as I know, applies to the the the system that Apple calls Gatekeeper, which is sort of their rudimentary built-in antivirus.
My understanding is there are some ways that malware can sidestep that.
DOUG. And then two related ones: network traffic leakage and memory leakage?
DUCK. Yes, obviously, you’re assuming that a process or software that is not supposed to be able to spy on your network traffic, or spy on your the details of your memory usage…
…you kind of assume that those protective features work.
Apparently, however, there are bypasses for both of those.
So there’s a way that a rogue process or server could get to look at your network traffic when really it’s not supposed to.
And there’s also a way that details about what’s in memory, including the layout of memory, could be accessed by processes that aren’t supposed to know it.
The thing with memory disclosure problems, as you can imagine, is, “What if they could suck out all my passwords?”
But sometimes, on modern operating systems, you might get a memory leakage that leaks something really, really, modest-sounding, like a memory address, which is just a number where this program lives in memory.
And of course, as you know, modern operating systems go to great trouble to have what’s called address space layout randomisation (ASLR), precisely so you can’t guess or work out where programs are in memory.
Because that makes them harder to attack with memory-based exploits.
But if you can leak even just an address out of memory and figure, “Aha, so that’s where you’re hiding XYZ feature in the kernel this time,” then you just broke ASLR.
So that’s another one you definitely want to patch.
DOUG. All right.
And this is a always a bad sign: elevation-of-privilege?
DUCK. That often used to be considered – some people still do consider it – a sort-of second tier vulnerability.
It’s only a silver medal on the exploit table, because obviously the one everyone wants is remote code execution.
But, if you think about it, elevation-of-privilege (EoP) and remote code execution…
…for the crooks, they go fantastically well together.
When you do your remote code execution, like we talked about with Log4Shell, you get to inject the code into the server process.
Now, if the server has very restricted rights, that might do less harm than you’d hoped.
But if you can combine an attack where you get some user rights with a trick that lets you grab more user rights, as long as you’re inside the network first, then basically you’ve turned a minor remote code execution flaw into a potentially huge one.
So, I never really consider those as second-tier exploits these days.
I figure, when I see an elevation-of-privilege patched, and it’s got a CVE number…
…I think, “I wonder which remote code execution exploit it was paired with.”
DOUG. Tes!
All right. And then last but certainly not least, privacy bypasses?
DUCK. Obviously, the tracking flaws that you mentioned affect your privacy, but that’s was down to what the wireless networking system was saying outside your computer.
These bypasses, as I understand it, are ones that allow locally running programs to read and write data that is supposed to be off limits.
You can see why that’s bad because it helps you: steal data; make unauthorized changes to system configurations; sneakily drop malware scripts; all sorts of stuff.
DOUG. Patch early, patch often!
On iOS: Settings > General > Software Update.
On your Mac: Apple menu > About this Mac > Software Update…
And Paul, just because we didn’t talk about Log4Shell, does that mean it can’t, in theory, affect Macs?
DUCK. In theory, as we said before, you could be vulnerable on a standalone computer or laptop.
This bug is not about being connected to the network; it’s about taking untrusted data and supplying it to a program that you think will use it harmlessly but ends up using it in a way that is malicious.
So you may nevertheless wish to go to that Naked Security article we mentioned earlier, and check out your Mac.
The risk is low, but why not check?
You will not be any worse off for looking.
DOUG. OK. That is “Apple security updates are out and not a Log4Shell mention in sight” on Naked Security.
And we will bring it home with the Oh! No! of the week.
I feel like I need a nap, and I am dehydrated… we’ve been through a lot this episode.
DUCK. It’s all that fire and smoke, Doug, [PUTS ON CROAKY VOICE] hurting your throat.
DOUG. Reddit user “ShakeYourBooty” writes…
Back in my early days of tech support, CRT monitors were very much the norm.
One user called in and told me, “My mouse is stuck and the cursor on the monitor is no longer moving.”
While trying to get more information from him, I kept hearing him grunt with effort, followed by a loud thump in the background.
After the third such instance, during the call, I finally asked, “What’s going on?”
“Oh, I’m trying to see if I can shake loose the mouse on my screen.” [THUD]
He was picking up his monitor off his desk and dropping it down in an attempt to “loosen” it.
I quickly told him to stop before he hurt himself.
DUCK. Oh, did he think the mouse cursor was like the minute hand on a clock?
That it was actually a physical thing inside there that moved around with a little magent, “Wheeee!”
DOUG. [LAUGHS] Yes!
I want that to not be true; I want that to be a made up story.
But you know, doing a little technical support myself in my younger years, I can’t put it past some people.
BOTH. [LAUGHTER]
DUCK. I did once have a problem on my Mac, many years ago, where there was a blob on the screen…
DOUG. [LAUGHS]
DUCK. It wast about the size of of a of a full stop, a period, in boldface when the font’s big.
It obscured just enough to be really, really annoying.
So I got my screen cleaner out, and I tried to clean it and I couldn’t, and I thought, “That’s funny, it’s not moving with the text on the screen, so it’s not an *actual* period in my article going up and down,” but I thought no more of it.
And then, a little bit later, I thought, “Oh my Lord, I wonder if I’ve got malware!”
It started moving of its own accord, this little pixel, moving across the screen!
DOUG. [LAUGHS]
DUCK. And then when I looked really closely, I realised a tiny little fly had got inside.
It was actually walking around on the side of the screen, as far as I could make out.
DOUG. [LAUGHS]
If you have an Oh! No! you’d like to submit, we’d love to read it on the podcast.
You can email tips@sophos.com; you can comment on any one of our articles; or hit us up on social: @NakedSecurity.
That’s our show for today, and a little housekeeping: we will be off for the next two weeks in observance of the holidays, so have a great holiday break, everybody!
We’ll be back with you in the New Year.
Thank you very much for listening, and for Paul Ducklin, I’m Doug Aamoth, reminding you until next time to…
BOTH. Stay secure!
DUCK. [SOTTO VOCE] And get those patches done.
Not enough time or staff?
Learn more about Sophos Managed Detection and Response:
24/7 threat hunting, detection, and response ▶
Leave a Reply