A fascinating but ominous software story dropped on Friday: a widely used file compression software package called “xz utils” has a cleverly embedded system for backdooring shell login connections, and it’s unclear how far this dangerous package got into countless internet-enabled devices. It appears the persona that injected this played a long game, gaining the confidence of the legitimate main developer, and thus empowered to release new versions themselves.

Andreas Freund reported this Friday morning on an industry security mailing list, leading many experts to spend the day poking under rocks and peering into the abyss of modern digital insecurity: “The upstream xz repository and the xz tarballs have been backdoored,” Freund wrote. It cleverly pokes a hole in the SSH daemon (sshd), which is essential to modern-day computing at the most fundamental level.

The risks if this hadn’t been discovered were extreme: as noted expert @thegrugq put it: “The end game would be the ability to login to every Fedora, Debian and Ubuntu box on the internet. If it isn’t a state actor it should be…” Cryptographer Filippo Valsorda said, “This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library.”

The problem was uncovered after Freund noticed that the new version slowed down their PostgreSQL database tests, and they started debugging why this happened. It turns out the backdoor causes a tiny but noticeable slowdown in performance, a big win for picky benchmarking types everywhere.

Xz-utils versions 5.6.0 and 5.6.1 were found to have the malicious programming added by “Jia Tan” (jiaT75) on GitHub. It got into the widely used Debian testing (pre-release) branch but not the stable branch. It also made it into the widely used OS X third-party package manager homebrew (although maintainers are optimistic it may not actually affect Apple machines). Red Hat also warned that some Fedora Linux 40 systems (as well as Rawhide, the pre-release for Fedora 41) may have gotten the malicious software. Jia Tan’s activities extended beyond just the xz-utils code, which people discovered on Friday.

Linked is the best dive into how the exploit works and its distribution, by thesamesam. In the realm of security bulletins, it’s now CVE-2024-3094 with a Common Vulnerability Scoring System score of 10. Here’s a large list showing which Linux distributions have which versions involved.

As Minneapolis security expert Ian Coldwater noted, “Open source maintainer burnout is a clear and present security danger. What are we doing about that?”

This June 2022 message from the original developer confessing to burnout illustrates how Jia Tan gained control over the software:

“I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.

It’s also good to keep in mind that this is an unpaid hobby project.

Anyway, I assure you that I know far too well about the problem that not much progress has been made. The thought of finding new maintainers has existed for a long time too as the current situation is obviously bad and sad for the project.

A new XZ Utils stable branch should get released this year with threaded decoder etc. and a few alpha/beta releases before that. Perhaps the moment after the 5.4.0 release would be a convenient moment to make changes in the list of project maintainer(s). Forks are obviously another possibility and I cannot control that. […]”

Lasse Collin, xz-devel mailing list, June 8, 2022

Some observers suspect the personas badgering Collin by email may have also been sockpuppets trying to shake control away from him. In a detailed report ars technica warned that even older versions could have security problems since the bad actor made many binary test file changes over the years.

The current official page says “Versions 5.2.12, 5.4.3 and later have been signed with Jia Tan’s OpenPGP key.” A quick report by Xe Iaso warns that openSUSE Tumbleweed and OpenSUSE MicroOS also need attention and only affect AMD64 Linux systems. Debian’s warning is here.

One source with tech experience points out that embedded devices — the many gizmos besides traditional phones and computers that often don’t receive software updates — could be at risk if this software is aboard. Alarmingly, squashfs-tools, which creates disk images for programs that run routers like openwrt, had xz support added in version 4.1.

There are openwrt spinoff projects that have the bad version today, we heard as well, and one was spotted in an Android-type device using termux terminal emulator to check versions. Apparently, it got into Termux around 30 days ago. [More below on this]

For the curious, if you run the command: xz --version on Linux, OS X or other Unix-based computers, it will report if the versions linked to malicious activity are present — although not all packages would necessarily have the malicious code, since it is cleverly injected during compiling processes. Qualified sysadmins can also use a simple bash script from Freund to check if they are “probably vulnerable” or not. Affected systems seem to need glibc based C compiling, as well as deb or rpm based package managers.

This whole affair has raised the specter of malicious compiling steps as a hidden avenue of attack, by keeping the harmful code largely out of the main codebase (which is open-source and reviewable by the public). The malicious test files were added here on February 23, but their obfuscated nature, dependent on other compiling steps to take effect, concealed the threat from obvious view.

A commenter on the main hackernews thread, rwmj, posted that they’d been pushed by Jia Tan to add the 5.6 version to Fedora 40 + 41:

“Very annoying – the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it’s ‘great new features’. We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.

He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.”

rwmj on HackerNews, March 29, 2024

On March 9 Jia Tan posted some version release notes which are actually about fixing bugs introduced by the backdoor:

5.6.1 (2024-03-09)
* liblzma: Fixed two bugs relating to GNU indirect function (IFUNC) with GCC. The more serious bug caused a program linked with liblzma to crash on start up if the flag -fprofile-generate was used to build liblzma. The second bug caused liblzma to falsely report an invalid write to Valgrind when loading liblzma.

Jia Tan-authored xz utils version release notes (included in on a huge series of GitHub activities logged – click ‘run’ to see all)

The Tan account was also helpful and responsive to other coders, making it all the more integral to managing the project over a long period of time. This is a very different threat than simply stealing a programmer’s logins, quickly changing and releasing new malicious code before anyone shuts off the credentials.

Jia Tan also pushed this update in the Ubuntu bug tracker system as late as March 28.

Another interesting twist from last July involves Jia Tan messing with oss-fuzz, an important Google security checking program which uses “fuzzing” or feeding wild inputs to programs to see if they have flaws.

In a detailed note, Evan Boehs flagged the immediately suspicious code that Jia Tan submitted in 2021 shortly after their GitHub account was created, into a different project called libarchive. One can see the Jia Tan changes to libarchive and today’s attempts to fix the possible vulnerability here.

Boehs also noticed Jia Tan took over the contact email at oss-fuzz away from Lasse Collin, the original developer in March 2023.

From Freund’s original report:

“That [audit] hook gets called, from _dl_audit_symbind, for numerous symbols in the main binary. It appears to wait for ‘RSA_public_decrypt@….plt’ to be resolved. When called for that symbol, the backdoor changes the value of RSA_public_decrypt@….plt to point to its own code. It does not do this via the audit hook mechanism, but outside of it.”

March 29 OpenWall report: backdoor in upstream xz/liblzma leading to ssh server compromise by Andres Freund. Note: The triple-period ellipsis here are likely added by the listserv automatically to hide email addresses – the actual code symbol is different.

The mention of RSA cryptography functions being subverted by xz-utils also recalls the famous difficulties that the RSA security company faced when the National Security Agency seemingly tricked it into including a weakened dual elliptic cryptography algorithm, part of a secret deal involving an encryption product called BSafe that hit the news in 2013.

The Biden White House has tried to stress that modern software programming including more memory-safe programming languages are essential to improving security (PDF, and yes they plugged the Rust language). However, the authorities will also need to take note of more subtle areas like compiler steps, makefiles and test files.

As of late Friday, the status of Lasse Collins, the legitimate developer credited with the original project, remains unclear.

---

Additional technical notes:

Filippo Valsorda explained technical details of the hack’s mechanism:

“The hooked RSA_public_decrypt verifies a signature on the server’s host key by a fixed Ed448 key, and then passes a payload to system(). It’s RCE [remote code execution], not auth bypass, and gated/unreplayable.

The payload is extracted from the N value (the public key) passed to RSA_public_decrypt, checked against a simple fingerprint, and decrypted with a fixed ChaCha20 key before the Ed448 signature verification.

RSA_public_decrypt is a (weirdly named) signature verification function. www.openssl.org/docs/manmast… (Why ‘decrypt’? RSA sig verification is the same op of RSA encryption. 🤷‍♂️)

The RSA_public_decrypt public key can be attacker-controlled pre-auth by using OpenSSH certificates. OpenSSH certs are weird in that they include the signer’s public key. OpenSSH checks the signature on parsing. github.com/openssh/open…

Here’s a script by Keegan Ryan for sending a custom public key in a certificate, which on a backdoored system will reach the hooked function. gist.github.com/keeganryan/a…

Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker’s key doesn’t verify. Unfortunately, this means that unless a bug is found, we can’t write a reliable/reusable over-the-network scanner.”

Filippo Valsorda on Bluesky, March 30, 2024

• A writeup by lcamtuf looks at the difficulties of protecting low-profile software libraries that rarely need attention, and the counterintelligence threat there.
• More about the termux program [1] [2] [3] & commits involving a malicious version [4] [5]
• OSS-Fuzz cleaning up the malicious actor’s contact info
• Squashfs-tools scrutinizes code pull request added March 25 seemingly by Collins
• Security notice by OpenSUSE
• On Saturday morning Lasse Collin made another code commit, removing a single period character: “CMake: Fix sabotaged Landlock sandbox check.” (source). The original period is very hard to spot when Jia Tan inserted it in February.

The following hashes appear on code commits related to the malicious software, which can be helpful in identifying where this software turned up on places like GitHub, examples [1] [2]:

88c8631cefba91664fdc47b14bb753e1876f4964a07db650821d203992b1e1ea
0f5c81f14171b74fcc9777d302304d964e63ffc2d7b634ef023a7249d9b5d875

Cover image composition by Dan Feidt. Cartoon by XKCD (Creative Commons), code sample image by @bl4sty.

Follow us on X (aka Twitter), Facebook, YouTube, Vimeo, Instagram, Mastodon, Threads, BlueSky and Patreon.

Please consider a tax-deductible donation to help sustain our horizontally-organized, non-profit media organization: