{"id":816,"date":"2024-10-15T12:45:21","date_gmt":"2024-10-15T18:45:21","guid":{"rendered":"https:\/\/wpsites.ucalgary.ca\/jacobson-cpsc\/?p=816"},"modified":"2024-10-15T12:45:26","modified_gmt":"2024-10-15T18:45:26","slug":"the-xz-backdoor-a-10-0-cvss-exploit-we-just-barely-avoided","status":"publish","type":"post","link":"https:\/\/wpsites.ucalgary.ca\/jacobson-cpsc\/2024\/10\/15\/the-xz-backdoor-a-10-0-cvss-exploit-we-just-barely-avoided\/","title":{"rendered":"The XZ Backdoor: A 10.0 CVSS Exploit We Just Barely Avoided"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

The beginning of the calendar year is an important time for the Linux Ecosystem, as it sees the release of many important projects and distributions. Perhaps most important is the coordinated release of the prolific GNOME desktop environment, who\u2019s biannual release cycle sees a new version in March and September (1), and the Ubuntu and Fedora distributions who rely on this environment, and release new versions shortly thereafter. 2024 was an especially important year, as the new Ubuntu 24.04, Noble Numbat, was designated an LTS release (2). Whereas typical Ubuntu versions are supported for a single year before deemed end-of-life, and thus requiring servers to upgrade to newer versions, LTS releases\u2014released every two years\u2014are supported significantly longer, and an even longer period for security fixes. Ubuntu 16.04, released back in 2016, is still slated to received Extended Security Maintenance until 2026 (3), and Noble Numbat was released with Standard Support until 2029, and Extended Security Support until 2034 (2).<\/p>\n\n\n\n

On March 29th, just a week before Ubuntu 20.04 was to release, Andres Freund, a Microsoft employee, was conducting personal tests after noticing strange errors and excessive CPU usage in sshd<\/code>, the OpenSSH daemon that allows for server administrators to perform remote maintenance to their servers (4). Unlike Telnet, SSH allows for secure, encrypted access to remote computers, and with the ubiquity of cloud machines provided in services like AWS, Google Cloud, and Azure, SSH has become an instrumental tool installed on almost every Linux Server. Andres\u2019 revealed a startling discovery: The XZ compression library had maliciously compromised the OpenSSH daemon, providing a backdoor that would allow an attacker root access to any server exposing an SSH server. This vulnerability was soon to be released not only to users upgrading to the anticipated Ubuntu 20.04, but to all Debian-based distributions including the likes of Fedora and RHEL. Fedora 40, the bleeding-edge nightly version, was already vulnerable. Immediately after this discovery, distributions rushed to patch the vulnerability; Ubuntu 20.04 was postponed by a week to ensure that the backdoor would not be deployed, and the XZ repository was removed from GitHub, with the two core contributors banned from the site. But how could a compression library install a backdoor into OpenSSH? How did this backdoor work? Who was behind it?<\/p>\n\n\n\n

XZ<\/h2>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

A logo used for the XZ Compression Library, contributed by Jia Tan<\/em><\/p>\n\n\n\n

Compression is an essential aspect of modern computing. If you\u2019ve ever downloaded or created a zip file, you\u2019ve used compression. The idea behind compression is almost akin to magic: given a set of arbitrary data, produce an output that is not only smaller than the original input, but data that can be reconstructed back to the initial input. There are two primary forms of compression: Lossless<\/em>, and Lossy<\/em>. A good example of these two are in music and images; formats such as .mp4<\/em> and .jpeg<\/em> are lossy forms of compression: the output isn\u2019t exactly the same as the input, but when we\u2019re working with medium likes images the slight deterioration in visual quality is either imperceptible, or an acceptable trade-off given the significant reduction in bandwidth that we achieve. Online video streaming is only possible thanks to such compression, as most internet connections would be unable to accommodate the raw stream. If we consider a standard HD stream of 1080 * 1920 = 2073600 pixels, sent every sixtieth of a second for 60 FPS, and assuming a pixel requires three bytes to store (256 values of R,G,B), that would equate to 355 MB every second.<\/p>\n\n\n\n

Lossless compression, on the other hand is an algorithm like the one we mentioned prior: it perfectly reconstructs the original source. In contrast to .mp4<\/em> and .jpeg<\/em>, .flac<\/em> and .png<\/em> are both lossless forms of compressed media and images. Lossless is often used in these contexts either for archival purposes or for higher fidelity, but often cannot be compressed down as well as lossless might. Unlike images, videos, and music, files like binary executables and libraries cannot afford to be losslessly compressed. A slight change in the byte code would be cataclysmic.<\/p>\n\n\n\n

XZ<\/em> is a popular implementation of the Lempel\u2013Ziv\u2013Markov chain compression algorithm, often abbreviated to LZMA. LZMA is a form of lossless compression algorithm called a Dictionary Compression Algorithm<\/em>. The idea behind it is to find frequent strings within the input, and assign them a single identifier that is much smaller than the original string. We then send this compressed stream, alongside a dictionary that contains a mapping of these uncompressed strings, and their corresponding identifiers. Lossless compression is very useful for a variety of purposes, and XZ in particular is used in Linux for: compressing kernel modules (5), compressing software packages in Debian and Fedora based distributions (Including Ubuntu), and compress network communication where the output must be lossless, such as in OpenSSH (6). XZ is so widespread in its usage that its preinstalled on almost every Linux Distribution.<\/p>\n\n\n\n

Lasse Collin<\/h3>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

Someday ImageMagick will finally break for good and we’ll have a long period of scrambling as we try to reassemble civilization from the rubble.<\/em><\/p>\n\n\n\n

The XZ project was created by a single individual, Lasse Collin, who wanted to create a port of Igor Pavlov\u2019s LZMA-SDK, which the latter used to create the popular 7zip application. Created as a hobby project, Lasse has been thanklessly maintaining the project since 2009, and despite how popular the library has become, and how widespread its usage has been, Lasse remains the largest contributor by a vast margin; his contributions of over 2,000 commits total close to a quarter million lines of code added, and is over four times as many commits as every other contributor combined (7). What began as a hobby quickly devolved into a critical part of the Linux ecosystem, and Lasse was neither provided thanks or assistance in the maintenance.<\/p>\n\n\n\n

Jia Tan<\/h3>\n\n\n\n

In 2021, a GitHub account under the pseudonym of Jia Tan was created; it would lay dormant until 2022 where they made their first commit to the XZ repository. The account quickly became a major contributor to the XZ project, which would have certainly been a welcome change for Lasse (8). Their contributions were such that Jia was eventually granted the status of co-maintainer 2023; this meant that, rather than having to create a pull request to the XZ library and have Lasse merge the changes into the mainline repository, Jia could make direct commits to the XZ repository. For the first time since the XZ project had been started, Lasse was no longer the primary contributor to the project, with Jia Tan taking over a majority of the project\u2019s maintenance (9); from when Jia was granted co-maintainer in 2023, to the discovery of the backdoor in March of 2024, Jia contributed 48,000 lines of code across 404 commits, compared to Lasse\u2019s 10,000 across 283. For a point of reference, Jia and Lasse\u2019s commits made up 96% of all contributions to the project, with all other contributors combined only making up 110 of the 2657 total (7).<\/p>\n\n\n\n

The Backdoor<\/h2>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

The common consensus is that Jia Tan had always been a malicious actor, and had thus spent several years slowly building up trust within the community, being granted co-maintainer in 2023, and continuing to prove themselves as a reliable, earnest contributor (10); as such, the creation of the backdoor was slow, and meticulous:<\/p>\n\n\n\n

    \n
  • On July 8th, 2023, Jia introduced a commit in oss-fuzz<\/code>, a project used by many open source projects, including XZ. oss-fuzz<\/code> performs fuzzing, or a means of automatically testing the project for edge cases, and catch invalid behavior. The commit disabled ifunc<\/code> fuzzing for the xz<\/code> project. (14). ifunc<\/code> is a mechanism in glibc<\/code> that allows for indirect function calls.<\/li>\n\n\n\n
  • On February 15th, 2024, Jia introduced a commit that stopped the contents of the m4<\/code> folder from behind committed to the repository. This folder contained the build-to-host.m4<\/code> file which was run during build time. XZ could be compiled one of two ways: either directly from the GitHub repository, which would be the bleeding-edge, latest commit, or by downloading release tarballs that were packed versions of the repository for specific versions, such as 5.6.0<\/code>. Package maintainers for the various distributions could then download that tar file, containing a versioned, stable versioned source, and build that. Distributions like Ubuntu and Fedora build from tarballs rather than source. Importantly Jia still put the content of this m4<\/code> repository into the tarball releases, but they were not checked into the GitHub repository such that they could be easily seen. The differences between the in-tree script and the one within the release tarballs reveal how inconspicuous the changes seem to be (12):<\/li>\n\n\n\n
  • On February 23rd, 2024, Jia introduced a set of insidiously innocuous commits that added test files to the repository (11). The XZ project has a test<\/code> folder that contains compressed LZMA files; this type of unit testing ensures that new commits do not break compression\/decompression using a variety of settings, and is not out of the ordinary. However, because these test files are compressed binaries, GitHub would not be able to report the contents, and there was no reason to suspect foul play after years of contribution to the project. The build-to-host.m4<\/code> acts on these test files during the build, typically to ensure the resultant binary acts as expected.<\/li>\n\n\n\n
  • On February 24th, 2024, Jia releases version 5.6.0 of XZ, publishing the tarball containing the malicious test files and m4<\/code> scripts (16). The backdoor operators in three stages (17):\n