The neon signs flickered outside my office window, casting long shadows across the worn leather of my desk. The city was a symphony of whispers and secrets, and tonight, the digital underbelly was humming a tune of its own. The scent of stale coffee and cheap cigarettes hung in the air as I pored over the details of my latest case: a new case had landed on my desk, ushered in on the winds of the cyberspace – a software supply chain attack, they called it. A poison injected into the veins of the XZ utility, a tool used by millions, a seemingly innocuous piece of code which could turn an ssh server into a weapon.
They called it a backdoor, a term as old as time but with a new, insidious twist in this digital age. A hidden pathway, woven into the fabric of the code, waiting to be exploited. A ghost in the machine, a silent predator lurking in the shadows of the ones and zeroes.
The call had come from an anonymous source only hours after the discovery by someone directly connected to Andres Freund who discovered the backdoor, their voice distorted by encryption, a whisper in the digital wind. They spoke of a compromised update, of malicious code injected into the heart of XZ. A compression tool used by SSHD, weaponized via this back door, and widely available. Andres, while testing a pre-release version of Debian Linux, had noticed that SSH logins were taking longer than usual and using a lot of CPU, and that valgrind, a utility for monitoring computer memory, was generating errors. This prompted him to look at the liblzma library and beyond belief there was a disguised feature bypassing SSH authentication and allowing remote code execution on vulnerable systems.
My investigation led me down the rabbit hole of open-source development, a world where trust was currency and vulnerabilities were gold. I traced the threads of the code, following the digital fingerprints left behind by the perpetrator. The trail led me to Jia Tan, a contributor to the XZ project, a name now whispered in hushed tones in the darker corners of cyberspace.
Tan's commits were clean, his contributions seemingly innocuous. But beneath the surface, I detected a subtle shift in the code's rhythm, a discordant note in the symphony of logic. The backdoor was cleverly disguised, a chameleon blending seamlessly into the surrounding code. Andres Freund’s email to the OSS-SECURITY mailing list had a subject of “backdoor in upstream xz/liblzma leading to ssh server compromise”. He outlines his analysis of the exploit. He observed the following requirements to activate the exploit.
a) TERM environment variable is not set
b) argv[0] needs to be /usr/sbin/sshd
c) LD_DEBUG, LD_PROFILE are not set
d) LANG needs to be set
e) Some debugging environments, like rr, appear to be detected. Plain gdb
appears to be detected in some situations, but not others
To reproduce outside of systemd, the server can be started with a clear
environment, setting only the required variable:
env -i LANG=en_US.UTF-8 /usr/sbin/sshd -D
The method was ingenious: a conditional statement, triggered only under specific circumstances, a digital trapdoor activated by a secret knock. The payload? A shell, a portal granting the attacker unfettered access to the compromised system. A silent takeover, a bloodless coup.
The backdoor initially intercepts execution by replacing the ifunc resolvers
crc32_resolve(), crc64_resolve() with different code, which calls
_get_cpuid(), injected into the code (which previously would just be static
inline functions). In xz 5.6.1 the backdoor was further obfuscated, removing
symbol names.
These functions get resolved during startup, because sshd is built with
-Wl,-z,now, leading to all symbols being resolved early. If started with
LD_BIND_NOT=1 the backdoor does not appear to work.
Below crc32_resolve() _get_cpuid() does not do much, it just sees that a
'completed' variable is 0 and increments it, returning the normal cpuid result
(via a new _cpuid()). It gets to be more interesting during crc64_resolve().
In the second invocation crc64_resolve() appears to find various information,
like data from the dynamic linker, program arguments and environment. Then it
perform various environment checks, including those above. There are other
checks I have not fully traced.
If the above decides to continue, the code appears to be parsing the symbol
tables in memory. This is the quite slow step that made me look into the issue.
Notably liblzma's symbols are resolved before many of the other libraries,
including the symbols in the main sshd binary. This is important because
symbols are resolved, the GOT gets remapped read-only thanks to -Wl,-z,relro.
To be able to resolve symbols in libraries that have not yet loaded, the
backdoor installs an audit hook into the dynamic linker, which can be observed
with gdb using
watch _rtld_global_ro._dl_naudit
It looks like the audit hook is only installed for the main binary.
That hook gets called, from _dl_audit_symbind, for numerous symbols in the
main binary. It appears to wait for "RSA_public_decrypt@....plt" to be
resolved. When called for that symbol, the backdoor changes the value of
RSA_public_decrypt@....plt to point to its own code. It does not do this via
the audit hook mechanism, but outside of it.
For reasons I do not yet understand, it does change sym.st_value *and* the
return value of from the audit hook to a different value, which leads
_dl_audit_symbind() to do nothing - why change anything at all then?
After that the audit hook is uninstalled again.
It is possible to change the got.plt contents at this stage because it has not
(and can't yet) been remapped to be read-only.
== Impact on sshd ==
The prior section explains that RSA_public_decrypt@....plt was redirected to
point into the backdoor code. The trace I was analyzing indeed shows that
during a pubkey login the exploit code is invoked:
sshd 1736357 [010] 714318.734008: 1 branches:uH: 5555555ded8c ssh_rsa_verify+0x49c (/usr/sbin/sshd) => 5555555612d0 RSA_public_decrypt@...+0x0 (/usr/sbin/sshd)
The backdoor then calls back into libcrypto, presumably to perform normal authentication
sshd 1736357 [010] 714318.734009: 1 branches:uH: 7ffff7c137cd [unknown] (/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.0) => 7ffff792a2b0 RSA_get0_key+0x0 (/usr/lib/x86_64-linux-gnu/libcrypto.so.3)
This discovery sent shockwaves through the digital community. Distributions scrambled to patch the vulnerability, security experts dissected the attack, and the blame game began. But the question remained: who was Jia Tan? A lone wolf, a disgruntled developer, or a pawn in a larger game?
As the days turned into nights, the mystery deepened. The digital trail grew cold, the whispers turned into silence. Jia Tan vanished into the ether, leaving behind only questions and a lingering sense of unease. Was it a state-sponsored attack, a corporate espionage operation, or the work of a rogue hacker?
The truth remained elusive. The shadows lengthened, and the city hummed with a thousand untold stories. The XZ attack was a reminder of the fragility of trust in the digital age, a testament to the constant battle between security and exploitation.
As I closed the case file, the smoke from my cigarette swirled in the dim light, forming a question mark that hung heavy in the air. The answer, it seemed, was lost in the labyrinthine corridors of cyberspace, a secret whispered only to the wind.