Cómo una auditoría de seguridad de rutina encontró un APT de nivel de núcleo en más de 100 gateways

Eight thousand one hundred ninety-two bytes. That's the exact surplus encrypted data Igor found inside an NFS read reply while doing a routine privacy audit — and that number is what unraveled a kernel-level APT hiding on over 100 production reverse gateways.

The Setup: 100+ Gateways with No Local Storage

Igor's client ran a custom Linux stack on a fleet of reverse gateways: a monolithic kernel without any modules and a static Golang application acting as both init and reverse proxy. The machines booted from external USB media — no internal storage — and mounted an NFS share hardcoded in the app for configuration, TLS certificates, and blacklist data. No SSH, no local disk. A clean, minimal attack surface, on paper.

Klaus, the POC, wanted Igor to test if an attacker could access PII from within the decrypted HTTP stream. Igor set up a test environment and a laptop with a mirrored port to capture traffic on both sides of the gateways. He built a Python script to correlate TLS four-tuples with PII, using an X-Orig-Connection header the gateway appended. Where he expected a normal distribution of packet timing, he found a skewed histogram for PII traffic — something was off.

The Anomaly: 8192 Extra Bytes in NFS Read Replies

Working late with no config files to validate his setup, Igor realized the gateways restarted daily in staggered 10-minute intervals, each reloading its NFS config. He could capture that NFS traffic from his mirrored port. After extracting an NFS read reply from production, he noticed the EOF value was 77685 — exactly 8192 bytes more than the Read length. The actual data was encrypted (uniform entropy), but the NFS file he expected was plaintext. When he mounted the same export from his test machine, he got a normal EOF of 1.

Comparing captures, he spotted the discrepancy: other NFS open requests had clean headers. These extra bytes weren't part of the legitimate file — they were an appendage, encrypted and constant across reboots.

The Eureka Moment: A Patched Kernel and a Hidden Golang Binary

Igor eventually traced the extra bytes to a patched kernel on the boot media. The kernel had been modified to read a hidden Golang binary from the USB drive at boot time, append it to NFS read replies, and then decrypt and execute it on the gateway. The binary was the real application — the one visible in the NFS share was a decoy. The attackers had compromised the USB boot media, patched the kernel image, and installed a backdoor that survived every reboot.

Igor's post details how the kernel patch worked: it intercepted NFS read operations, injected the encrypted payload, and let the kernel's own decompression / decryption routine load the implant. The Golang app on the NFS share was left intentionally vulnerable to distract from the real action happening at the kernel level.

The whole investigation started with a skewed histogram and ended with a boot-chain compromise that had been running for months. The next time you audit a system, don't stop at the application layer — check what's actually booting from that USB drive. The 8192 bytes were the only giveaway, and Igor caught them at 3:34 AM.

Source: Apt Encounters of the Third Kind
Domain: igor-blue.github.io