Why the network is the threat hunter’s ground truth — and how to think about hunting before you touch a single log.

This is Part 1 of a three-part series for SOC analysts, threat hunters and DFIR practitioners. Part 1 covers the why and the mindset. Part 2 goes deep on the technical hunts (long connections, lateral movement, beacons, DNS, protocol anomalies, outliers). Part 3 is hands-on: lab setup, tooling, datasets and exercises.

Every incident responder has lived this moment. The EDR console is green. The SIEM has no high-severity alerts. Then a third party calls: your data is on a leak site, or one of your IPs is talking to known infrastructure.

The numbers say this is still common. Mandiant’s M-Trends 2026 reports a global median dwell time of 14 days, up from 11 days the year before. For cyber-espionage intrusions the median was 122 days. One espionage campaign (BRICKSTORM) persisted for nearly 400 days on network appliances — devices that typically run no EDR agent at all. And 48% of intrusions were still first detected by someone outside the victim organization.

That gap — between “nothing is alerting” and “we are compromised” — is exactly where threat hunting lives. And in my experience, the network is the best place to start looking.

What threat hunting is (and what it isn’t)

Threat hunting is the proactive, human-led search for adversary activity that your existing controls did not detect. The key word is proactive: you are not waiting for an alert. You start from the assumption that something may already be inside and go looking for evidence that proves or disproves it.

Active Countermeasures (ACM), the team behind RITA and AC-Hunter, frames it even more sharply in their Network Threat Hunting training: a hunt is a proactive validation of every system connected to the network, executed without assumptions, which produces a compromise assessment for each device. In other words, the output of a good hunt is not “we found nothing”. It is “we checked these hosts against these behaviours, and here is the disposition of each one.”

It helps to separate hunting from its neighbours:

• Detection engineering / SOC monitoring is reactive and automated. A rule fires, an analyst triages. It only catches what someone already wrote a rule for.
• Incident response starts once you know there is an incident. Scope is defined by the alert or the report.
• Threat hunting starts with a question, not an alert. Its outputs feed both of the above: confirmed findings go to IR; repeatable patterns become new detections.

If your “hunting” is re-running yesterday’s SIEM queries and closing the tickets, you are doing monitoring. That is valuable, but it is not hunting.

Why start on the network?

ActiveCounterMeasures (ACM) training uses a line that has stuck with me for years: “you can hide processes, but you can’t hide packets.”

On the endpoint the attacker is on home turf. They can unhook EDR, kill or blind agents with vulnerable drivers, clear event logs, run in memory, or simply live on a device that never had an agent — a VPN concentrator, a firewall, a printer, an OT controller, a hypervisor. Logs on a compromised host are evidence the attacker controls.

The network is different. For an intrusion to be useful, the attacker needs to talk to it:

• They need a command-and-control (C2) channel to send instructions and receive results.
• They need to move laterally to reach the data or the domain controller or to any crown jewel assets.
• They usually need to move data out for performing data exfiltration.

Each of those leaves traffic. A sensor placed off-host, on a SPAN port or TAP, records that traffic independently of the compromised endpoint. The attacker can encrypt the content, but they cannot easily hide that a connection happened, how long it lasted, how often it repeated, how many bytes moved, or which domain was resolved first.

This is also why ACM argues against starting a hunt from raw syslog. Syslog was never designed as a security data source, message formats vary by vendor, and much log review collapses into signature matching — what they call the “1990s anti-virus model”. Network metadata, by contrast, is uniform: every connection has a source, a destination, a port, a protocol, a duration and a byte count, no matter which OS or app produced it.

The C2 lens: focus on persistence first

The most practical idea in ACM’s How to Threat Hunt Your Network is ordering your work by threat. Instead of trying to look at everything, start with the evidence that most strongly indicates compromise: persistent outbound communication.

An implant has two ways to stay in touch with its operator:

1. Long connections — keep one session open for hours or days and push commands down it.
2. Beacons — open short sessions on a schedule, just long enough to check in (“any tasks for me?”), then disconnect.

Either way, the pattern is persistent, and persistence is something you can measure from connection metadata alone. ACM’s five-step method builds on that:

1. Identify persistent communication channels leaving the network.
2. Analyse the protocol being used.
3. Evaluate the internal host originating the traffic.
4. Check the reputation of the destination.
5. Disposition: allow-list as benign, or escalate to incident response.

We will walk through each of these technically in Part 2.

Three (or four) ways to start a hunt

NetWitness’ overview of network threat hunting types groups hunts into three families. Splunk’s PEAK framework adds a fourth. Here is how each looks on the wire:

• Hypothesis-driven. You start from a testable statement. “If an implant is using HTTPS C2 from our server VLAN, we will see regular outbound TLS sessions to a destination no other server talks to.” Precise and efficient, but only as good as the hypothesis.
• Intelligence-driven. You start from external knowledge: a vendor report, an ISAC advisory, a CERT bulletin. “Campaign X uses DNS TXT records for tasking and domains registered in the last week.” Great for distributed environments where threats change faster than rules.
• Anomaly / baseline-driven. You start from what normal looks like and hunt the deviations: the one host talking to a new country, the only user-agent seen once, the server that suddenly uploads 20 GB. This finds novel attacks and insiders, but it needs mature data and analysts who know the environment.
• Model-assisted (M-ATH, from PEAK). You use statistics or machine learning to score behaviour — for example, a beacon score — and humans investigate the top of the list. RITA’s beacon scoring, which we use in Parts 2 and 3, is a simple version of this.

In practice, good hunts mix them. A threat-intel report gives you the hypothesis; a baseline tells you what is abnormal; a model ranks the candidates.

PEAK’s three phases are also a useful wrapper for any hunt: Prepare (pick the topic, research, check that the data exists), Execute (analyse, pivot, investigate) and Act (document, automate the detection, communicate).

Where hunting sits on the maturity curve

Two classic models help explain why network hunting pays off.

The Hunting Maturity Model (David Bianco, originally at Sqrrl) goes from HMM0 (relies on automated alerting only) to HMM4 (most successful hunts are automated into detections). Most teams I work with sit at HMM1–HMM2: they collect data and can search for IOCs, but rarely create new analytics. Network hunting is one of the fastest ways to climb, because the analytics — duration, frequency, volume, rarity — are simple to express and automate.

Bianco’s Pyramid of Pain explains the other half. Blocking an IP or a hash costs the adversary almost nothing. Detecting their behaviour — how their C2 checks in, how their tunnel abuses DNS — is near the top of the pyramid and is expensive for them to change. Beacon timing and DNS subdomain patterns are behavioural. Rotating infrastructure does not make them go away.

What you need to see: network data sources

The PuppyGraph guide to network threat hunting makes a point that is easy to forget: network evidence is only useful when you can connect it to who and what. You need the traffic, plus the context to tie an IP to a host, a user and a process. Here are the main sources and their trade-offs:

• Full packet capture (PCAP) — tcpdump, Arkime, or a commercial recorder. Maximum fidelity: payloads, file carving, protocol reconstruction. Heavy on storage, so retention is often measured in days. Best for deep-dive validation, not for scanning months of history.
• Network security monitoring logs (Zeek) — Zeek turns packets into rich, structured logs: conn.log, dns.log, http.log, ssl.log, x509.log, files.log and more. Roughly 1–2% of the size of the PCAP, so you can keep months. This is the workhorse for most of what follows in this series.
• NetFlow / IPFIX / VPC flow logs — cheap, everywhere, built into routers, switches and cloud platforms. No application layer, and often sampled, so beacon and DNS analysis is limited. Still excellent for volume, rarity and east-west network hunting.
• Firewall and proxy logs — widely available; useful for egress analysis and URL/user-agent hunting. Quality varies by vendor; some firewalls only log session end.
• DNS logs — resolver query logs or passive DNS. Critical for DNS-based C2, DGA and newly-registered-domain hunts.
• Identity and asset context — DHCP leases, VPN sessions, authentication logs, CMDB/asset inventory. Without these, “10.1.4.37” is just a number.
• Endpoint telemetry — EDR network events or Sysmon Event ID 3 tie a connection to the process that made it. This is your pivot target after the network hunt finds something.

Sensor placement matters more than tools

The best analytics fail if the sensor is in the wrong place. A few rules from the field:

• Capture inside the egress firewall, before NAT. Otherwise every outbound connection appears to come from the firewall’s public IP and you lose the internal host.
• Capture before the internal DNS resolver forwards queries, or log on the resolver itself. Otherwise all DNS appears to come from the resolver.
• Do not ignore east-west traffic. Perimeter sensors see C2 and exfiltration, but not lateral movement between internal segments. Core switch SPANs or virtual TAPs in hypervisors and cloud VPCs close that gap.
• Measure your visibility before you hunt. Confirm that the sensor actually sees every egress path, including guest Wi-Fi, OT links, cloud egress and direct internet breakouts at branch offices.

Know the limits

Network hunting is powerful, but it is not magic. Be honest about the blind spots:

• Encryption. TLS 1.3, encrypted ClientHello (ECH) and DNS over HTTPS (DoH) hide content and, increasingly, destination names. You will rely more on metadata: timing, size, certificates, and fingerprints such as JA4+.
• Volume. Unbounded searches across terabytes overwhelm analysts. Scope every hunt.
• Identity ambiguity. DHCP churn, NAT, shared proxies and cloud IPs that change owners make attribution harder.
• Noisy baselines. Software updaters, telemetry agents, chat clients and monitoring tools all beacon. Expect false positives and build an allow-list as you go.

What a successful hunt produces

A hunt that “found nothing” is still a success if it produces outputs. At the end of every hunt you should have:

1. A disposition for every host in scope — benign, suspicious (needs more data), or compromised (escalated to IR).
2. An updated allow-list of known-good long connections and beacons, with the business reason recorded.
3. New or improved detections for anything that can be automated.
4. A list of visibility gaps discovered along the way (missing logs, unmonitored network segments).
5. Documentation of scope, data, queries and findings, so the hunt can be repeated.

Measure the hunting program by those outputs — time-to-disposition, detections created, gaps closed — not by the number of hunts run.

Coming up in Part 2

In Part 2 we get our hands dirty. We will hunt C2 the way an analyst actually does it: finding long connections and beacons in Zeek conn.log, catching DNS tunnels by counting unique subdomains, spotting outliers in bytes, user-agents and certificates, and scoring each finding so you can make a defensible call.

References and further reading

• Active Countermeasures — How to Threat Hunt Your Network
• Active Countermeasures — Network Threat Hunting training slides (PDF)
• PuppyGraph — Network Threat Hunting
• NetWitness — Types of Network Threat Hunting
• Splunk SURGe — The PEAK Threat Hunting Framework
• Mandiant — M-Trends 2026
• SANS — FOR572: Advanced Network Forensics: Threat Hunting, Analysis, and Incident Response

Long Live Cyber Defenders!!!

Hunting on the Wire : A Practical Guide to Network Threat Hunting — Part 1 was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why File Transfer?

Between 2020 and 2025, the same extortion group — Cl0p — executed an identical playbook against four consecutive managed file transfer products: Accellion FTA, Fortra GoAnywhere, Progress MOVEit, and the Cleo product suite. The 2023 MOVEit campaign alone impacted over 2,700 organizations and more than 90 million individuals. Exploitation of the Cleo flaw was still active as late as February 2025.

Note what didn’t happen there. No AES was cracked. No elliptic curve collapsed. What was exploited was SQL injection, authentication bypass, and remote code execution in software sitting at the perimeter — software that held the organizations’ most sensitive data.

This is the first lesson, and it’s worth stating up front so it doesn’t get lost: the strength of your cipher is the easiest part of the problem. The hard parts are protocol design, key handling, and verification discipline.

But there’s another side of the spectrum that’s rarely discussed. At the opposite end from enterprise-grade MFT servers lies a far more resource-constrained class of data transfer: small devices sending data over narrow channels with no supporting infrastructure at all. No TLS, no certificate authority, no internet connection. There, data transfer often happens with no encryption whatsoever — not because nobody cares, but because of the prevailing assumption that cryptography is too expensive for devices that small.

Four Questions, One Primitive.

Every file transfer protocol — from SCP to S3 multipart uploads — must address four aspects:

• Confidentiality — can an intermediary read the contents? This is the one everyone remembers.
• Integrity — large files are split into chunks. If an attacker swaps chunk five with one from another file, reorders chunks, or truncates the transfer mid-way while claiming it’s complete, the received file is corrupted without anyone noticing. This one gets overlooked.
• Authenticity— a valid chunk that’s recorded and replayed remains cryptographically valid unless a sequence number is bound into the authentication.
• Forward secrecy — your keys will eventually leak. The question is how much becomes exposed when they do. With one static key for the system’s whole lifetime, the answer is: everything, including traffic an attacker recorded three years ago.

AEAD addresses the first three all at once:

A stands for associated data: it is unencrypted but is also authenticated. This is where the chunk number, the last chunk marker, and the sender’s ID are placed — plaintext, but cannot be modified without causing the tag verification to fail.

Proper AEAD returns a failure, not garbage plaintext, when the tag doesn’t match. This isn’t an academic detail: when the Ascon-AEAD128 implementation was submitted to OpenSSL, one review finding was that decryption released the plaintext before tag verification completed. Bugs of that class still happen in the most heavily audited project in the world.

Ascon & X25519

On August 13, 2025, NIST finalized SP 800–232, a lightweight cryptography standard built on the Ascon family — including Ascon-AEAD128, Ascon-Hash256, and its XOF variant. Ascon had previously won as the primary lightweight-encryption choice in the CAESAR competition.

What makes it interesting: a sponge/duplex construction with a 320-bit state, no key schedule to expand and store, and no S-box lookup tables in memory — so no cache-timing leakage from lookup tables either. A single permutation handles encryption, hashing, and key derivation, so an entire protocol can be built from one primitive: Ascon-XOF works as a KDF out of the box, no need to import HKDF-SHA256 separately. Library support is moving: wolfSSL added it in April 2025, with OpenSSL support following.

⚠️ Ascon v1.2 ≠ SP 800–232. The final standard changes the initial value, adjusts to little-endian formatting for microcontroller performance, and adds options such as nonce masking. The test vectors differ. I implemented v1.2; if you need interoperability today, target SP 800–232.

For key exchange, X25519 beats NIST P-256 on all three metrics that matter on constrained devices: 523 ms, 47 KB, 55 mW, versus P-256’s 1,582 ms, 64 KB, 103 mW (Tanksale 2024, 48 MHz MCU).

But more important than the choice of curve is its ephemerality. A key pair generated fresh for each session and discarded afterward means a leak today doesn’t unlock yesterday’s recordings. The age file encryption format does exactly this: for every single file, a new ephemeral X25519 key is generated and run through ECDH against the recipient's public key to wrap the file key.

One caveat: raw ECDH is vulnerable to man-in-the-middle (MitM) attacks. Without authentication, an attacker can swap both public keys, establish two separate shared secrets, and read everything in between. ECDH provides confidentiality, not identity.

The Design

Two layers of keys. A static pre-shared key (PSK), provisioned onto both devices in advance, is used only to authenticate the handshake — never to encrypt data. A session key derived via ECDH handles the actual data chunks. This separation is what makes the scheme work: the PSK isn’t repeatedly exposed with every chunk, and a leaked session key doesn’t compromise any other session, past or future.

Four decisions worth highlighting:

• The ephemeral public key is sent encrypted and authenticated under the PSK. This is what closes the MitM hole — an attacker without the PSK can’t inject their own public key, since the handshake packet would fail tag verification.
• The counter doubles as the nonce and lives in the associated data. The receiver reconstructs the nonce without it ever being sent in full, and because the counter is authenticated, it can’t be forged for a replay.
• Verification order: counter first, tag second. Stale packets are rejected with zero cryptographic operations.
• Ephemeral private keys and shared secrets are explicitly zeroed immediately after the session key is derived. Without this step, forward secrecy is a claim on paper, not in practice.

Data frame: ID 2 · counter 8 · timestamp 4 · ciphertext 8 · tag 16 = 38 bytes. Handshake frame: ID 2 · nonce 8 · pubkey 32 · tag 16 = 58 bytes.

For an 8-byte chunk, the cryptographic overhead is nearly four times the size of the data — the worst-case scenario that could be designed. For a 64 KiB chunk, that same overhead becomes 0.05%.

The Result

Before any measurement was taken: X25519 was checked against the RFC 7748 test vectors; Ascon-128 against the reference implementation, including 400 random combinations of AD length and plaintext length; plus negative tests with a corrupted tag, corrupted ciphertext, and corrupted AD. A crypto implementation that runs without errors doesn’t mean it’s correct — broken encryption still produces convincing-looking random bytes.

Hardware: ESP32 at 240 MHz, LoRa SX1276, SF7/BW125. The four schemes were run round-robin so environmental drift would be spread evenly across all of them.

ECDH is practically free. The difference between the second and third rows is 230 cycles and 0.14 KB of RAM, in exchange for forward secrecy and MitM resistance. The handshake itself takes 342.6 ms, but since it happens only once every 100 chunks, it amortizes to 3.43 ms per chunk — 0.17% of the send interval.

Ascon needs about 35% fewer cycles than AES-GCM for decryption (25,273 vs. 38,631 — equivalently, AES-GCM takes roughly 53% more cycles than Ascon), even though the ESP32 has an AES hardware accelerator. The reason: the accelerator handles the AES block cipher, but GHASH still runs in software, and on an 8-byte payload, the GHASH overhead outweighs whatever the accelerator saves.

This finding reverses, though, on 64 KiB chunks on a CPU with AES-NI. AES-GCM’s per-byte cost drops sharply, while Ascon stays bound to a fixed number of permutation rounds per 8-byte block. Ascon’s advantage is an advantage on constrained devices with short messages, not a universal one.

This is why age chose ChaCha20-Poly1305 over AES-GCM: not because AES is weak, but because AES is hard to make fast and safe without hardware support. Pick a primitive based on the platform and message size, not on whichever was standardized most recently.

Five attack scenarios were run against separate attacker nodes. Ascon+ECDH withstood eavesdropping, injection, replay, and MitM-on-handshake. The static-PSK scheme withstood eavesdropping, injection, and replay too — MitM-on-handshake doesn’t apply to it, since it has no ECDH handshake to attack — but it failed the one test that actually matters for key compromise: forward secrecy. I recorded three key-rotation periods, leaked the second period’s session key, and tried decrypting the other two. With static PSK, all three periods were exposed. With Ascon+ECDH, periods 1 and 3 both failed tag verification — rejecting the packets outright rather than handing the application incorrect plaintext.

Now for the part that usually doesn’t make it into articles like this. I flooded the channel with fake chunks:

The cryptographic side works perfectly. Zero forged packets got through, and verification latency stayed flat regardless of attack rate. And availability still dropped to 20%.

Confidentiality, integrity, and authenticity are three properties. Availability is a fourth, and cryptography doesn’t provide it — that’s handled elsewhere, with rate limiting, channel scheduling, and per-identity quotas. Tie that back to MOVEit: those organizations used TLS; their data was encrypted in transit. What failed was authorization and input validation sitting on top of the cryptography.

Limitations

Test conditions were controlled: two nodes, 1-meter distance, line-of-sight. The payload was a fixed 8 bytes, so Ascon’s partial-block code path never executes during application use (though it was verified separately via multi-block test vectors). Performance figures don’t generalize across platforms — 24,624 cycles on the Xtensa LX6 doesn’t map directly onto Cortex-M or x86–64. This is an Ascon v1.2 implementation, not SP 800–232. Side-channel analysis is out of scope; any crypto implementation needs an independent audit before production use. And as shown above, availability under flooding was not addressed by the cryptographic design at all.

Conclusion

Lightweight cryptography is mature enough for serious use now. Ascon has a NIST standard behind it, and on small devices with short messages it beats AES-GCM by a wide margin. Ephemeral key exchange is nearly free — systems still running on a static key for the device’s entire lifetime are carrying technical debt that will eventually come due. Framing details decide everything: chunk numbers and end-markers bound into the associated data are the difference between a secure transfer and an encrypted one that can still be truncated or scrambled. And correct cryptography is a prerequisite, not a guarantee — all four MFT products breached over those four years used sound encryption. The failure was never in the cipher.

Reference: Implementation of Lightweight Cryptography Ascon and Elliptic Curve Diffie-Hellman (ECDH) on LoRa Data Transmission Mechanism

IoT Security: Securing File Transfers Over Untrusted Channels was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kunai (https://why.kunai.rocks/) is built specifically for Linux threat hunting, using eBPF and Rust to collect and analyze system activity. It can also monitor activity inside containers and apply the same threat-hunting rules across them.

In this article, I’ll walk through my setup of Kunai v0.7.0-rc.1 on Ubuntu Server 24.04, from enabling BPF LSM and installing Kunai to configuring it and creating a simple detection rule.

1. Enabling BPF LSM

First, check the currently enabled Linux Security Modules:

cat /sys/kernel/security/lsm

Kunai’s hardening mode requires bpf to be present in this list.

We need to add the LSM configuration to GRUB, Instead of editing /etc/default/grub manually using nano, the following command updates GRUB_CMDLINE_LINUX_DEFAULT using sed.

sudo sed -i \
's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="lsm=lockdown,capability,landlock,yama,apparmor,bpf"/' \
/etc/default/grub

Verify the configuration:

sudo cat /etc/default/grub

Then update GRUB and reboot:

sudo update-grub
sudo reboot

After reboot, verify it:

cat /sys/kernel/security/lsm

You should see:

lockdown,capability,landlock,yama,apparmor,bpf

This step is important because Kunai’s --harden installation checks for BPF LSM before continuing.

2. Download and Configure Kunai

Create a working directory:

mkdir Kunai
cd Kunai

Download the Kunai release and make the binary executable.

wget https://github.com/kunai-project/kunai/releases/download/v0.7.0-rc.1/kunai-amd64

chmod +x ./kunai-amd64

Then generate the default configuration:

./kunai-amd64 config --dump > kunai.conf

Move the configuration into the Kunai configuration directory:

sudo mkdir -p /etc/kunai/
sudo mv ./kunai.conf /etc/kunai/kunai.conf

3. Install Kunai as a Systemd Service

Now install Kunai with hardening and systemd support:

sudo ./kunai-amd64 install \
--config /etc/kunai/kunai.conf \
--systemd \
--enable-unit \
--harden

The important options are:

• --config — specifies the Kunai configuration.
• --systemd — creates a systemd service.
• --enable-unit — enables the service.
• --harden — enables the hardened installation checks.

After installation, check the generated events:

sudo systemctl start kunai
sudo tail -f /var/log/kunai/events.log

At this point, Kunai should be collecting system activity.

For better tuning, In my setup I disabled send_data to reduce unnecessary event generation.

sudo systemctl stop kunai
sudo nano /etc/kunai/kunai.conf
sudo systemctl restart kunai

4. Creating a Detection Rule

Kunai can monitor all system events, providing deep visibility but generating a large volume of data. Detection rules allow users to focus only on specific events and behaviors that matter for security monitoring.

This is where Kunai Detection rules become useful.

Create a rules directory:

sudo mkdir -p /etc/kunai/rules/

Create a rule file:

sudo nano /etc/kunai/rules/example_rule.kunai

For example, we can detect a binary pretending to be a Linux kernel worker thread:

name: mimic.kthread
type: detection
meta:
  tags: [ 'os:linux' ]
  attack: [ T1036 ]
  authors: [ qjerome ]
  comments:
    - tries to catch binaries masquerading kernel threads
match-on:
  events:
    kunai: [execve, execve_script]
matches:
  $task_is_kthread: .info.task.flags &= '0x200000'
  $kthread_names: .info.task.name ~= '^(kworker)'
condition: not $task_is_kthread and $kthread_names
severity: 10

The idea is simple:

The rule evaluates fields from the Kunai event itself. .info.task.flags is used to determine whether the task has the kernel-thread flag, while .info.task.name is checked for the kworker naming pattern. The final condition triggers when those two properties don't match.

The rule also maps the behavior to MITRE ATT&CK T1036, which is associated with masquerading.

Then update the main Kunai configuration so it knows where the detection rules are stored.

sudo nano /etc/kunai/kunai.conf

Then restart Kunai:

sudo systemctl restart kunai

This command below creates a benign test case where a normal user-space binary is given a name resembling a Linux kernel worker thread. When the binary executes, Kunai should see the execve event and evaluate it against the detection rule

cp /usr/bin/ls /tmp/kworker
/tmp/kworker

Now search the Kunai log for the detection:

sudo grep -RniE 'mimic.kthread' \
/var/log/kunai/events.log 2>/dev/null | tail -50

If the rule is working, Kunai should report the detection.

Beyond Detection Rules

Detection rules are only one part of Kunai’s capabilities. Kunai also provides mechanisms for filtering events, scanning for indicators of compromise, and taking automated response actions.

For example, Kunai can be integrated with YARA-based scanning to inspect files for known patterns, while automated actions can be used to respond to detected activity. These capabilities allow Kunai to move beyond passive monitoring toward a more active detection and response workflow.

In this article, however, we focused on the fundamentals: collecting Linux telemetry, understanding Kunai events, creating a detection rule, and validating that detection with a controlled test.

Conclusion

Setting up Kunai is more than simply installing a security agent. The combination of eBPF visibility and custom detection rules makes Kunai useful for investigating Linux activity and building practical threat detections.

The important lesson is to start with visibility, understand the events being generated, and then create focused rules for the behavior you actually want to detect.

Kunai Engineering #1 — Building Linux Threat Detection with Kunai on Ubuntu Server was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tracing a Web RCE All the Way to Lateral Movement, Before Any Alert Fired

This one starts with an assumption, a company has already been compromised. A threat actor got in through a vulnerability, quietly established a foothold, and SentinelOne hasn’t raised a single alert about it yet. My job here isn’t to wait for a detection to fire, it’s to go looking for the intrusion myself and see how far it went.

So that’s what I’ll walk through. I’ll trace the whole thing from the very first suspicious process, follow the chain step by step, and rebuild the full attack path from raw telemetry alone. At the end I’ll share the attack topology, the mitigation steps I took, and the indicators of compromise so you can reuse them.

Note: Make sure Singularity Operations Center is turned on (Preferences > Features) so you have access to all the modules used here, from Event Search and Storyline to Graph, Forensic Collection, and RemoteOps.

There are two endpoints in scope, both already running the SentinelOne sensor:

• Linux (Web Server): runs the company’s public-facing web app.
• Windows (Workstation): a regular employee’s machine on the same network.

Worth stressing up front: I’m going in assuming a breach already happened, but with zero alerts on the board. Everything below comes purely from hunting through raw telemetry, not from anything the detection engine handed me.

Finding the Entry Point (T1190)

Since one of the endpoints is an internet-facing server, my first hypothesis is Exploitation of a Public-Facing Application (T1190). The reasoning is simple, if someone broke in through the web app, the evidence should show up as the web service behaving in a way it normally never would. I ran the hunt in MITRE ATT&CK order, earliest stage first, so the story reads cleanly and I don’t skip a link in the chain.

First I head into Event Search and set up the filter:

The point of this T1190 query isn’t to look for malware. It’s to look for a relationship, a web app process (python3, python, node) becoming the parent of a shell process (sh, bash, dash). A healthy web server serves requests and returns pages, it has no reason to spawn a shell. The moment that pattern shows up, it’s a strong candidate for command injection or RCE.

| filter( event.type == "Process Creation"
  AND src.process.name contains:anycase( "python3","python","node" )
  AND tgt.process.name in:matchcase( "sh","bash","dash" ) )
| columns event.time, event.id, event.type, site.id, site.name,
  agent.uuid, timestamp, src.process.storyline.id, src.process.user,
  src.process.cmdline, src.process.image.path, tgt.process.storyline.id,
  tgt.process.cmdline, tgt.process.image.path
| sort - timestamp
| limit 1000

The query returns results, and once I scroll across to the command line, one row jumps straight out.

If you look closely at that command line. It opens with host -t MX example.com, a legitimate-looking DNS lookup, then chains two completely unrelated commands after a semicolon, chmod +x on a file in the uploads directory, then running that file in the background. That’s the classic fingerprint of command injection. The original feature only meant to verify a domain, but because the input wasn’t validated, the attacker slipped extra commands in through the shell separator. One shell running several unrelated commands is a lot more telling than just seeing python spawn sh, and that’s what pushes this from odd to genuinely bad.

I click the Source Process Unique ID on that row to jump into a new tab opens showing the process lineage, python3 app.py as the source, spawning /bin/sh as the target, which then runs the injected commands. Purple AI also flags that both processes are unsigned and are executing a file out of the uploads directory, two more reasons to be suspicious.

From this detail page I grab the single most useful artifact for the rest of the investigation, the Source Process (Parent) StoryLine ID. In SentinelOne, the Storyline ID is the thread that ties every child process from one root activity into a single story. Once I have that ID, I can rebuild the whole chain without guessing.

Pivoting on the Storyline ID

Now I filter again, this time not by process pattern but by the Storyline ID that I just grabbed.

| filter(src.process.storyline.id "d1c3a4d8-0c76-55cd-2b32-3e4dee33f4fa" 
OR tgt.process.storyline.id "d1c3a4d8-0c76-55cd-2b32-3e4dee33f4fa" )
| columns event.time, event.id, event.type, site.id, site.name, agent.uuid,
timestamp, src.process.storyline.id, src.process.user, src.process.uid,
src.process.cmdline, src.process.image.path, tgt.process.storyline.id,
tgt.process.user, tgt.process.uid, tgt.process.cmdline, 
tgt.process.image.path
| sort timestamp
| limit 1000

A process named update_agent shows up, running out of /home/rocky/apps/uploads/update_agent. The name is picked on purpose to sound like a legitimate update process, a common masquerading trick so it doesn’t draw attention when an analyst skims a process list. But the context gives it away, a binary running from a web app’s upload directory is not a normal update. This is our implant candidate, and it becomes the center of the next few steps.

Confirming the Download Path (T1105)

To find out how update_agent landed on the server, I check for a shell process downloading over curl or wget, mapping to Ingress Tool Transfer (T1105).

| filter( event.type == "Process Creation"
  AND tgt.process.name in:anycase( "curl","wget" )
  AND tgt.process.cmdline contains:anycase( "http" ) )
| columns event.time, event.id, event.type, site.id, site.name,
  agent.uuid, timestamp, src.process.storyline.id, src.process.user,
  src.process.cmdline, src.process.image.path, tgt.process.storyline.id,
  tgt.process.user, tgt.process.uid, tgt.process.cmdline, tgt.process.image.path
| sort - timestamp | limit 1000

Confirmed. There’s a curl process pulling the file from an external server, and that’s where I get the suspicious IP. I click the curl process to open its Storyline, and there’s a run of bash child processes worth digging into.

This Storyline shows a textbook post-exploitation pattern, after getting a foothold, the attacker runs curl to pull the payload, mixes in id to check what privileges they have, and grep to sift through information. That’s not an app doing its job automatically, that’s a human getting oriented inside a system they just took over. The download destination, 192.168.122.20, becomes the key indicator I chase next.

Validating Command and Control (C2) Communication

An implant is useless to an attacker if they can’t control it. Since I already have that suspicious IP, the logical next move is to prove whether the compromised server actually reaches back out to it, the Command & Control channel.

| filter(event.type "IP Connect" AND src.ip.address "192.168.122.11" AND 
dst.ip.address "192.168.122.20")
| columns event.time, event.id, event.type, site.id, site.name, agent.uuid,
timestamp, src.process.storyline.id, src.process.user, src.process.uid,
src.process.cmdline, src.process.image.path, src.ip.address, src.port.number,
dst.ip.address,
dst.port.number, event.network.direction, event.network.protocolName,
event.network.connectionStatus
| sort timestamp
| limit 1000

Three columns settle it here, the OUTGOING direction, destination port 8080 (http-alt), and the SUCCESS status. Together they confirm the implant isn’t just sitting there, it established two-way communication with the attacker’s infrastructure. Port 8080 over something that looks like HTTP is a deliberate choice, C2 traffic is often dressed up as web traffic so it blends into normal outbound flow and slips past perimeter inspection. From a defender’s seat, a single SUCCESS connection to an unknown IP like this is enough to move the case from suspicion to confirmed.

Then I click the update_agent process to open its Storyline. Scrolling down, another important find turns up, a crontab process, which tells me the attacker planted a way to keep their payload alive, in other words, persistence.

The crontab confirms persistence (T1053.003). The payload is scheduled to re-run on its own, so rebooting the server or killing the process alone won’t remove it. Eradication later needs to clear this cron entry too, not just the implant file.

Lateral Traversal to Secondary Victim (T1021.004)

After taking the web server, the next step is checking for lateral movement. I move to T1021.004 (Remote Services: SSH) and look for an outbound SSH connection from the compromised server to another endpoint.

| filter( event.type == "IP Connect" AND dst.port.number == 22 )
| columns event.time, event.id, site.id, site.name, agent.uuid,
  timestamp, src.process.storyline.id, src.process.user,
  src.process.cmdline, src.process.image.path, src.ip.address,
  src.port.number, dst.ip.address, dst.port.number,
  event.network.direction, event.network.protocolName,
  event.network.connectionStatus
| sort - timestamp | limit 1000

Scrolling across makes it obvious, there’s an ssh command line heading to a different endpoint, from the web server (192.168.122.11) to the employee workstation (192.168.122.12), using the accounts budi, Budi, and luminousvl.

To confirm the jump actually landed on the other side, I check the Storyline on the destination endpoint.

Trying three different account names (budi, Budi, luminousvl) shows the attacker testing the credentials they found, and sshd-session.exe appearing on the Windows side proves at least one attempt worked. This is the moment the attack crosses platforms, from Linux to Windows, which makes it a lot more dangerous, from a server you might have assumed was isolated, the attacker now has a foothold on an employee’s workstation, much closer to user data and identities.

Key hunting point: Up to here, SentinelOne still hasn’t raised a single alert. Around 23:36 there’s nothing yet that trips an automatic detection. The entire chain so far, initial access, C2, persistence, and lateral movement, was uncovered purely through proactive hunting. That’s exactly how attacker dwell time gets long when you only rely on alerts.

Impact & the First Alert (T1490)

With lateral movement onto the workstation confirmed, I filter Process Creation for trusted Windows executables used to modify or delete system state, a common pre-ransomware pattern mapping to Inhibit System Recovery (T1490).

| filter(event.type == "Process Creation" AND 
(tgt.process.name in: anycase("vssadmin.exe", "wbadmin.exe",
"cipher.exe", "wmic.exe", "bcdedit.exe", "diskshadow.exe",
"fsutil.exe", "wevtutil.exe", "schtasks.exe", "reg.exe")
OR tgt.process.cmdline
contains: anycase("delete")))
| columns event.time, event.id, site.id, site.name, agent.uuid,
src.process.storyline.id, src.process.user, src.process.uid,
src.process.cmdline, src.process.image.path, tgt.process.storyline.id,
tgt.process.user, tgt.process.uid, tgt.process.cmdline,
tgt.process.image.path
| sort event.time
| limit 1000

I click through to the process Storyline, and you can see how busy the attacker got on this box, a long run of processes in a short window.

One cmd.exe branches into vssadmin.exe (shadow copy deletion, T1490) alongside whoami, net, nltest, and netstat, discovery commands mapping privileges, accounts, domain trust, and network state. This discovery-then-destroy sequence is a standard pre-ransomware pattern. Notably, it’s this destructive activity, not the initial access, that finally crosses SentinelOne’s detection threshold.

Sure enough, this is where SentinelOne finally fires an alert, two Critical alerts at the same time, one for Lateral Movement and one for PowerShell deleting Volume Shadow Copies.

The PowerShell alert shows Anti Exploitation / Fileless as the detection engine, executed under the luminousvl account, with the analysis explicitly flagging an attempt to delete all VSS shadow copies (T1490). Purple AI adds a process-correlation overview that speeds up triage.

The alert only fires in the final minutes, after initial access, C2, persistence, and lateral movement had already gone undetected. Teams relying solely on alerts would engage this incident far too late. Proactive hunting is what compresses dwell time here, and could have broken this chain well before the destructive stage.

Root cause confirmed that a command injection vulnerability in the internet-facing web app allowed arbitrary command execution from unvalidated input, leading to RCE and the implant being planted.

Mitigation & Response

Response depends on business context. Cutting network access on a 24/7 production system isn’t a decision made lightly, but since the victim here is a landing page, there’s more room to act. Either way, one rule is non-negotiable that secure the evidence first, then contain. Rushing to remediate destroys the artifacts needed to understand the attack.

1. Forensic Collection: Secure the Evidence First

Before touching anything, I pull the full collection and evidence off the server, especially if it’s a system that deals with customers daily. SentinelOne’s Forensic Collection can be run straight from the console.

Agent Management > Click Endpoint > Actions > Endpoints >
Forensics Collection > Common Artifacts > Run Collection > Agent Tasks
Page > Run > Two-Factor Authentication > Tasks & Updates > (select the
collection) > Download File

Click Actions to retrieve Forensics Collection

Then click Run Collection to get Artifact.

Once it’s extracted, I open the collection in a text editor. Two artifacts give immediate context. First, in NetworkPorts, the ss.txt file shows the server’s port 22 (SSH) is open to anyone (0.0.0.0:22).

SSH bound to 0.0.0.0:22 is what made the lateral movement trivial. This isn’t a code bug, it’s a segmentation failure that an administrative service with no network ACL turns a single-host compromise into a network-wide one.

The .bash_history reads like a linear attack log: find -iname “.txt” to hunt for secrets, read the operator notes, set up cron persistence, then pivot via SSH. The history -d 72 at the end is an anti-forensics attempt, deleting one line to cover tracks. It’s futile here, since SentinelOne’s telemetry sits outside the attacker’s reach, which is exactly the case for centralized logging over host-local logs that can be tampered with.

2. Deeper Investigation via Remote Shell

Caution: Remote Shell should only be used after getting the server owner’s permission for forensic work. Access: Actions > Endpoints > RemoteShell.

ops_notes.txt is the pivot point between initial access and lateral movement. Plaintext workstation credentials on the web server gave the attacker a direct path to Windows, no exploitation required. Credential storage like this on an internet-facing host remains one of the costliest and most common mistakes in real breaches

Next, the dropped file, an ELF executable created at 22:22:23 on 3 Sept 2026 consistent with the attack timeline.

3. Network Containment

Cutting network access on a 24/7 production system isn’t a unilateral call, it carries real business impact. For a landing page server like this one, though, temporary containment while the investigation continues is a reasonable move.

Click Actions, then choose Disconnect from Network.

I do the same for the Windows endpoint hit by lateral movement, so the attacker’s movement is cut off on both sides. Once it’s done, both endpoints show as Isolated.

Wrapping Up

From an investigation that started entirely from hunting, not from an alert, I was able to rebuild the full attack chain:

python3 (web app) -> shell command-> curl -> C2 implant -> C2 connection -> crontab -> reading the credential file -> outbound SSH (lateral movement) -> deleting shadow copies.

Alerts are the last safety net, not the first. A capable attacker stays under the detection threshold, and the stages that matter most, initial access and lateral movement, are usually the quietest. Proactively querying the data is what shrinks dwell time and turns you from a reactive victim into a hunter.

Attack Chain Summary

Attack Scenario Topology

Overall, the flow from the attacker to the web server and then to the employee workstation looks like this:

Indicator of Compromise (IoC)

A summary of the indicators from this investigation, ready to reuse for sweeping across other environments:

Closing

This investigation shows that with Event Search, Storyline, and Forensic Collection, a single analyst can trace a cross-platform compromise from scratch to the end, well before the detection engine speaks up. Compromise assessment isn’t really about fancy tooling, it’s about being persistent with your questions and patient enough to pull the thread one piece at a time.

In the next post, I’ll cover RemoteOps and how to build a fuller, more mature Forensic Collection so the DFIR process in SentinelOne can go even deeper.

See you in the next one guys. Reyhan Al Farel.

Threat Hunting & Compromise Assessment in SentinelOne Singularity Platform was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

Onboarding a new customer into SentinelOne sounds simple enough. Create the tenant, set up the accounts, push the agent, move on. But there’s a question that doesn’t answer: if that endpoint actually got hit, would any of it work? You don’t really know until you test it.

So that’s what we did here. This walkthrough covers the full onboarding tenant creation, user provisioning, agent deployment and then goes one step further: we ran an test attack against the endpoint once it was live.

Not a real one. We used Atomic Red Team to simulate, Which gives you the same behavior a ransomware incident would like files getting mass-created, a ransom note dropped somewhere, PowerShell doing sketchy things. just without anything actually getting encrypted or spreading past the box.

If the platform picks it up end to end, that tells us the tenant’s configured right, the agent’s actually reporting, and the detection and response side of things works the way it’s supposed to.

Tenant (Account & Site) Creation in SentinelOne

1. Log in to the SentinelOne Management Console using a Global Admin/Admin account with permission to create a new tenant.
2. Open the scope panel using the (>) control next to Global.

3. Click the (+) icon to add a new site.

4. Since this tenant is being created for Incident Response purposes, select “IR” as the site type.

5. Enter the required site information, then click Next.

6. For policy settings, inherit the Global Default Policy, or you can modify on you own policy.

7. after that you can click on Next and it will bring you to the summary and click on Create Site.

8. Generate and copy the Site Token. This token is required to install the agent on the target endpoint.

Customer User Account Creation

1. Navigate to Policies and Settings > User Management > Console Users.

2. Click New User

3. Enter the user’s email address and assign the user to the site/tenant created before.

Agent Onboarding on the Suspected Compromised Endpoint

1. Go to Agent Management, confirm you are in the correct site, then click Packages.

2. Search for a package compatible with the target endpoint. In this case, the affected device runs Windows 10, so an .msi/.exe package was selected via Package > Action > Download Package.

3. Run the downloaded installer. It will prompt for the Site Token generated, paste it into the installer and click Next.

4. The installer proceeds with agent installation.

5. Once installation completes, confirm the SentinelOne agent process is running in the background.

6. Return to Agent Management > Endpoints. The newly onboarded desktop now appears in the endpoint list. Click the endpoint name to review its status.

Lab Testing Incident Analysis & Investigation

Two complementary test cases were used to exercise different SentinelOne detection engines: a reputation-based detection using a known malicious test file, and a behavioral simulation of ransomware activity using the Atomic Red Team framework.

1. Reputation-Based Detection

• Atomic Red Team was pre-installed on the endpoint before the SentinelOne agent was deployed. Shortly after the agent came online, multiple entries appeared under Unresolved Alerts.

• Atomic Red Team was pre-installed on the endpoint before the SentinelOne agent was deployed. Shortly after the agent came online, multiple entries appeared under Unresolved Alerts.

• The file located at
  \Device\HarddiskVolume3\AtomicRedTeam\atomics\T1221\src\Calculator.docx
  was identified as a Trojan by SentinelOne’s Cloud Intelligence Reputation engine as medium severity.
• Purple AI generated an automatic natural-language explanation of the detection, and the Mitigation Actions panel confirmed the process was Killed and the file Quarantined successfully.

2. Behavioral Simulation, Atomic Red Team T1486, Atomic Test #10 (Akira Ransomware)

• To evaluate SentinelOne’s behavioral detection capability against ransomware-style activity, Atomic Test #10 (“Akira Ransomware drop files with .akira extension and ransomnote”) was executed from an elevated PowerShell session on the endpoint.

• Following execution, the endpoint’s local disk showed 100 newly created files with the .akira extension, along with a ransom note (akira_readme.txt) dropped on the Desktop.

• Event Search (Deep Visibility) was used to query for PowerShell-related activity using the PowerQuery below:

| filter( src.process.displayName == "Windows PowerShell" )
| columns event.time, event.id, event.type, site.id, site.name, agent.uuid,
  timestamp, src.process.storyline.id, src.process.user, src.process.uid,
  src.process.cmdline, src.process.image.path, tgt.file.path, tgt.file.size,
  tgt.file.type, tgt.file.isSigned
| sort - timestamp
| limit 1000

• The query returned 162 matching events. One entry of particular interest showed the command script
  echo “” >> $env:Userprofile\Desktop\akira_readme.txt,
  executed by a signed powershell.exe process spawned from the legitimate RuntimeBroker.exe.
• Purple AI flagged this activity as significant, reporting 105 file creations, 147 file modifications, and classifying the underlying indicators as 339 general, 76 ransomware-related, 33 evasion, 21 injection, 7 reconnaissance, and 1 exploitation indicator, with no associated network or DNS activity.

Remote Remediation via Remote Shell

• Since this activity was a lab test rather than a genuine compromise, cleanup was performed directly from the console using SentinelOne’s Remote Shell capability (Endpoint > Actions > Response > Remote Shell), avoiding the need for local or RDP access to the endpoint.

• Clicking Start Session prompted for a 2FA acting as a step-up verification before remote command execution is permitted.

• Once connected, the following cleanup commands were attempted

del C:\Users\PC-Labs\Desktop\akira_readme.txt
del C:\test.*.akira

• Cleanup was verified with the commands below; an empty result and a False return respectively confirmed no artifacts remained on disk.

Get-ChildItem C:\test.*.akira -ErrorAction SilentlyContinue
Test-Path "C:\Users\<username>\Desktop\akira_readme.txt"

Conclusion

This lab exercise successfully validated the end-to-end SentinelOne Singularity onboarding workflow, from tenant creation through agent deployment and initial incident investigation. The platform demonstrated strong, fully automated performance against a known-bad static threat, with detection, killing, and quarantine completed.

The behavioral ransomware simulation (Atomic Red Team T1486, Test #10) executed successfully and produced substantial forensic telemetry, which Purple AI was able to interpret and summarize meaningfully for an analyst a clear strength for investigation speed before they’re going into more deeper analysis.

SentinelOne Singularity Onboarding: Tenant Creation, Agent Deployment and a Lab Test was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

SentinelOne’s Fetch Logs action pulls exactly that: agent activity logs, plus a slice of the endpoint’s own Event Viewer logs, bundled into a downloadable package, no scripting required. It’s not a full forensic image, and it’s not meant to be. It’s the fast first pass that tells you whether you’re looking at noise or something that actually warrants isolating the host and reaching for the heavier tools.

In this article, we’ll walk through pulling logs from an endpoint, and what’s actually inside the package once you unzip it.

1. Collect the logs

You can access to the SentinelOne Console first after that you can go to the Inventory > Endpoint

• Select the Endpoint you want to collect the logs

• Click on Actions > Endpoint > Troubleshooting > Fetch Logs

• It will pop up Fetch Logs options that you need to select the Agent Logs and Endpoint Logs after that you can click on Fetch Logs

2. Download the collected logs

After you fetch the logs, you must be wondering. Where can I download it? Because after you click the Fetch logs, there is no pop up again or even the logs is not directly downloaded.

• After you click on the Fetch logs you may need to wait 1–5 minutes until the logs is fully fetched, to check. You can go to the Activities tab menu on your left.

• Now if you already see the Download File, you can click it and download the fetched logs
• The file is successfully downloaded.

3. Download the collected logs

After you download and extract the files, you can see a lot of files in it

• Scan_report.txt

Contains information about summary of the last scan: files scanned, and any malicious detections found

For example, on this logs we found out a lot of Malicous files because this endpoint is tested as a labs and installed with a lot of script from the Atomic Red Team.

• LastScanReport.logs

Verbose scan engine log (mostly “file locked, skipped” noise or not detections)

• FullActivityAnalyzerReport.txt

This file contains information about the SentinelOne Agent activity used to troubleshoot interoperability or performance problems that might be related to SentinelOne Agent conflict with other applications.

• PlatformLogs\EventViewer Folder: Includes key logs like:
  - Application
  - System
  - Security
  - Hardware Event
  - Kernel Event
  - Plus SentinelOne’s own event channels again

• PlatformLogs\Misc Folder

This Misc Folder have a lot of information that you can see through.

• NetStat-All.txt : contains active connections
• DnsCache.txt : resolver cache
• VssLog.txt : shadow copy status
• AllApps.txt : installed software
• LoadedModules.txt : per-process loaded modules
• AdvFirewall.txt : firewall rule export

Conclusion and Final Thoughts

Fetch Logs isn’t a full forensic collection, and it was never meant to be one. What it gives you is the answer to the one question that actually matters at the start of triage. This step is easy to skip, especially when an alert fires and the instinct is to go straight for the bigger tools. A five-minute log pull is usually enough to tell you which pile something belongs in before you jump it to the bigger tools to check more deeper.

Collecting and Reviewing Endpoint Logs with SentinelOne, Before Going Deep was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

One more welcome back.

Part 1 gave you six single-host artifact categories. Part 2 took you past where the classic checklist stops — auditd, kernel-module and eBPF rootkits, containers and Kubernetes, cloud credential theft, supply chain compromise, file integrity baselines, timestomping, memory forensics, and hunting at fleet scale — and closed with a five-step methodology: baseline, hypothesis, collect, analyze, repeat.

That methodology is only as good as the hypotheses you feed it, though. Knowing the artifacts is necessary but not sufficient — the actual skill in threat hunting is generating a hypothesis worth chasing and knowing which artifacts answer it.

So that’s where Part 3 picks up: eight worked scenarios, in the same hypothesis → artifacts → pivot shape as the backdoor-user example from Part 2, covering everything from cryptomining to ransomware staging to living-off-the-land exfiltration.

After that, we will talk about why attackers do what they do, how to avoid drowning your on-call team in false positives, and close out the series with the checklist and challenge that ties all three parts together.

Let’s finish this.

Worked Hypotheses: Eight Scenarios to Practice On

The backdoor-user hypothesis above is a great starting example, but the real skill here is generating and chasing many hypotheses. Here are eight more, in the same hypothesis → artifacts → pivot format, drawn from the artifact categories above.

1. Cryptomining infection. Hypothesis: “This server’s CPU usage spiked, and I think it’s mining cryptocurrency.” Artifacts: the process list for miner signatures (xmrig, minerd, or simply an unexplained high-CPU process with a masqueraded name), outbound connections to known mining-pool ports (3333, 4444, 5555, 14444), and cron or systemd-timer entries that silently relaunch the miner if it's killed. Recent real-world campaigns have specifically used tampered PAM modules to hide XMRig activity from process listings — so this scenario is a natural pivot back into the persistence section above.

2. Web shell on an internet-facing application. Hypothesis: “I think this web server has been compromised through the application, not SSH.” Artifacts: access logs for POST requests to unusual .php/.jsp/.aspx paths returning 200, recently modified files inside the webroot, files containing functions like eval, base64_decode, system, or exec that don't match the application's known codebase, and — critically — a process tree showing the web server's own user (www-data, apache) spawning a shell, which should never happen in a healthy application.

3. Container escape / privileged container abuse. Hypothesis: “A container on this host may have broken out to the underlying host.” Artifacts: unexpected host-level processes owned by the container runtime’s UID, docker.sock mounted inside a container, host paths mounted read-write inside a container that shouldn't need them, and new host-level cron or systemd entries created immediately after a suspicious container was scheduled.

4. Cloud credential theft via SSRF. Hypothesis: “The app on this instance has an SSRF vulnerability, and I think credentials were stolen from the metadata service.” Artifacts: application logs showing requests to 169.254.169.254, cloud audit logs showing API calls from that instance's role originating from an unfamiliar source IP, and IAM actions being called that the application itself would never need.

5. Supply chain compromise via CI/CD. Hypothesis: “A build pipeline on this box pulled in a malicious dependency.” Artifacts: pip/npm/gem install logs for unfamiliar or typosquatted package names, curl | bash patterns in build scripts, outbound connections initiated during a build step rather than at runtime, and modified lockfiles that don't match version control.

6. Ransomware staging (pre-encryption). Hypothesis: “I think this host is being staged for a ransomware deployment, not already encrypted.” Artifacts: mass file enumeration activity, tampering with backup services (checking whether backup agents or snapshot schedules were recently disabled), and an unusual volume of outbound traffic — staging for double-extortion exfiltration — shortly before any encryption activity begins. Worth calling out on its own: a lot of Linux ransomware specifically targets ESXi hosts and NAS appliances rather than general-purpose servers, so if you run either, they deserve their own dedicated hunt.

7. Lateral movement via SSH key reuse. Hypothesis: “The attacker moved from this host to others using stolen SSH credentials.” Artifacts: known_hosts entries for systems the account owner wouldn't normally reach, authorized_keys additions on destination hosts matching a key first seen on the origin host, and correlating wtmp login timestamps across multiple hosts to reconstruct the pivot path.

8. Living-off-the-land data staging and exfiltration. Hypothesis: “Data was collected and staged for exfiltration using built-in tools, not custom malware.” Artifacts: large tar/zip/gzip archives appearing in /tmp or /var/tmp shortly before a large outbound transfer, curl/wget/scp invocations with unusual destinations in shell history, and — for cases where outbound HTTP is blocked but DNS isn't — DNS- or ICMP-based exfiltration channels.

The “Why” Matters

Here’s something I’ve learned from years of hunting: attackers have patterns.

• They want credentials → they’ll check /etc/shadow, memory, bash history
• They want persistence → they’ll modify cron, systemd timers, PAM, or ld.so.preload, or add SSH keys
• They want to move laterally → they’ll SSH to other systems, leaving traces in known_hosts and shell history
• They want to exfiltrate → they’ll create archives, which show up in filesystem listings, or lean on DNS tunneling when the front door is watched

When you understand why attackers do what they do, you know where to look.

A Note on False Positives

Everything above will occasionally point at something completely innocent, and it’s worth naming that plainly rather than letting new hunters get burned by it. A new sysadmin account, a legitimate cron job dropped by Ansible or Puppet, and normal known_hosts growth from a jump host all look identical to attacker activity at first glance. This is exactly why Step 1 — establishing a real baseline from golden images, config management state, and your asset inventory — isn't optional scaffolding. It's the difference between a hunt that finds evil and a hunt that just generates noise for the on-call team.

Your Hunting Checklist

Before you start your next Linux hunt, here’s a quick cheat sheet based on the SANS FOR577 poster (which lives on my wall, by the way). FOR577 is SANS’s dedicated course on Linux Incident Response and Threat Hunting, and the poster distills a huge amount of that curriculum into something you can glance at mid-hunt.

The Challenge

Here’s my challenge to you: pick one Linux system this week and hunt it.

Not a critical production server. Maybe a test box. Maybe a container. Just pick something and go through the checklist.

Look at the user accounts. Check the cron jobs and systemd timers. Examine the shell history. Check /etc/ld.so.preload. See what's listening on the network. If it's a container host, check for docker.sock exposure while you're at it.

You might find nothing. That’s fine — now you have a baseline.

Or you might find something weird. And that weird thing? That’s where the fun begins.

Because the artifacts are there. They’re always there. You just need to know where to look.

One Last Thing

Three parts and roughly two dozen artifact categories ago, I told you I used to be terrified of Linux DFIR and hunting. I’m not anymore, and if you’ve made it this far, I’d bet you’re a lot less terrified than when you started Part 1 too.

That’s really the whole point of this series. Not to hand you a checklist to run once and file away, but to build the instinct — the artifact mindset — so that the next time someone asks “could this box be compromised?” you don’t feel that old pass-on-Linux dread. You open a terminal instead.

If you run this against a real system and find something, I’d genuinely like to hear about it — drop it in the comments. If you find nothing, that’s a win too; you just built a baseline you can trust next time something looks off.

Go hunt something.

Long Live Cyber Defender!!!

Hunting the Evil : A Practical Guide to Linux Threat Hunting — Part 3 was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

When investigating a compromised Linux machine, the first challenge is usually not a lack of evidence.

It is the opposite.

A single server can generate thousands of log entries, authentication records, audit events, service messages, and other traces of activity. Buried somewhere inside that data may be the evidence needed to understand what happened.

The problem is figuring out where to start.

Instead of manually collecting individual files from a Linux host, this walkthrough uses UAC (Unix-like Artifact Collector) to gather a structured collection of forensic artifacts and then brings the relevant evidence into DFIR Companion for further examination.

The goal is simple:

Collect first. Organize the evidence. Correlate the activity. Then investigate.

Why Collecting Linux Evidence Matters

A Linux system leaves behind traces of activity in many different places.

Authentication attempts may appear in authentication logs. System events can be recorded by syslog, while Linux auditing can provide another layer of visibility through audit.log. Individual applications and services may also maintain their own records.

Looking at only one source can therefore give an incomplete picture.

For example, an authentication event might not immediately look suspicious. But if that event is followed by an unusual command, a new process, or another related system event, the overall sequence may tell a very different story.

That is why the first stage of an investigation is making sure the evidence is collected in a way that allows those relationships to be examined later.

Preparing UAC

For this investigation, I used UAC (Unix-like Artifact Collector) to acquire artifacts from the Linux host.

The version used in this walkthrough is UAC 3.3.0.

The archive can be downloaded directly from the UAC release:

curl -L https://github.com/tclahr/uac/releases/download/v3.3.0/uac-3.3.0.tar.gz -o uac3.3.0.tar.gz

After downloading the archive, extract it:

tar -zxvf uac-3.3.0.tar.gz

Then enter the extracted directory:

cd uac-3.3.0

At this stage, the collector is ready to be executed.

Building the Evidence Collection

UAC provides different collection profiles. For this walkthrough, the full profile is used to obtain a broad collection of artifacts from the host.

Run the collector with elevated privileges:

sudo ./uac -p full <path>

The <path> should be replaced with the location where you want the collected evidence to be stored.

Using elevated privileges is important because some system artifacts are not accessible to a normal user.

Once UAC finishes, the resulting collection is packaged into a compressed archive, accompanied by a log describing the collection process.

This archive is what we will take into the analysis environment.

Moving from Acquisition to Analysis

After the collection has completed, transfer the generated archive to the machine where the forensic analysis will be performed.

Once it arrives, extract the archive.

The extracted directory represents the evidence gathered from the Linux host.

This is an important point in the workflow: the original host is the source of evidence, while the analysis environment is where we examine it.

Keeping these stages separate makes the investigation easier to manage and reduces the need to repeatedly interact with the original system.

Setting Up the Investigation in DFIR Companion

With the UAC collection available, the next stage is to create an investigation in DFIR Companion.

Start by creating a new case using this button below.

The case should contain the basic investigation information, including:

• Case ID
• Case name
• Investigator

These details provide the investigation with an identifiable context before any artifacts are imported.

Then turn the AI analysis feature on

Giving DFIR Companion the Evidence

DFIR Companion provides an artifact-import function that allows the collected evidence to be added to the case, by using this button.

However, there is an important consideration before uploading anything:

Do not assume that uploading more artifacts will automatically improve the investigation.

A large collection can contain useful information, but it can also contain a significant amount of irrelevant data.

The better approach is to understand the incident first and then select artifacts that can help answer the investigation questions.

For a Linux server, examples of potentially relevant evidence include:

• audit.log
• auth.log
• Service-specific logs
• Other logs generated by applications running on the host

The exact selection depends on what is being investigated.

For example, an investigation involving suspicious login activity would naturally place more emphasis on authentication-related evidence. An investigation involving a particular service would require greater attention to that service’s logs.

The point is not to collect everything simply because it is available.

The point is to collect evidence that can help explain the event.

Letting the Evidence Process

Once the relevant artifacts have been imported, DFIR Companion processes the evidence.

At this stage, the tool can analyze the available artifacts and produce findings that can help direct the investigation.

Rather than manually reviewing every piece of collected data from beginning to end, the generated results provide a starting point for identifying events that deserve closer attention.

After processing has finished, the investigation can be explored through the available dashboard views.

Reviewing the Investigation Dashboard

After the artifacts have been processed, DFIR Companion presents the investigation through a centralized dashboard.

The dashboard separates the investigation into two main areas.

Active Leads highlights findings that currently need attention. In this case, it points to issues such as a MongoDB instance running without access control and an external SSH login followed by privilege escalation to root.

Since Your Last Review provides a quick view of newly updated findings and changes in severity, making it easier to spot developments that may require further investigation.

This dashboard gives the analyst a quick overview of the most important findings before diving deeper into the underlying evidence.

Exporting the Investigation Results

Once the analysis has been completed, the results can be exported for further use.

DFIR Companion can provide investigation outputs such as:

• Forensic reports
• IOC blocklists
• Forensic timelines

These outputs can be useful for documenting the investigation and communicating the results to other analysts or incident responders.

However, exported results should still be reviewed before being treated as the final forensic record.

Below are the result of the report

Final Thoughts

Linux forensic investigations can quickly become difficult when evidence is scattered across different logs and system sources.

A collection tool such as UAC can simplify the acquisition stage by gathering a broad set of artifacts from the host. Once that evidence has been transferred to an analysis environment, DFIR Companion can help turn the collected data into findings, timelines, attack mappings, and indicators that are easier to investigate.

But the most important part of the workflow is not the automation.

It is correlation.

A single event rarely explains an incident by itself. Its significance often becomes apparent only after it is connected with other activity occurring around the same time.

That is why a good forensic workflow should not be:

Collect everything → upload everything → trust the findings.

Instead, it should be:

Collect → select → correlate → investigate → validate.

UAC provides the evidence foundation. DFIR Companion helps organize and analyze that evidence. The investigator provides the context and makes the final determination.

And ultimately, that is what forensic investigation is about — not simply finding suspicious artifacts, but building an evidence-supported explanation of what happened, when it happened, and why the evidence supports that conclusion.

Turning Linux Host Data into Forensic Evidence with UAC (Unix-like Artifact Collector) and DFIR… was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

A compromised Windows machine can contain thousands of files, event logs, registry artifacts, Prefetch files, application logs, and other traces of attacker activity. Collecting everything is usually unnecessary. The real challenge is collecting the right evidence, processing it into a useful format, and then connecting the different artifacts together.

In this tutorial, I will show a practical workflow using KAPE and DFIR Companion.

The workflow to collect and investigate is:

Windows Host → KAPE Collection → KAPE Modules → Parsed Artifacts → DFIR Companion → Investigation

The goal is not to let AI replace forensic analysis. Instead, DFIR Companion can help correlate fragmented artifacts and make the investigation easier to understand.

1. Collect Evidence with KAPE

In evidence collection KAPE is very useful, because it allows us to quickly collect forensic artifacts from a Windows system without manually searching through the entire filesystem.

Place KAPE on the system that will perform the collection.

Open:

gkape.exe

This launches KAPE’s graphical interface.

Enable:

Use Target Options

This allows us to select the KAPE targets that should be collected.

2. Select the Evidence Source

For this investigation, the source is the Windows system partition:

C:\

The destination is the location where KAPE will store the collected evidence. I normally recommend keeping the collected evidence separate from the original source. Also, keep Flush disabled unless there is a specific reason to overwrite the destination. I also use %d in the output filename so the collection date is included automatically.

3. Choose the KAPE Targets

There is no single KAPE target configuration that is perfect for every investigation.

The targets should depend on the incident.

For this experiment, I used:

• !BasicCollection
• Prefetch

The Basic Collection gives us several important evidence sources, including filesystem data, logs, registry hives, and other common forensic artifacts.

Prefetch is collected separately because it can provide useful evidence about program execution.

I also used a custom KAPE target for XAMPP logs because those logs were important to the web-server portion of the investigation.

This is an important KAPE feature:

If the evidence you need is not included in an existing target, create your own target.

For example, if the investigation involves XAMPP, Apache, PHP, or another application, its logs may be worth collecting even if they are not part of your normal collection profile.

4. Run the KAPE Collection

Save it as Zip file

Then execute the collection.

KAPE will collect the selected artifacts and create the forensic dump.

Depending on the amount of data collected and the speed of the system, this can take some time.

The result is your KAPE evidence set in the zip file.

5. Parse the KAPE Evidence

The KAPE collection we did before contains the original forensic artifacts.

Some of those artifacts are already human-readable, but many are much easier to analyze after processing. This is where KAPE Modules become useful.

Move the KAPE dump to the analysis machine.

Open KAPE again and enable:

KAPE Modules

The module source should point to the KAPE dump that was just collected. The module destination is where the processed results will be stored. Again, I keep Flush disabled and use %d in the output filename.

6. Run the Appropriate KAPE Module

The module you choose depends on the investigation.

For this experiment, I used:

!EZParser

This module processes many Windows forensic artifacts using Eric Zimmerman’s forensic tools.

Click Execute and allow KAPE to process the evidence.

The result is a collection of parsed forensic data.

A large portion of the output is available in formats such as CSV, which makes it much easier to search, filter, and analyze.

However, there is an important limitation.

Parsed output is not the same thing as the original evidence. Not every artifact will necessarily be parsed.

The original evidence should always be preserved and reviewed when necessary.

Think of the workflow like this:

Original Evidence → Parsed Evidence → Investigation

The parsed files make analysis easier, but they should not replace the original forensic artifacts.

7. Do Not Upload Everything to DFIR Companion

This is probably the most important practical lesson from this workflow.

It is tempting to take the entire KAPE output and upload everything into DFIR Companion.

More evidence does not automatically mean a better investigation.

Too many unrelated artifacts can add noise and make correlation harder.

Instead, select artifacts based on the question you are trying to answer.

For example:

What executed?
Check Prefetch, process creation logs

What files were accessed?
Check Filesystem artifacts, application logs

For this experiment, I selected:

• Prefetch
• Apache access logs

These artifacts were chosen because the incident related to remote code execution from a web.

8. Create New Case in DFIR Companion

Create new case of analysis by pressing this button in the DFIR Companion console.

Enter the data needed like Case ID, Case Name and Investigator in the pop up.

9. Import the Evidence into DFIR Companion

Now we can move to DFIR Companion.

Enable the AI analysis feature. Here

The purpose of this feature is to allow DFIR Companion to analyze the artifacts we provide and correlate information between them.

Use the artifact upload function to import the evidence.

Select the parsed artifact that you want the AI to analyze

The DFIR Companion also able to analyze some unparsed log such as .eml .json .csv .txt and .text. So that other evidence file like .evtx .pcap image file need to be parsed to relevant input

For apache access log its acceptable by the DFIR Companion because its in text format.

10. Assign Severity

After importing an artifact, DFIR Companion asks for a severity level.

In this experiment, the tools used to generate the artifacts did not provide a severity score.

Therefore, Info can be used when there is no better classification available.

Severity should not be confused with proof of malicious activity.

A forensic artifact can be useful even when it is classified as informational.

For example, a Prefetch entry may not be malicious by itself, but it can become important when correlated with:

• a suspicious process
• an unusual command
• a web request
• a file-access event

That correlation is where the investigation becomes interesting.

Wait for the artifact to be imported and analyzed.

11. Choosing The Right Dashboard

When everything is done, we can change the dashboard template by changing the type of the dashboard.

For example we can choose the Triage view to see the findings alerted by DFIR Companion and the timeline.

Through analyst dashboard view we can see the attack mapping and the IoCs marked

12. Report and Presentation

The tool also provide us with the presentation feature

The contents of the presentation slide are the summary and detailed findings found by the tool.

After analyzing the artifacts, DFIR Companion provides export functions for generating outputs such as forensic reports, IOC blocklists, forensic timelines, etc

Conclusion and Final Thoughts

This workflow shows how KAPE and DFIR Companion can complement each other in a Windows forensic investigation. KAPE handles evidence collection and parsing, while DFIR Companion helps analyze and correlate the selected artifacts through findings, timelines, attack mapping, and IoCs.

The key is to collect and provide relevant evidence, rather than simply uploading everything. DFIR Companion can also export investigation results such as forensic reports, IOC blocklists, and forensic timelines. However, the generated findings should still be validated against the original evidence before being used as final forensic conclusions.

From KAPE to DFIR Companion: A Practical Windows Forensic Investigation Workflow was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome back.

If you read Part 1, you already have six artifact categories down: modified user accounts, shell history, persistence mechanisms, networking, processes, and log files. That’s the classic Linux DFIR checklist — solid ground, and if you ran it against a real system like I challenged you to, you probably came away with either a clean baseline or a “huh, that’s weird” moment worth chasing.

If you’re arriving fresh, go grab Part 1 first — the artifact mindset it lays out (attackers touch things, and everything they touch leaves a trace) is the foundation everything below builds on.

This is where the checklist runs out and the real hunting starts. Six single-host artifact categories will catch a lot, but they won’t catch the attacker who’s using eBPF to blind your tooling at the kernel level, the cryptominer hiding behind a tampered PAM module, or the credential thief who never touched a shell at all and just asked your cloud’s metadata service nicely. Part 2 covers auditd, rootkits — including the eBPF-based stealth techniques reshaping Linux rootkit tradecraft right now — containers, Kubernetes, and cloud credential theft, supply chain compromise, file integrity baselines, timestomping, memory forensics, hunting at fleet scale, and finally, the systematic methodology that ties every artifact category from both parts into one repeatable hunting process.

Let’s keep going.

The Auditd Advantage: When You Need More

Sometimes basic logs aren’t enough. That’s when auditd becomes your best friend.

Auditd is Linux’s built-in auditing framework. When properly configured, it can log file accesses, system calls, user activity — basically everything.

Where to look:

• /var/log/audit/audit.log (good luck reading that raw)
• Use ausearch to search: ausearch -m USER_AUTH -ts today
• Use aureport for summaries: aureport -l (login reports)

The format is painful, but the data is priceless.

Rootkits: The Stealthy Ones (2026 Update)

Rootkits deserve special attention because they’re designed specifically to not be found. Historically, that meant kernel modules — Syslogk, Mirai variants, and similar Linux rootkits that operate in kernel space, hide their processes, and generally make your life difficult.

Hunting for kernel module rootkits:

1. Check loaded kernel modules:

lsmod | grep -v "^\("$(cat /proc/modules | cut -d" " -f1 | tr '\n' '|')"\)"

1. Look for modules with weird names or that don’t match your baseline.
2. Use rootkit detectors:

rkhunter --check
chkrootkit

These aren’t perfect, but they catch known rootkits.

1. Look for signs of hiding:

• Processes that don’t show up in ps but exist in /proc
• Network connections to suspicious IPs without corresponding processes
• Files that disappear when you try to list them

1. Check for kernel tainting:

cat /proc/sys/kernel/tainted

Non-zero values mean something loaded that tainted the kernel — possibly a malicious module.

The Syslogk rootkit I covered in my Taiwan presentation is a great example. It hides itself from lsmod but leaves traces — like the /proc/syslogk interface. If you know what to look for, you can find it.

But kernel modules are no longer the whole story. The current frontier of Linux rootkit tradecraft is eBPF — the in-kernel virtual machine originally built for safe, high-performance networking and observability, now being repurposed for exactly the kind of stealth that used to require a hand-rolled kernel module. Recent research has documented eBPF-capable rootkits like LinkPro and VoidLink, both of which specifically target cloud and container environments rather than traditional bare-metal servers — which should sound familiar, because it’s exactly the gap in “where we hunt” that this article is about to address.

What makes eBPF rootkits different from a classic LKM: instead of just hiding files or processes, an eBPF program can attach at the syscall or tracepoint level and selectively filter what monitoring tools themselves see — effectively blinding EDR and audit tooling at the source, rather than lying to ls after the fact. That's a meaningfully harder problem than a userland LD_PRELOAD hook.

Add this to your rootkit-hunting routine:

# Enumerate loaded eBPF programs
bpftool prog list
# Look closely at anything attached via kprobe or tracepoint that you can't account for
bpftool prog list | grep -E "kprobe|tracepoint"

Unexplained eBPF programs — especially ones attached to syscalls like openat, execve, or connect — deserve the same scrutiny you'd give an unexplained kernel module. This is a fast-moving area of research, so treat rootkit hunting as something you revisit quarterly, not something you check once and consider solved.

Beyond the Single Host: Containers, Kubernetes, and Cloud

Here’s the thing about the checklist above: it assumes a single, persistent, bare-metal-style Linux box. That’s a fine assumption in 2015. It’s a dangerous assumption now — because if Linux runs the world, an increasing share of that world is ephemeral, orchestrated, and cloud-native, and none of it behaves the way a traditional server does.

Container and Kubernetes forensics. The first thing that breaks about your normal workflow: ephemeral containers mean your “check file mtimes” instinct often doesn’t work, because the entire filesystem can vanish the moment the container restarts. What to check instead:

• docker.sock exposed inside a container — this is effectively a root-equivalent handle to the whole host, and mounting it into a container is one of the most common ways an attacker escalates from "I popped an app" to "I own the node."
• Privileged containers (--privileged, or securityContext.privileged: true in a Kubernetes pod spec) — flag these on sight during any hunt.
• Host paths mounted read-write inside a container that the workload has no legitimate reason to need — /, /etc, /var/run/docker.sock, or any hostPath volume broader than the app actually requires.
• New host-level cron entries, systemd units, or SSH keys that appear immediately after a suspicious container was scheduled — this is often the first sign an escape actually happened, since the attacker’s next move is almost always to plant persistence somewhere that survives the container’s death.
• /var/log/containers/ and your orchestrator's own audit log (Kubernetes API server audit logs, in particular) for exec calls into running pods — an interactive kubectl exec into a production pod, outside of a deploy or a known debugging session, is worth investigating on its own.

Cloud credential theft. Every major cloud provider exposes an instance metadata service at 169.254.169.254, and it is one of the single richest targets on a compromised cloud host — a successful request can hand over the full temporary credentials for whatever IAM role the instance is running as, no further exploitation required. Hunt for:

• Application logs showing outbound requests to 169.254.169.254 — especially from an app with a known SSRF vulnerability, or one that has no legitimate reason to talk to the metadata service at all.
• Your cloud provider’s own audit trail (CloudTrail, GCP Audit Logs, Azure Activity Log) showing API calls made using that instance’s role, originating from a source IP that isn’t the instance itself — that’s the tell that stolen credentials are being used off-box.
• Locally cached credentials: ~/.aws/credentials, ~/.config/gcloud/, ~/.azure/ — and cloud CLI command history as its own artifact class, right alongside shell history.
• IAM actions being called that the application itself would never plausibly need — a web app’s role suddenly calling iam:CreateUser or s3:GetObject against buckets outside its own scope is a loud signal once you know to look for it.

Supply Chain Compromise

Not every compromise starts with someone SSHing in. A growing number start upstream — in a build pipeline, a dependency, or a base image — and the first artifacts you’ll see are nowhere near /etc/passwd.

What to check:

• pip, npm, and gem install logs for unfamiliar package names, especially typosquats of popular packages (a single transposed letter is enough).
• curl | bash patterns inside build scripts and CI configuration — a shockingly common install pattern that pipes an unverified remote script straight into a shell with build-pipeline privileges.
• Outbound network connections initiated during a build step rather than at runtime — legitimate dependency installation rarely needs to phone home to anywhere but the package registry itself.
• Modified lockfiles (package-lock.json, Pipfile.lock, Gemfile.lock) that don't match what's committed in version control — a mismatch here means something changed dependency resolution outside of your normal review process.

File Integrity Baselines

You can’t spot a tampered system binary if you don’t know what the untampered version looked like. This is the one category of hunting that pays for itself the moment you set it up, because most of the tooling is already sitting on the box.

The fast, no-extra-tooling option — package-manager-native verification:

# Debian/Ubuntu
debsums -c
dpkg --verify

# RHEL/CentOS/Fedora
rpm -Va

Both compare installed files against the cryptographic hashes recorded at install time. A modified pam_unix.so (see the persistence section above) or a trojanized ls/ps/netstat binary will show up here immediately, without needing any separate agent.

For a more rigorous, ongoing baseline, tools like AIDE or Tripwire build and periodically re-check a full file integrity database across the filesystem — worth the setup effort on anything you consider high-value, precisely because it turns “was this file changed?” from a forensic question into a scheduled report.

Timestomping Detection

Attackers who know they’re being hunted will reset a file’s modification time to blend in with its surroundings — touch -r against a legitimate neighboring file is a one-line move. What they forget, more often than you'd think, is that changing a file's content or permissions updates its ctime (change time) regardless of what they do to mtime.

stat -c '%x %y %z %w' /path/to/suspicious/file

That prints access time, modify time, change time, and (on filesystems that support it) creation time side by side. A file whose mtime says “six months ago” but whose ctime says “eleven minutes ago” has been touched recently — the timestomping attempt is the finding.

When Disk Isn’t Enough: Memory Forensics

Everything in this article so far assumes evidence lives on disk. Fileless execution via memfd_create (mentioned in the processes section) and in-memory-only implants exist specifically to break that assumption.

For a live acquisition, LiME (Linux Memory Extractor) captures a full memory image you can pull off-box. For analysis, Volatility 3 has mature Linux support and can reconstruct process lists, network connections, and loaded kernel modules directly from that memory image — catching things that never left a trace on disk at all. You won’t need this for most hunts. You’ll be very glad you know it exists for the ones where the disk artifacts just don’t add up.

Hunting at Fleet Scale

Every technique above describes hunting one box at a time. That’s fine for an incident. It doesn’t scale to “check all 500 production hosts for this indicator by end of day.”

This is where a tool like osquery earns its keep — it exposes the operating system as a queryable SQL table, so a question like “which hosts have an unexpected line in /etc/ld.so.preload" becomes a single query you run across your entire fleet instead of 500 SSH sessions. Pair it with a runtime detection layer like Falco for containers, and the checklist in this article stops being a manual process and starts being something you can actually operationalize.

A Systematic Approach: Putting It All Together

So you have all these artifacts. Now what? How do you actually hunt systematically?

Here’s my process:

Step 1: Establish Your Baseline

You can’t find “weird” if you don’t know what “normal” looks like. What processes usually run? What users normally exist? What cron jobs are legitimate? Your best sources of ground truth here are golden images, your config management state (Ansible/Puppet/Chef), and your CMDB or asset inventory — not just “I’ve been staring at this server long enough to know.”

Step 2: Form a Hypothesis

“I think someone might have compromised this server and created a backdoor user account.”

Step 3: Collect Relevant Data

Based on your hypothesis, gather:

• /etc/passwd and /etc/shadow modification times
• List of users with shells
• sudoers file contents
• SSH authorized_keys files

Step 4: Analyze and Investigate

Found a user that shouldn’t exist? Great — now pivot:

• Check shell history for that user
• Look for their login times in wtmp
• See what processes they ran
• Check cron jobs they might have created

Step 5: Repeat

One finding leads to another. Follow the thread.

That’s the process — baseline, hypothesis, collect, analyze, repeat — and it’s deliberately simple, because the complexity in Linux hunting was never in the methodology. It’s in knowing which artifacts to feed into it. That’s what these two posts have been building toward: fourteen artifact categories, from /etc/passwd modification times to eBPF program enumeration to cloud IAM audit trails, all in service of the same five-step loop.

Here’s my challenge to you, same as before, but bigger this time: pick one Linux system this week — a test box, a container, a cloud instance, whatever you’ve got — and hunt it with everything from both parts. Check the accounts. Check ld.so.preload. Check for eBPF programs you can't explain. If it's a cloud instance, check whether anything's been talking to 169.254.169.254. If it's a container host, check for an exposed docker.sock.

You might find nothing. That’s fine — now you have a real baseline, built the right way, and you’ll know the moment something changes.

Or you might find something weird. And that weird thing? That’s where the fun begins.

Because the artifacts are there. They’re always there — on a bare-metal box, in a container, in a cloud audit log, in an eBPF program table nobody thought to check. You just need to know where to look. Now you do.

See you in Part 3!

Long Live Cyber Defender!!!

Hunting the Evil : A Practical Guide to Linux Threat Hunting — Part 2 was originally published in FMI Cyber Security Consulting Services on Medium, where people are continuing the conversation by highlighting and responding to this story.