Botnet detection techniques for finding modern C2

This guide covers the detection layer only: how SOC analysts and detection engineers find botnet activity on the wire, and where each technique breaks. If you need the fundamentals — what is a botnet, how the underlying malware spreads, and the major families — start with the pillar page instead.

Botnet detection is the practice of identifying compromised hosts by the network behavior their command-and-control (C2) traffic produces — beaconing periodicity, DNS anomalies, and encrypted-channel fingerprints — rather than by matching known signatures. Because every bot must communicate with its controller to be useful, the control channel is the most durable place to look.

From signatures to behavior — why botnet detection moved beyond IOCs

How are botnets detected? In practice, through behavioral analysis of the control channel: beaconing periodicity, anomalous DNS activity such as domain generation algorithm (DGA) queries and fast flux, encrypted-session fingerprints, and flow-level behavior. Signature and indicator-of-compromise (IOC) matching still has a place, but it only catches infrastructure someone has already seen, reported, and cataloged.

IP and domain reputation lists are the floor. A blocklist stops the commodity bot reusing last month's infrastructure, and a signature-driven intrusion detection and prevention system (IDS/IPS) reliably flags the payloads and C2 handshakes it holds definitions for. The problem is decay: every entry describes infrastructure or tooling an attacker can replace in minutes, so a list loses value from the moment it ships.

The churn is measurable. In its Botnet Threat Update for January to June 2026, Spamhaus observed Sliver overtake Cobalt Strike as the most-encountered C2 framework, rising 58% in six months, while the botnet command-and-control servers it tracked fell 30% to 14,952. Spamhaus also observed .cn C&C domains surge 771% while abused registrations at the REGRU registrar fell 90% — operators relocate wholesale when pressure arrives. Any detection anchored to framework-specific signatures is chasing a target that moves faster than the signature pipeline can. For a defender's-eye view of that churn, see what Sliver beaconing looks like in practice.

Behavioral detection inverts the problem. Instead of enumerating known-bad artifacts, you baseline how hosts normally communicate — destinations, timing, volumes, protocols — and surface deviations consistent with a control channel. Network anomaly detection supplies that baselining discipline; the sections that follow apply it to the signals a botnet cannot avoid producing. A bot can swap frameworks, domains, and servers overnight. What it cannot do is stop checking in, stop resolving its rendezvous points, or stop shipping the same client everywhere — and each of those behaviors is observable.

Detecting command-and-control beaconing

Beaconing is the check-in rhythm a bot maintains with its command and control server: the implant sleeps, wakes, asks for tasking, and sleeps again. That rhythm is the most botnet-specific signal on a network, and finding it is a timing problem, not a payload problem. MITRE tracks the transport as T1071, Application Layer Protocol, because check-ins typically ride ordinary HTTPS or DNS.

The naive detector works on periodicity. Group flows by host pair, compute the gaps between successive connections, and look for a distribution that is too regular — a fixed 60-second sleep produces near-identical inter-arrival times that no human browsing pattern generates. Low variance across dozens of observations makes a strong beacon candidate.

Attackers know this, which is why every serious C2 framework ships jitter: each sleep interval is randomized within a configured band, spreading the inter-arrival distribution until simple variance thresholds stop firing. Naive interval matching fails here. But jitter randomizes each individual gap — it does not remove the underlying schedule.

Frequency-domain analysis exploits that weakness. Transform the connection timeline into the frequency domain — a Fourier transform is the standard tool — and a recurring beacon concentrates energy into a peak at its underlying check-in frequency. Jitter widens the peak; it does not erase it. As long as the beacon recurs, it has a fundamental frequency to find, an approach demonstrated publicly against jittered C2.

‍

Flow records are the practical substrate for all of this: they carry the timestamps, byte counts, and durations the analysis needs without any payload capture. Network traffic analysis covers that methodology in depth.

Know the failure modes before you deploy. Low-and-slow beacons that sleep for hours may recur too rarely to resolve inside your analysis window. Event-triggered callbacks that fire on user activity never form a stable frequency at all. And enterprises are full of legitimate periodicity — NTP, update checks, telemetry agents, monitoring probes — so periodicity alone is a shortlist generator, not a verdict. Corroborate candidates with destination rarity, session-size consistency, and the DNS signals in the next section before an analyst ever sees an alert.

DNS-based detection — DGA and fast flux

Bots have a rendezvous problem — they must find their controller even after domains get seized — and the two dominant solutions, domain generation algorithms and fast flux, both leave fingerprints all over DNS.

A DGA generates large numbers of candidate domains on a schedule; the operator registers only a few, and each bot walks the list until one resolves. MITRE catalogs the technique as T1568.002, and its Detection Strategy DET0419 (analytics AN1178–AN1181) formalizes the signals defenders converged on: domain names with high character entropy and low lexical similarity to real language, repeated queries to never-seen domains, and — the loudest tell — sustained NXDOMAIN failures as a host walks candidates that were never registered.

Fast flux attacks the takedown model from the other side: the domain stays put while the infrastructure behind it rotates. The 2025 Five Eyes fast flux advisory describes single flux, where A records rotate through compromised hosts, and double flux, where the name servers rotate as well — MITRE's T1568.001 (DET0485). Its observables are concrete: time-to-live (TTL) values so low a domain may change IP every 3–5 minutes, tens to hundreds of IPs cycled per day, and inconsistent geolocation across a domain's resolutions. The advisory also explains why IP blocking fails — each address is retired before a blocklist can propagate. DNS can carry the C2 channel itself as well, tracked as T1071.004 (DET0400).

The high-yield DNS signals, in the order most teams check them:

1. High-entropy, random-looking domain names with no lexical meaning
2. Sustained NXDOMAIN spikes from a single internal host
3. Newly registered or never-before-seen domains receiving repeated queries
4. Abnormally low DNS TTLs — seconds to a few minutes
5. One domain resolving to tens or hundreds of IPs daily
6. Inconsistent geolocation across a domain's resolved IPs
7. Rotating name-server (NS) records alongside rotating A records
8. Flow records showing many short-lived connections to changing IPs

‍

DGA and fast flux produce distinct DNS signals, each with a different benign look-alike.

That second risk column is structural, and the advisory says so plainly: separating malicious fast flux from legitimate CDN and load-balancer behavior "remains an ongoing challenge." Low TTLs and rotating IPs describe every major CDN. Context separates them — CDN rotations stay inside well-known hosting ranges, while fast flux cycles through large numbers of compromised hosts pressed into service as proxies and relay points. Baseline your resolver logs first; every signal above is a deviation measurement, not an absolute.

Detecting encrypted C2 without decryption

When the C2 channel is encrypted — MITRE tracks the technique as T1573, Encrypted Channel — payload inspection returns nothing, and decrypting at scale is frequently infeasible: TLS interception is costly, and implants that pin certificates simply refuse to connect through it. The practical alternative is fingerprinting the client without touching plaintext.

TLS fingerprinting hashes the parameters a client offers during its handshake — versions, cipher suites, extensions — all visible before encryption begins. A C2 implant ships one TLS stack, so its handshake looks the same wherever it lands. JA3 built the original fingerprint, but in 2023 Chromium began randomizing TLS extension ordering, which broke JA3's stability for a large share of web traffic and motivated its successor, JA4, specified by FoxIO. JA4 resists that randomization, and the broader JA4+ suite extends past TLS to fingerprint TCP traffic itself. Zeek added native JA4 support in January 2026, putting the technique within reach of any team running open-source network monitoring.

Now the limit the primary sources leave unstated: neither the JA4 specification nor the Zeek write-up quantifies false positives, collisions, or drift. Both problems are real. Fingerprints collide — thousands of benign applications share common TLS libraries, so a hash matching a known implant may equally match an ordinary tool built on the same stack. Fingerprints also drift — every client update can change the handshake, silently aging your match lists. Treat a fingerprint as one correlating signal, never a verdict.

This is the general shape of evasion-aware detection. Encryption removes the payload but preserves the handshake and the timing, so fingerprinting pairs with the metadata analysis from the beaconing section — session durations, packet-size distributions, and byte ratios all stay observable in encrypted traffic. The case for modeling C2 behavior instead of decrypting it runs through this entire stack.

Machine learning detection and the false-positive problem

Published machine learning results for botnet detection look spectacular. A 2024 study reported a 1.53% false-positive rate (FPR) on the IoT-23 dataset while identifying 100% of C2 communications packet-based and 94% flow-based. A 2025 stacking-classifier paper reported 97.94% test accuracy on UNSW-NB15 — with 99.99% accuracy on its training set. Read the fine print before budgeting against numbers like these.

Start with what the datasets are. IoT-23 is real malware detonated on real hardware — but in a controlled lab, not a production network. UNSW-NB15 is synthetic traffic generated by an IXIA test tool. Neither contains the messy, drifting, policy-violating traffic of a live enterprise, which is exactly where false positives are born. Benchmark accuracy measures fit to the benchmark.

Now the arithmetic — and to be clear, this is our interpretation, not a claim either paper makes. A 1.53% FPR sounds negligible until the base rate multiplies it: against one million flows a day, a modest volume for a mid-size enterprise, 1.53% is roughly 15,300 false alerts every day. No SOC triages that. By the same logic, we read 99.99% training accuracy as a classic overfitting signal — the model has memorized its lab, and labs do not transfer.

Benchmark results measure fit to the benchmark, not performance on live enterprise traffic.

None of this makes ML useless — it makes precision engineering the real work. Constrain models to narrow, well-baselined questions, correlate their output with the periodicity and DNS signals above, enrich candidates with threat intelligence, and suppress known-benign periodic services before anything alerts. That tuning discipline is detection engineering, and it is what separates a benchmark from a detector. The lesson is vendor-agnostic — the base rate, not any product's model, is why every detection category struggles with botnet false positives.

Detecting decentralized, agentless, and P2P botnets

Modern botnets are increasingly built from devices no security team manages. On Cloudflare's network, the record-scale DDoS bombardment of late 2025 — attacks Cloudflare links to the Aisuru-Kimwolf botnet — traced to a population of "primarily Android TVs" estimated at 1–4 million hosts. In March 2026, the US Justice Department and international partners dismantled the infrastructure behind four botnets that had compromised more than three million IoT devices, as reported by Krebs on Security. Nobody installs an endpoint agent on a TV: for unmanaged IoT devices, the endpoint telemetry column is empty by construction, and the network is the only place detection can happen.

Endpoint and network telemetry see different halves of a botnet, and only the network covers agentless devices.

Residential-proxy botnets sharpen the problem. In 2026, Google's threat-intelligence group (GTIG) described the NetNut/Popa proxy network: at least two million devices enrolled through SDKs embedded in smart TVs and streaming boxes, with 316 distinct threat clusters observed using suspected NetNut exit nodes in a single week. That traffic egresses from ordinary consumer-ISP address space, so reputation and geolocation heuristics score it benign — though the fast flux advisory's flow and geolocation primitives still transfer.

Peer-to-peer architectures remove the central server entirely: there is no single C2 to sinkhole, and flow analysis alone does not solve detection. Graph-based methods model the communication topology instead — published topological detectors averaged an F1 score of 99.140%, though in 2020 and on benchmark topologies. Treat that as research-grade evidence the approach works, not a shipping capability.

Takedowns tell the same story from the other direction. Europol reported 1,025 servers seized in Operation Endgame — effective precisely because those operations had centralized C2. Fast flux, DGA, and P2P all exist to escape that failure mode, which is why finding the devices conscripted into a botnet cannot be outsourced to takedowns.

Mapping botnet detection to MITRE ATT&CK and NIST CSF

Framework mapping keeps this work auditable. Current MITRE ATT&CK is v19.1 — 15 tactics, with the former Defense Evasion split into Stealth (TA0005) and Defense Impairment (TA0112) — and its Detection Strategies and Analytics constructs debuted in v18, released October 28, 2025. On the NIST CSF 2.0 (CSWP 29, released February 26, 2024) side, continuous network monitoring under DE.CM-01 is the primary anchor, DE.CM-09 covers endpoint and computing monitoring, and DE.AE-02, DE.AE-03, and DE.AE-07 cover event analysis, multi-source correlation, and intelligence enrichment. The crosswalk below ties each signal in this guide to both frameworks — and into your broader threat detection program.

Each botnet detection signal on this page maps to a named MITRE ATT&CK technique and a NIST CSF 2.0 DETECT subcategory.

Modern approaches to botnet detection

Everything above converges on one operating posture: assume compromise, watch the network, and model behavior — because the botnet you most need to find runs on a device you do not manage, over a channel you cannot decrypt, using infrastructure no blocklist has seen. Network detection and response (NDR) is the operational layer where these techniques run continuously as engineered, correlated detections rather than one-off hunts. A broad market of botnet detection tools implements parts of this stack; evaluating that landscape is a separate exercise from understanding the techniques it rests on.

How Vectra AI approaches botnet detection

Vectra AI treats botnet detection as a behavioral modeling problem rather than a signature problem. Attack Signal Intelligence models how command and control behaves on the network — beaconing cadence, DNS patterns consistent with domain generation, encrypted-channel characteristics — and correlates those behaviors across hosts to separate a genuine control channel from routine periodicity. Because the analysis is network-based, it applies equally to the agentless devices where no endpoint telemetry will ever exist.

Conclusion

Botnet detection has moved to where attackers cannot follow: the behaviors a bot must exhibit to function. Signatures and blocklists still filter commodity noise, but the durable signals are periodicity that survives jitter, DNS behavior that survives domain churn, fingerprints and timing that survive encryption, and network visibility that survives the absence of an agent. Build those four, map them to MITRE ATT&CK and NIST CSF, and tune with the base rate in mind.

‍