Encrypted traffic analysis: encrypted traffic inspection without decryption

Encrypted traffic analysis is the practice of inferring security-relevant facts about a network session from what remains observable when the payload is encrypted: handshake attributes, packet sizes and timing, flow records, and destination context. Rather than breaking encryption open, it reads the metadata that encryption necessarily leaves exposed.

It is the encrypted-specific branch of network traffic analysis, and it exists because attackers moved into encryption along with everyone else. This guide covers what survives Transport Layer Security (TLS) 1.3, QUIC, and Encrypted Client Hello (ECH), how TLS fingerprinting turns cleartext handshakes into client identities, and when decrypting is still worth its cost.

What is encrypted traffic analysis?

General traffic analysis studies flows of every kind, plaintext included; this page covers only the encrypted-specific question of which signals still support detection once the payload is unreadable.

The discipline, often abbreviated ETA, also goes by encrypted traffic inspection and encrypted traffic analytics, three names for one metadata-first approach. Within a network security program it lives inside network detection and response (NDR), complementing endpoint detection and response (EDR) by watching traffic from devices no agent can reach.

How much traffic and how much malware is actually encrypted?

The honest answer depends on which population you measure. Roughly 95% or more of web traffic now loads over HTTPS, and Android and Mac exceed 99% for public sites only (Google Transparency Report, reported around October 2025). Google's October 2025 HTTPS by default announcement corroborates those figures.

Attacks moved with it. A cloud-security vendor's telemetry recorded 87.2% of 32.1 billion blocked attacks arriving over encrypted channels between October 2023 and September 2024, up 10.3% year over year (Network World). In the same dataset, malware accounted for 86.5% of blocked encrypted attacks, and manufacturing was the most-targeted industry at 13.5 billion attacks.

WatchGuard's biannual threat report found 96% of blocked malware delivered over TLS in the second half of 2025 (WatchGuard via GlobeNewswire, published February 2026).

Page loads, blocked attacks, and blocked malware are three different populations measured over three different periods: three facts, not one trend.

A three-row table separating HTTPS page-load share, blocked attacks over encrypted channels, and blocked malware over TLS by population, period, and source.

What survives encryption: the metadata that still reaches your sensors

Every encrypted session opens with a negotiation, much of it in the clear. The TLS handshake exposes the ClientHello with its offered cipher suites, extension list, and application-layer protocol negotiation (ALPN) values, plus the server name indication (SNI) naming the requested host. All of this arrives before any payload is protected, which is why the TLS handshake anchors encrypted traffic analysis without decryption. Certificate metadata such as issuer and validity adds infrastructure context, with one protocol caveat worth stating precisely: TLS 1.3 encrypts the server Certificate message under handshake keys (RFC 8446), so a passive sensor reads certificates off the wire only on TLS 1.2 and earlier sessions, and otherwise sources them by resolving the destination itself.

Beneath the handshake sits the flow layer, and it never encrypts. Packet sizes, inter-arrival timing, direction, session duration, byte and packet counts, and upload-to-download ratios describe the shape of a conversation, and they read the same for legacy SSL encrypted traffic as for TLS 1.3. Two classic feature sets formalize this: the sequence of packet lengths and times (SPLT) and the initial data packet. NetFlow and IP Flow Information Export (IPFIX) records are the usual collection substrate, the same summaries that power broader traffic analysis.

Destination context completes the picture: which port carries which protocol, and whether the pairing makes sense. Certificate pinning belongs here too. A pinned client accepts only a predefined certificate for a service, which defeats interception proxies and gives analysts a stable certificate relationship to observe. Because direction and volume survive, a session whose upload dwarfs its download toward an unfamiliar destination is a classic exfiltration shape.

None of this is speculative. ENISA cataloged six use cases for the discipline in November 2019: application identification, network analytics, user-information identification, encrypted malware detection, fingerprinting, and DNS tunneling detection (ENISA). Treat it as canon that established feasibility, not a source of current statistics. It also warns that these techniques lower user privacy expectations, a tension the decrypt decision below inherits.

Which handshake and flow attributes matter most for detection?

The table below summarizes what reaches a sensor, what it tells an analyst, and the caveat each observable carries.

A seven-row table showing which session observables remain visible under encryption, what each contributes to detection, and its limitations.

What encryption actually takes away

Encryption removes the payload bytes: full URLs beyond the hostname, request and response bodies, file contents, credentials, and the byte patterns that signature engines and content-matching data loss prevention depend on. Any control that must read content goes blind, which is exactly the trade the decrypt decision weighs later. What survives is the shape of the conversation, and the shape is where most detection signal lives.

TLS fingerprinting, from JA3 to JA4

TLS fingerprinting turns the cleartext handshake into a client identity, computed from bytes every session must send before encryption starts. Salesforce engineers open-sourced JA3 in 2017 and later paired it with JA3S: JA3 concatenates five ClientHello fields, the TLS version, cipher suites, extension list, elliptic curves, and elliptic curve point formats, then hashes the string into a 32-character MD5 value (Salesforce Engineering). JA3S does the same for the server's reply. Because client software rarely changes how it builds a handshake, the hash behaves like an identity: malware built on a given library presents the same JA3 fingerprint no matter which domain it contacts.

A worked fingerprint

Picture a beacon calling out over TLS: fixed cipher-suite list, fixed extensions, fixed curve preferences, so its JA3 hash is identical on every connection. A hash captured in one incident can be hunted across the entire estate.

What breaks JA3

JA3's weakness is that it hashes the extension list in the order observed. When Chrome began randomizing ClientHello extension order, the same browser started producing a different JA3 hash on every connection, and naive JA3 matching collapsed. Operators of offensive tooling exploit the same property deliberately: mutate the handshake and the fingerprint changes.

JA4 fingerprinting is the answer. FoxIO released JA4 in September 2023 as a randomization-resistant redesign and grew it into the JA4+ suite (FoxIO JA4+ repository): JA4 for TLS including QUIC, JA4S for server responses, JA4H for HTTP, JA4L and JA4LS for latency, JA4X for X.509 certificates, JA4SSH for SSH, JA4T, JA4TS, and JA4TScan for TCP, and JA4D and JA4D6 for DHCP and DHCPv6. Open sensors have kept pace: Zeek documented JA4 support in January 2026 (Zeek). Parts of the JA4+ suite are license-gated, a constraint the limitations section returns to, and a determined operator can still mutate any fingerprint, so treat JA3 fingerprinting and JA4 fingerprinting output as strong signals, not identities.

TLS 1.3, QUIC, and Encrypted Client Hello: what actually breaks

Each protocol generation trims encrypted traffic visibility further. TLS 1.3 removed the static key exchanges that passive decryption depended on: with perfect forward secrecy, every session key comes from an ephemeral exchange, and a recorded session cannot be decrypted later even if the server's private key is compromised.

QUIC, the UDP-based transport beneath HTTP/3, encrypts most of the transport metadata that TCP leaves visible and moves traffic to UDP port 443. Adoption is routinely overstated: 40.0% of websites support HTTP/3 as of July 2026 (W3Techs); traffic share, a different measurement, runs roughly 34 to 35% at the content delivery network (CDN) edge and about 21% at page-load level (Cloudflare Radar).

One misconception needs correcting: in baseline QUIC and TLS 1.3, an on-path observer can still recover the SNI, because the Initial packet and ClientHello are protected with keys derived from public values. Only Encrypted Client Hello encrypts it.

A six-row comparison showing SNI recoverable under both transports, transport metadata lost under QUIC, and packet-level and flow features surviving in both.

Encrypted Client Hello: standardized, barely deployed

Encrypted Client Hello encrypts the inner ClientHello, SNI included, under a server public key, leaving only an outer ClientHello visible. It became RFC 9849, Standards Track, in March 2026 after years as draft-ietf-tls-esni; RFC 9848 defines its DNS bootstrapping.

Deployment lags the standard. Measuring traffic from January to March 2025, Corrata found slightly under 10% of the top 1 million sites supporting ECH, only 0.06% of connections actually using it, and support falling to 3% in the top 1,000 and 1% in the top 100 (Corrata, Living with ECH). The same study found no iOS support, and every observed outer ClientHello carried the same SNI, cloudflare-ech.com, leaving ECH sites indistinguishable from one another; about 17% of ECH-enabled sites fell into malicious or risky categories. That measurement is roughly 14 months old and predates RFC 9849, so real usage has probably risen.

Server-side support is arriving: NGINX 1.29.4 shipped native ECH in February 2026 (NGINX), and OpenSSL announced ECH for its upcoming 4.0 release (OpenSSL). Even under full ECH, an analyst keeps timing, packet size, flow shape, the outer ClientHello, and JA4, so what ECH changes about visibility into network traffic is narrower than feared. Plan for it as an architectural certainty, not a present-tense emergency.

How ECH failed in practice: CVE-2026-42505

ECH's first real-world stumble was an implementation defect, not a flaw in the specification. CVE-2026-42505, published July 8, 2026, sits in Go's crypto/tls: pre-shared key identities leaked into the unencrypted outer ClientHello, letting a passive observer de-anonymize ECH handshakes, the exact outcome ECH exists to prevent. Rated a medium 5.3 on the Common Vulnerability Scoring System (CVSS v3.1) and classed under Common Weakness Enumeration entry CWE-201, it was fixed in go1.25.12, go1.26.5, and go1.27.0-rc.2 (GO-2026-5856). The defender's lesson: a standardized privacy mechanism can still leak the thing it protects when implemented incorrectly.

Detecting DNS over HTTPS tunneling

DNS over HTTPS (DoH) wraps DNS queries inside HTTPS on port 443, removing them from resolver logs and DNS-layer monitoring. Detection falls back on flow behavior: ENISA listed DNS tunneling detection among the discipline's founding use cases, and the observables hold up. Tunneled DoH produces query volumes, packet-size patterns, and timing rhythms unlike ordinary browsing, and flows to unsanctioned DoH resolvers stand out on their own.

Encrypted traffic inspection: decrypt, selectively decrypt, or analyze only

Encrypted traffic inspection is really three postures, not two: decrypt broadly, decrypt selectively by risk tier, or analyze without decrypting. The contrast case is deep packet inspection (DPI), the payload-matching engine inside classic intrusion detection and prevention systems. Encrypted payloads blind DPI; it was built for a network that no longer exists.

Broad decryption carries real costs: TLS inspection appliances add latency and capacity load, and turn encrypted traffic management into a permanent program of certificate distribution, key handling, and breakage triage. Certificate pinning breaks outright behind an interception proxy. Re-exposed content creates privacy exposure and legal risk. Perfect forward secrecy sharpens the bill: passive SSL decryption of recorded traffic is gone, so interception must happen in-line, at session time.

NIST SP 1800-37, finalized in September 2025, is the authoritative vendor-neutral reference, documenting risk-based approaches to TLS 1.3 visibility with build guidance from the National Cybersecurity Center of Excellence (NCCoE). Its logic is the middle path: decrypt the narrow tiers whose risk justifies the cost, and analyze the rest. Under the NIST Cybersecurity Framework 2.0, that is how continuous monitoring (DE.CM) and adverse event analysis (DE.AE) stay achievable when payloads are encrypted.

A standards dispute marks the poles. ETSI TS 103 523-3 defines Enterprise Transport Security (ETS, formerly eTLS), a middlebox protocol enabling passive decryption in controlled environments. The IETF objected to the TLS branding, and the EFF objected on forward-secrecy grounds (EFF, 2019). ETS appears here for balance, as the decrypt-side pole, not a recommendation.

That is also the difference between AI-based encrypted traffic analysis and SSL/TLS decryption: decryption re-exposes content for payload tools to read, while analysis infers intent from metadata and behavior without exposing content.

A decrypt-vs-analyze decision framework

Regulatory constraints decide much of this in practice: the General Data Protection Regulation (GDPR), the Payment Card Industry Data Security Standard (PCI DSS), and the Health Insurance Portability and Accountability Act (HIPAA) each restrict what decrypted content may touch.

A seven-row decision matrix assigning decrypt, selectively decrypt, or analyze-only postures to traffic categories by risk tier and regulatory constraint.

The realistic 2026 posture lands where the matrix points: risk-based selective decryption for a narrow set of traffic, with no-decrypt analysis carrying everything else.

Privacy, GDPR, and the limits of inspecting employee traffic

In the EU, decrypting employee or customer traffic collides with GDPR and national labor law, which constrain monitoring of personal communications even on corporate networks. ENISA's 2019 warning applies to the no-decrypt side too: metadata analysis lowers privacy expectations even when content stays sealed. Practical governance follows three rules: document the lawful basis, scope inspection to defined risk tiers, and prefer metadata wherever it answers the detection question. Map the specifics with counsel.

Encrypted traffic analysis in practice: three detection cases

Three documented cases show what metadata-first detection looks like against real tradecraft. Each doubles as a template for hunting in encrypted traffic.

Salt Typhoon. The joint advisory AA25-239A from CISA, NSA, FBI, and international partners describes the campaign's encrypted covert channels: Secure Shell (SSH) on non-standard high ports, generic routing encapsulation (GRE) tunnels, and IPsec, used for lateral movement and to hide command-and-control traffic (CISA, August 2025). FBI statements around the advisory put the scale at a minimum of 200 U.S. companies, with victims across 80 countries (TechCrunch). The bureau reconfirmed the campaign as ongoing in February 2026 (CyberScoop). The detection signal was metadata throughout: encrypted sessions on ports where they do not belong, anomalous authentication patterns, and unusual outbound connections and egress volumes. When adversaries encrypt their command traffic, behavioral analysis of flow metadata is what surfaces the covert channel.

Cobalt Strike command and control. Default and cracked Cobalt Strike builds produce a stable, recognizable ClientHello and a paired server response, so JA3 and JA3S fingerprints identify the beacon and team server in the cleartext handshake, before the tunnel encrypts, with regular beaconing intervals corroborating the match (The DFIR Report, 2022). The handshake exposes frameworks hiding inside encryption; operators can mutate the fingerprint, which is exactly why JA4 exists.

Malware abusing legitimate TLS-protected services. A network-security vendor's 2021 malware telemetry documented droppers and loaders using TLS connections to reputable cloud, paste, chat, and code-hosting services, plus Tor, for staging, command and control (C2), and exfiltration. Destination reputation and encryption both worked against defenders, so the remaining signal was behavioral: unusual periodicity, upload-to-download ratios, and session sizes toward otherwise-trusted destinations, along with TLS and certificate metadata anomalies. Encryption plus legitimate-service abuse defeats payload inspection and domain reputation at once, leaving flow behavior as the residual signal for spotting encrypted malware.

Mapping these cases to MITRE ATT&CK

All three map onto the command and control tactic (TA0011) in MITRE ATT&CK version 19.1; every technique below is observable from network metadata.

A three-row table mapping encrypted channel, application layer protocol, and protocol tunneling techniques to the metadata signals that reveal them.

Every one of these cases was detectable from metadata alone, which is the entire argument for encrypted traffic analysis.

Where encrypted traffic analysis struggles

Vendor pages tend to present machine learning (ML) classification of encrypted traffic as a solved problem. The peer-reviewed literature does not. A 2024 survey in Sensors by Alwhbi, Zou, and Alharbi maps the gap between benchmark results and production reality (MDPI, also via the NSF public-access repository). The recurring limits:

• Concept drift. Models decay as real traffic distributions shift away from their training data, so benchmark accuracy overstates field performance.
• Adversarial evasion. Padding and timing manipulation defeat classifiers that lean on size and interval features (arXiv 2105.14564).
• Dataset bias. Benchmark corpora such as the VPN and non-VPN sets do not resemble a production enterprise, so models generalize poorly. A comparative study of encrypted malicious traffic detection found that the lack of well-recognized datasets and feature sets leaves published model performance impossible to compare reliably (arXiv 2203.09332).
• False-positive cost. On high-volume flow telemetry, even a small false-positive rate becomes an operational disaster measured in analyst hours.
• Fingerprint mutability. JA3 and JA4 are strong signals, not identities; an operator who controls the client controls the fingerprint.

License terms are a further, under-discussed constraint: Security Onion 3.0.0, released March 31, 2026, gates its JA4+ integration behind explicit acceptance of the FoxIO license (Security Onion). Check license posture before building on the full suite.

None of this argues for a return to signatures: in the second half of 2025, 23% of detected malware evaded signature-based detection entirely (WatchGuard via GlobeNewswire). The honest framing: lab accuracy above 90% is real, and it is not the same as field performance. Baselining methodology belongs to network anomaly detection, and pairing model output with behavioral analytics context keeps the false-positive cost survivable.

Modern approaches to encrypted traffic analysis

The discipline is converging on metadata-first detection: TLS fingerprints as standard sensor output, flow behavior feeding anomaly models, and threat detection that correlates network evidence with identity context instead of matching payloads. Whether you are weighing commercial NDR tools or open sensors such as Zeek and Security Onion, the questions are capability questions: does it work without decryption, does it cover QUIC as well as TLS over TCP, does it produce fingerprints you can pivot on, and does it degrade gracefully when ECH arrives.

How Vectra AI thinks about encrypted traffic analysis

Vectra AI approaches encrypted traffic analysis from an assume-compromise starting point: capable attackers will get in, and once inside they will encrypt their command and control as the rest of the network does. If detection depends on reading payloads, encryption has already decided the outcome, so the methodology treats metadata as primary evidence rather than a fallback. The useful question is not "can we read the payload" but "does this flow behave like an attacker": does it beacon on a rhythm no sanctioned service uses, does it move data in shapes no workload should, does the identity behind it act as it always has. Answering that requires behavioral analysis across network and identity together, and it requires separating attack signal from the noise of benign anomalies, because at enterprise flow volume, noise is the default. That signal-over-noise discipline is what makes encrypted traffic analysis operational inside network detection and response rather than academic.

Conclusion

Encryption took the payload away from defenders, not the conversation's shape, and the shape is where detection now lives: handshake metadata, TLS fingerprints, flow records, and timing survive TLS 1.3, QUIC, and, in large part, even ECH. Treat ECH as an architectural certainty and plan sensors that keep working when it arrives. Decrypt narrowly where risk and law justify the cost, analyze everything else, and judge every approach by what it sees without decrypting.

‍