OpenSSL "HollowByte" DoS Vulnerability — Memory Exhaustion via Malformed ClientHello (11-Byte Trigger) — Threadlinqs Intelligence
As of 2026-07-18, OpenSSL "HollowByte" DoS Vulnerability — Memory Exhaustion via Malformed ClientHello (11-Byte Trigger) is a medium-severity vulnerability threat, tracked by Threadlinqs Intelligence with 9 detection rules (Splunk SPL, Microsoft KQL, Sigma) and 16 indicators of compromise.
Threat ID: TL-2026-1471 · Severity: MEDIUM · Status: ACTIVE · Category: VULNERABILITY
Okta Red Team disclosed 'HollowByte', an OpenSSL flaw in which an 11-byte TLS ClientHello with a falsified 3-byte length header forces malloc() of up to 131 KB before any real payload arrives;
HollowByte is a remote, unauthenticated denial-of-service vulnerability in OpenSSL's TLS handshake message-buffer allocation logic, publicly disclosed by the Okta Red Team on 2026-06-XX and widely reported starting 2026-07-17/18. Every TLS handshake record carries a 4-byte header, three bytes of which declare the length of the body that is supposed to follow. In vulnerable OpenSSL versions, the code path Read Header -> grow_init_buf() -> OPENSSL_clear_realloc() -> malloc(attacker_size) grows the receive buffer to the full attacker-declared size immediately upon header arrival, before a single byte of the actual body has been received and before any authentication or handshake validation occurs. For a ClientHello, this allows an 11-byte packet (4-byte header + 7 bytes of handshake preamble) with an inflated length claim to force allocation of up to 131 KB of heap memory per connection. Because the server then blocks the worker thread waiting indefinitely for body bytes that are never sent, an attacker can open large numbers of such connections cheaply.
The second-order effect is what makes HollowByte durable rather than transient: on glibc-based Linux systems, when OpenSSL frees the allocated buffer after the connection is torn down, glibc's allocator retains small-to-medium freed chunks in its internal free lists rather than releasing them back to the kernel via sbrk/munmap. By varying the claimed length across successive connection waves, an attacker prevents the allocator from cleanly reusing previously freed chunks, driving progressive heap fragmentation. This fragmentation persists even after the malicious connections are closed and the attack traffic stops — Okta's research states 'the only fix is to kill the process,' i.e., a full worker/process restart is required to reclaim the wasted memory, not merely waiting out the connections.
Okta demonstrated the attack against NGINX (compiled against a vulnerable OpenSSL) in two environments: a 1 GB RAM instance was driven to an out-of-memory kill after roughly 547 MB was consumed and fragmented, and a 16 GB RAM instance had approximately 25% of total system memory (~4 GB) locked up without tripping standard connection-rate or connection-count limiting defenses, since the attack requires comparatively few connections and low bandwidth to inflict disproportionate memory cost.
OpenSSL's maintainers classified the fix as a 'bug or hardening' change rather than a formally tiered security vulnerability (Critical/High/Medium/Low), and consequently issued no CVE, no dedicated vulnerability advisory, and no changelog entry calling out the issue by name — it shipped silently inside the routine OpenSSL 4.0.1 release (and 3.6.3/3.5.7/3.4.6/3.0.21 backports) on 2026-06-09, a release that also happened to bundle unrelated, separately-CVE'd fixes (e.g., a PKCS7_verify() use-after-free and a QUIC PATH_CHALLENGE memory-growth issue). This absence of a CVE/advisory materially complicates detection for defenders whose patch-management pipelines gate on CVE feeds or OpenSSL's official security advisory list, since HollowByte does not appear there.
The remediation implemented in OpenSSL upstream pull requests #30792 (master/4.0), #30793 (3.6/3.5/3.4 backport), and #30794 (3.0 backport) replaces the eager full-size allocation with incremental buffer growth: init_buf is now expanded only as bytes actually arrive on the wire, so an attacker's declared-but-undelivered length costs the server nothing beyond the bytes genuinely received.
Notably, the fix as shipped covers only the TLS handshake code path. Reporting on the disclosure states that DTLS handshake processing uses the same buffer-sizing logic and has not been remediated, meaning UDP-based TLS (DTLS) services built on OpenSSL may remain exposed to an equivalent memory-exhaustion primitive. OpenSSL is embedded, directly or transitively, in NGINX, Apache httpd, Node.js, Python (ssl module), Ruby, PHP, MySQL, PostgreSQL, and is the default T
Weaknesses (CWE)
CWE-400, CWE-770, CWE-789, CWE-20
Target sectors: technology, government administration, finance, health, telecoms, retail, cloud-hosting
Target regions: Global
Detections & IOCs
As of 2026-07-28, this threat has 9 detection rule(s) across Splunk SPL, Microsoft KQL and Sigma, and 16 indicator(s) of compromise. Detection query text and full IOC values are available to authenticated users and programmatically via the Threadlinqs MCP server (Purple tier). View plans.
VULNERABILITY, MEDIUM, threat intelligence, cybersecurity, T1595, T1587.004, T1190, T1203, T1499.003, T1499.001, T1498.001, T1489, T1518, T1046