HollowByte: OpenSSL Pre-Authentication TLS DoS Flaw Bloats Server Memory With 11-Byte Payload — Threadlinqs Intelligence
As of 2026-07-17, HollowByte: OpenSSL Pre-Authentication TLS DoS Flaw Bloats Server Memory With 11-Byte Payload is a medium-severity vulnerability threat, tracked by Threadlinqs Intelligence with 9 detection rules (Splunk SPL, Microsoft KQL, Sigma) and 21 indicators of compromise.
Threat ID: TL-2026-1457 · Severity: MEDIUM · Status: ACTIVE · Category: VULNERABILITY
Okta's Red Team disclosed HollowByte, a pre-authentication denial-of-service weakness in OpenSSL where an unauthenticated attacker sends a malicious 11-byte TLS handshake header declaring a much
HollowByte is a memory-exhaustion denial-of-service weakness in OpenSSL's TLS handshake-message parsing, publicly detailed by Okta's Red Team on July 16, 2026 (published via BleepingComputer July 17, 2026) and independently by Okta's own security advisory (sec.okta.com). The root cause is a classic allocate-before-validate pattern: during the ClientHello / handshake message phase, OpenSSL reads a 4-byte handshake message header containing a 3-byte length field that declares how much payload data will follow. Vulnerable versions call grow_init_buf(), which in turn calls OPENSSL_clear_realloc() -> malloc(), to pre-allocate a receive buffer sized to the attacker-declared length BEFORE any of the actual payload bytes have arrived on the wire or been validated. An attacker who sends just 11 bytes total -- a TCP/TLS connection open plus a malformed 4-byte header claiming an oversized body -- can force the server to allocate up to 131KB per connection while the worker thread blocks indefinitely waiting for payload data that never arrives, an amplification ratio of roughly 11,900:1.
Because each individual allocation (131KB) and each individual connection may fall under naive connection-count or per-request size limits, standard rate-limiting and connection-ceiling defenses are insufficient to stop the attack -- an attacker opens many low-and-slow connections, each looking unremarkable in isolation. Okta's testing showed the effect compounds via glibc's memory allocator behavior: glibc does not immediately return small-to-medium freed allocations to the OS, retaining them for reuse. Waves of connections using randomized claimed sizes prevent heap consolidation and fragment the process heap, so memory remains bloated even after the offending connections are torn down. Okta tested unpatched and patched OpenSSL instances running NGINX under load; in a 1GB-constrained test environment, unpatched OpenSSL servers were driven to an OOM-kill after allocating roughly 547MB; in a higher-spec 16GB environment, the technique consumed approximately 25% of total system memory.
OpenSSL's maintainers characterized the fix as a hardening improvement rather than a security vulnerability warranting a CVE, and it shipped silently as three merged pull requests (#30792, #30793, #30794) bundled into the same June 9, 2026 release train that formally patched 17 numbered CVEs across the actively-maintained branches (4.0.1, 3.6.3, 3.5.7, 3.4.6, 3.0.21). The release train's headline issue was CVE-2026-45447 (CVSS 9.8 Critical), a heap use-after-free in PKCS7_verify(): when a PKCS#7 or S/MIME SignedData structure's digestAlgorithms field is present as an empty ASN.1 SET, OpenSSL may incorrectly free a caller-owned BIO during verification, so a subsequent BIO_free() by the calling application triggers a use-after-free that can crash the process, corrupt the heap, or potentially enable remote code execution; it affects OpenSSL from the 1.0.2/1.1.1 branches through 4.0, with fixes also back-ported to premium-support branches 1.1.1zh and 1.0.2zq. The same release also fixed CVE-2026-34183 (Moderate), unbounded memory growth in the QUIC PATH_CHALLENGE handler -- a remote peer can flood a QUIC client or server with PATH_CHALLENGE frames, forcing allocation of a PATH_RESPONSE frame per challenge that is only freed once the (malicious, non-cooperating) peer acknowledges it, exhausting heap memory and causing abnormal termination -- a resource-exhaustion bug in the same allocate-on-untrusted-input family as HollowByte itself, affecting OpenSSL 4.0, 3.6, 3.5, 3.4, and 3.0 (not 1.1.1/1.0.2). Other issues in the same train included CVE-2026-34182 (CMS AuthEnvelopedData forgery), CVE-2026-35188 (double-free in OCSP staple verification, 4.0.1/3.6.3), CVE-2026-42764 (NULL pointer dereference in QUIC server packet handling, 4.0.1/3.6.3/3.5.7), CVE-2026-45445 (AES-OCB IV bypass enabling universal forgery, all versions), plus roughly ten Low-severity issues spanning ASN.1 parsin
Weaknesses (CWE)
CWE-789, CWE-400, CWE-770, CWE-416
Target sectors: technology, cloud-hosting, finance, government administration, health, ecommerce, telecoms, education
Target regions: Global
Detections & IOCs
As of 2026-07-28, this threat has 9 detection rule(s) across Splunk SPL, Microsoft KQL and Sigma, and 21 indicator(s) of compromise. Detection query text and full IOC values are available to authenticated users and programmatically via the Threadlinqs MCP server (Purple tier). View plans.
VULNERABILITY, MEDIUM, threat intelligence, cybersecurity, CVE-2026-45447, CVE-2026-34183, CVE-2026-34182, CVE-2026-35188, CVE-2026-42764, CVE-2026-45445, T1595, T1592, T1590, T1587, T1583, T1190, T1046, T1518, T1070, T1203