# OpenSSL "HollowByte" DoS Vulnerability — Memory Exhaustion via Malformed ClientHello (11-Byte Trigger)

> Okta Red Team disclosed 'HollowByte', an OpenSSL flaw in which an 11-byte TLS ClientHello with a falsified 3-byte length header forces malloc() of up to 131 KB before any real payload arrives; repeated randomized-size waves fragment the glibc heap in a way that persists after disconnection, requiring a process restart to recover. Demonstrated against NGINX, no CVE assigned (treated as a silent hardening fix), patched in OpenSSL v4.0.1 with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21 (released 2026-06-09); DTLS handshake processing shares the same logic and remains unfixed.

- **Published:** 2026-07-18T00:00:00Z
- **Last reviewed:** 2026-07-18T00:00:00Z
- **Canonical:** https://intel.threadlinqs.com/threat/TL-2026-1471
- **ID:** TL-2026-1471
- **Severity:** MEDIUM
- **Category:** VULNERABILITY
- **Status:** ACTIVE
- **Detections:** 9 · **IOCs:** 16 (full data via the Threadlinqs MCP server — Purple tier)

## Description

HollowByte is a remote, unauthenticated denial-of-service vulnerability in OpenSSL's TLS handshake message-buffer allocation logic, publicly disclosed by the Okta Red Team on 2026-06-XX and widely reported starting 2026-07-17/18. Every TLS handshake record carries a 4-byte header, three bytes of which declare the length of the body that is supposed to follow. In vulnerable OpenSSL versions, the code path Read Header -> grow_init_buf() -> OPENSSL_clear_realloc() -> malloc(attacker_size) grows the receive buffer to the full attacker-declared size immediately upon header arrival, before a single byte of the actual body has been received and before any authentication or handshake validation occurs. For a ClientHello, this allows an 11-byte packet (4-byte header + 7 bytes of handshake preamble) with an inflated length claim to force allocation of up to 131 KB of heap memory per connection. Because the server then blocks the worker thread waiting indefinitely for body bytes that are never sent, an attacker can open large numbers of such connections cheaply.

The second-order effect is what makes HollowByte durable rather than transient: on glibc-based Linux systems, when OpenSSL frees the allocated buffer after the connection is torn down, glibc's allocator retains small-to-medium freed chunks in its internal free lists rather than releasing them back to the kernel via sbrk/munmap. By varying the claimed length across successive connection waves, an attacker prevents the allocator from cleanly reusing previously freed chunks, driving progressive heap fragmentation. This fragmentation persists even after the malicious connections are closed and the attack traffic stops — Okta's research states 'the only fix is to kill the process,' i.e., a full worker/process restart is required to reclaim the wasted memory, not merely waiting out the connections.

Okta demonstrated the attack against NGINX (compiled against a vulnerable OpenSSL) in two environments: a 1 GB RAM instance was driven to an out-of-memory kill after roughly 547 MB was consumed and fragmented, and a 16 GB RAM instance had approximately 25% of total system memory (~4 GB) locked up without tripping standard connection-rate or connection-count limiting defenses, since the attack requires comparatively few connections and low bandwidth to inflict disproportionate memory cost.

OpenSSL's maintainers classified the fix as a 'bug or hardening' change rather than a formally tiered security vulnerability (Critical/High/Medium/Low), and consequently issued no CVE, no dedicated vulnerability advisory, and no changelog entry calling out the issue by name — it shipped silently inside the routine OpenSSL 4.0.1 release (and 3.6.3/3.5.7/3.4.6/3.0.21 backports) on 2026-06-09, a release that also happened to bundle unrelated, separately-CVE'd fixes (e.g., a PKCS7_verify() use-after-free and a QUIC PATH_CHALLENGE memory-growth issue). This absence of a CVE/advisory materially complicates detection for defenders whose patch-management pipelines gate on CVE feeds or OpenSSL's official security advisory list, since HollowByte does not appear there.

The remediation implemented in OpenSSL upstream pull requests #30792 (master/4.0), #30793 (3.6/3.5/3.4 backport), and #30794 (3.0 backport) replaces the eager full-size allocation with incremental buffer growth: init_buf is now expanded only as bytes actually arrive on the wire, so an attacker's declared-but-undelivered length costs the server nothing beyond the bytes genuinely received.

Notably, the fix as shipped covers only the TLS handshake code path. Reporting on the disclosure states that DTLS handshake processing uses the same buffer-sizing logic and has not been remediated, meaning UDP-based TLS (DTLS) services built on OpenSSL may remain exposed to an equivalent memory-exhaustion primitive. OpenSSL is embedded, directly or transitively, in NGINX, Apache httpd, Node.js, Python (ssl module), Ruby, PHP, MySQL, PostgreSQL, and is the default TLS library on most Linux distributions, giving this bug an extremely broad blast radius across internet-facing TLS termination points even though no in-the-wild exploitation has been reported and impact is limited to availability (no code execution or data disclosure).

## MITRE ATT&CK

- T1595 Active Scanning
- T1587.004 Exploits
- T1190 Exploit Public-Facing Application
- T1203 Exploitation for Client Execution
- T1499.003 Application Exhaustion Flood
- T1499.001 OS Exhaustion Flood
- T1498.001 Direct Network Flood
- T1489 Service Stop
- T1518 Software Discovery
- T1046 Network Service Discovery
- T1070 Indicator Removal
- T1588.005 Exploits
- T1592.002 Software
- T1590.005 IP Addresses
- T1499.004 Application or System Exploitation
- T1082 System Information Discovery

## Sources

- [OpenSSL HollowByte: A DoS Hiding in 11 Bytes](https://sec.okta.com/articles/2026/06/openssl-hollowbtye-a-dos-hiding-in-11-bytes/)
- [OpenSSL DoS Vulnerability (HollowByte) - GBHackers](https://gbhackers.com/openssl-dos-vulnerability/)
- [OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests](https://thehackernews.com/2026/07/openssl-hollowbyte-flaw-could-freeze.html)
- [HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload](https://www.bleepingcomputer.com/news/security/hollowbyte-ddos-flaw-bloats-openssl-server-memory-with-11-byte-payload/)
- [OpenSSL "HollowByte" Vulnerability Lets Hackers Crash Servers With Just 11 Bytes](https://cybersecuritynews.com/openssl-hollowbyte-vulnerability/)
- [Grow the init_buf incrementally as we receive data (PR #30792)](https://github.com/openssl/openssl/pull/30792)
- [Grow the init_buf incrementally as we receive data, 3.6/3.5/3.4 backport (PR #30793)](https://github.com/openssl/openssl/pull/30793)
- [Grow the init_buf incrementally as we receive data, 3.0 backport (PR #30794)](https://github.com/openssl/openssl/pull/30794)
- [OpenSSL Release and Advisory Timeline](https://openssl-library.org/news/timeline/)
- [OpenSSL HollowByte DoS Lets 11-Byte TLS Payload Exhaust Server Memory](https://www.mallory.ai/stories/019f6c1e-690e-7f1b-90fd-20831af27147)

## Full data

Detection queries (Splunk SPL / Microsoft KQL / Sigma) and IOC values require the Threadlinqs MCP server (Purple tier): https://intel.threadlinqs.com/mcp

Canonical: https://intel.threadlinqs.com/threat/TL-2026-1471