OpenSSL "HollowByte" DoS Vulnerability — Memory Exhaustion via Malformed ClientHello (11-Byte Trigger)

OpenSSL "HollowByte" DoS Vulnerability (TL-2026-1471), also tracked as HollowByte, is a medium-severity software vulnerability, first published 2026-07-18. It has no confirmed attribution, affects OpenSSL Software Foundation OpenSSL, maps to 16 MITRE ATT&CK techniques (T1046, T1070, T1082), and is covered by 9 detection rules and 16 indicators of compromise.

Key facts for TL-2026-1471

Threat ID
TL-2026-1471
Also known as
HollowByte
Severity
MEDIUM
Status
ACTIVE
Category
VULNERABILITY
First published
2026-07-18
Last reviewed
2026-07-18
Attribution confidence
LOW
Motivation
UNKNOWN
Target sectors
technology, government administration, finance, health, telecoms, retail, cloud-hosting
Target regions
Global
Detection rules
9
Indicators of compromise
16

Malware and tooling in OpenSSL "HollowByte" DoS Vulnerability

Malware and tooling: Okta Red Team HollowByte PoC harness (NGINX test environment)

Okta Red Team disclosed 'HollowByte', an OpenSSL flaw in which an 11-byte TLS ClientHello with a falsified 3-byte length header forces malloc() of up to 131 KB before any real payload arrives; repeated randomized-size waves fragment the glibc heap in a way that persists after disconnection, requiring a process restart to recover. Demonstrated against NGINX, no CVE assigned (treated as a silent hardening fix), patched in OpenSSL v4.0.1 with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21 (released 2026-06-09); DTLS handshake processing shares the same logic and remains unfixed.

How OpenSSL "HollowByte" DoS Vulnerability works

HollowByte is a remote, unauthenticated denial-of-service vulnerability in OpenSSL's TLS handshake message-buffer allocation logic, publicly disclosed by the Okta Red Team on 2026-06-XX and widely reported starting 2026-07-17/18. Every TLS handshake record carries a 4-byte header, three bytes of which declare the length of the body that is supposed to follow. In vulnerable OpenSSL versions, the code path Read Header -> grow_init_buf() -> OPENSSL_clear_realloc() -> malloc(attacker_size) grows the receive buffer to the full attacker-declared size immediately upon header arrival, before a single byte of the actual body has been received and before any authentication or handshake validation occurs. For a ClientHello, this allows an 11-byte packet (4-byte header + 7 bytes of handshake preamble) with an inflated length claim to force allocation of up to 131 KB of heap memory per connection. Because the server then blocks the worker thread waiting indefinitely for body bytes that are never sent, an attacker can open large numbers of such connections cheaply.

The second-order effect is what makes HollowByte durable rather than transient: on glibc-based Linux systems, when OpenSSL frees the allocated buffer after the connection is torn down, glibc's allocator retains small-to-medium freed chunks in its internal free lists rather than releasing them back to the kernel via sbrk/munmap. By varying the claimed length across successive connection waves, an attacker prevents the allocator from cleanly reusing previously freed chunks, driving progressive heap fragmentation. This fragmentation persists even after the malicious connections are closed and the attack traffic stops — Okta's research states 'the only fix is to kill the process,' i.e., a full worker/process restart is required to reclaim the wasted memory, not merely waiting out the connections.

Okta demonstrated the attack against NGINX (compiled against a vulnerable OpenSSL) in two environments: a 1 GB RAM instance was driven to an out-of-memory kill after roughly 547 MB was consumed and fragmented, and a 16 GB RAM instance had approximately 25% of total system memory (~4 GB) locked up without tripping standard connection-rate or connection-count limiting defenses, since the attack requires comparatively few connections and low bandwidth to inflict disproportionate memory cost.

OpenSSL's maintainers classified the fix as a 'bug or hardening' change rather than a formally tiered security vulnerability (Critical/High/Medium/Low), and consequently issued no CVE, no dedicated vulnerability advisory, and no changelog entry calling out the issue by name — it shipped silently inside the routine OpenSSL 4.0.1 release (and 3.6.3/3.5.7/3.4.6/3.0.21 backports) on 2026-06-09, a release that also happened to bundle unrelated, separately-CVE'd fixes (e.g., a PKCS7_verify() use-after-free and a QUIC PATH_CHALLENGE memory-growth issue). This absence of a CVE/advisory materially complicates detection for defenders whose patch-management pipelines gate on CVE feeds or OpenSSL's official security advisory list, since HollowByte does not appear there.

The remediation implemented in OpenSSL upstream pull requests #30792 (master/4.0), #30793 (3.6/3.5/3.4 backport), and #30794 (3.0 backport) replaces the eager full-size allocation with incremental buffer growth: init_buf is now expanded only as bytes actually arrive on the wire, so an attacker's declared-but-undelivered length costs the server nothing beyond the bytes genuinely received.

Notably, the fix as shipped covers only the TLS handshake code path. Reporting on the disclosure states that DTLS handshake processing uses the same buffer-sizing logic and has not been remediated, meaning UDP-based TLS (DTLS) services built on OpenSSL may remain exposed to an equivalent memory-exhaustion primitive. OpenSSL is embedded, directly or transitively, in NGINX, Apache httpd, Node.js, Python (ssl module), Ruby, PHP, MySQL, PostgreSQL, and is the default TLS library on most Linux distributions, giving this bug an extremely broad blast radius across internet-facing TLS termination points even though no in-the-wild exploitation has been reported and impact is limited to availability (no code execution or data disclosure).

MITRE ATT&CK techniques used in TL-2026-1471

Discovery

T1046 Network Service Discovery; T1082 System Information Discovery; T1518 Software Discovery

Defense Evasion

T1070 Indicator Removal

Initial Access

T1190 Exploit Public-Facing Application

Execution

T1203 Exploitation for Client Execution

Impact

T1489 Service Stop; T1498.001 Direct Network Flood; T1499.001 OS Exhaustion Flood; T1499.003 Application Exhaustion Flood; T1499.004 Application or System Exploitation

Resource Development

T1587.004 Exploits; T1588.005 Exploits

Reconnaissance

T1590.005 IP Addresses; T1592.002 Software; T1595 Active Scanning

Affected products and versions in OpenSSL "HollowByte" DoS Vulnerability

  • OpenSSL Software Foundation — OpenSSL
    Vulnerable versions: 3.0.x < 3.0.21; 3.4.x < 3.4.6; 3.5.x < 3.5.7; 3.6.x < 3.6.3; 4.0.0
    Fixed in: 4.0.1; 3.6.3; 3.5.7; 3.4.6; 3.0.21
  • F5 / NGINX — NGINX (linked against vulnerable OpenSSL)
    Vulnerable versions: Any build linking a vulnerable OpenSSL version
    Fixed in: Rebuild/relink against patched OpenSSL

Remediation for OpenSSL "HollowByte" DoS Vulnerability

Patches

  • OpenSSL 4.0.1 (master/4.0 branch, PR #30792)
  • OpenSSL 3.6.3 / 3.5.7 / 3.4.6 (PR #30793 backport)
  • OpenSSL 3.0.21 (PR #30794 backport)

Immediate actions

  • Upgrade OpenSSL to 4.0.1, or the matching backport (3.6.3, 3.5.7, 3.4.6, 3.0.21) or later, on all internet-facing TLS-terminating hosts
  • Rebuild/restart dependent services (NGINX, Apache httpd, Node.js, MySQL, PostgreSQL, Python/Ruby/PHP TLS bindings) that statically link or bundle OpenSSL after the package upgrade
  • Do not rely on connection-rate or connection-count limiting alone — HollowByte's per-connection cost/impact ratio evades those controls
  • Schedule periodic worker/process restarts on TLS-terminating services still pending patch to clear fragmented heap memory as a stopgap

Workarounds

  • Front internet-facing TLS services with a WAF/load balancer that enforces stricter idle-handshake timeouts and per-connection memory ceilings
  • Reduce TLS handshake timeout values to shorten the window an attacker can hold a connection open without sending the declared body
  • Periodically restart affected worker processes to reclaim fragmented heap memory until patching is complete

Longer-term hardening

  • Add explicit memory-usage and RSS-growth-rate alerting per TLS-terminating process, independent of connection-count metrics
  • Track OpenSSL upstream commits/PRs and NEWS.md, not only the official CVE/security-advisory list, since silent hardening fixes (like this one) never receive a CVE
  • Inventory all statically-linked or vendored OpenSSL copies (containers, appliances, embedded builds) since standard OS package upgrades will miss them
  • Evaluate DTLS-terminating services for the same exposure, since the shipped fix does not cover DTLS

Weaknesses (CWE) in OpenSSL "HollowByte" DoS Vulnerability

CWE-400, CWE-770, CWE-789, CWE-20

Timeline of OpenSSL "HollowByte" DoS Vulnerability

  • Okta Security publishes 'OpenSSL HollowByte: A DoS Hiding in 11 Bytes,' naming the vulnerability and detailing the NGINX demonstration and glibc heap-fragmentation mechanics.
  • Okta Red Team identifies the OpenSSL init_buf eager-allocation flaw during internal handshake fuzzing/research and privately reports it upstream.
  • Okta repeats the NGINX test on a 16 GB RAM instance, locking up roughly 25% (~4 GB) of total system memory without tripping standard connection-rate or connection-count limiting controls.
  • Okta validates real-world impact against NGINX: a 1 GB RAM test instance is driven to an out-of-memory kill after ~547 MB of fragmented heap is consumed.
  • OpenSSL ships v4.0.1 plus backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21, silently containing the incremental init_buf growth fix (PRs #30792, #30793, #30794) with no CVE or dedicated advisory; the same release bundles unrelated, separately-CVE'd fixes including a PKCS7_verify() use-after-free and a QUIC PATH_CHALLENGE memory-growth issue.
  • BleepingComputer and Cybersecurity News independently publish coverage of HollowByte, both flagging the CVE-less silent patch as a detection and patch-management blind spot.
  • The Hacker News publishes 'OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests,' amplifying the disclosure to a mainstream security audience.
  • Threadlinqs Intelligence Platform ingests the disclosure via automated RSS hunt and opens TL-2026-1471 for research and detection engineering.
  • GBHackers publishes coverage of the disclosure, cited as the hunt-triggering source for this threat record.

Sources cited for OpenSSL "HollowByte" DoS Vulnerability

Threats related to OpenSSL "HollowByte" DoS Vulnerability

Detection coverage for TL-2026-1471

As of 2026-07-18, Threadlinqs Intelligence publishes 9 detection rule(s) for TL-2026-1471 across Splunk SPL, Microsoft KQL and Sigma, covering 16 indicator(s) of compromise. The whole corpus is readable without an account; a free account unlocks full detection query text in Splunk SPL, Microsoft KQL and Sigma; paid tiers add raw indicator values, correlation and the MCP server. Threadlinqs MCP server · View plans.

Threadlinqs Intelligence — Real-Time Threat Detection Platform

[ 0 threats ] [ 0 det ] [ CRIT: 0 ] [ HIGH: 0 ]
// threat_feed
$ sort --newest
Showing all threats

Latest Threats