OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS (No CVE Assigned)

OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS (TL-2026-1459), also tracked as HollowByte, is a medium-severity software vulnerability, first published 2026-07-17. It has no confirmed attribution, affects OpenSSL Project OpenSSL, maps to 15 MITRE ATT&CK techniques (T1190, T1203, T1498), and is covered by 9 detection rules and 16 indicators of compromise.

Key facts for TL-2026-1459

Threat ID
TL-2026-1459
Also known as
HollowByte
Severity
MEDIUM
Status
PATCHED
Category
VULNERABILITY
First published
2026-07-17
Last reviewed
2026-07-17
Attribution confidence
LOW
Motivation
UNKNOWN
Target sectors
technology, government administration, finance, health, retail, telecoms, education, cloud-hosting
Target regions
Global
Detection rules
9
Indicators of compromise
16

Malware and tooling in OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

Malware and tooling: Apache httpd (mod_ssl), NGINX, OpenSSL

Okta Red Team discovered a memory-exhaustion flaw in OpenSSL's TLS handshake buffer allocation logic where an 11-byte crafted ClientHello-stage payload forces malloc() to reserve up to 131 KB based on an unvalidated attacker-declared size, and glibc heap fragmentation prevents the OS from reclaiming memory after the connection drops. OpenSSL treated it as a hardening improvement rather than a formally numbered CVE, fixing it via incremental buffer growth in 4.0.1 with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21.

How OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS works

The vulnerability, publicly nicknamed "HollowByte," lives in the TLS handshake message-reading path of OpenSSL's state machine (ssl/statem/statem_lib.c). When OpenSSL reads a handshake message header it extracts a 4-byte, peer-declared payload-length field and immediately grows the connection's init_buf to that full claimed size via grow_init_buf() -> OPENSSL_clear_realloc() -> malloc(), before a single byte of the actual payload has arrived on the wire. Because the length field is attacker-controlled and unvalidated against the bytes actually delivered, an attacker can send an 11-byte packet (the minimal TLS record + handshake header needed to declare an oversized length) and force the server to allocate up to 131 KB of heap per connection while sending little to no follow-on data.

The attack's potency comes from a second-order effect: glibc's allocator does not always return freed heap arenas to the OS when a connection is torn down, and if the attacker randomizes the declared size on each new connection, freed chunks are the wrong size to be reused for the next allocation. This produces persistent heap fragmentation that survives connection closure, so repeated low-bandwidth connection churn compounds memory pressure over time rather than being bounded by the theoretical per-connection maximum. Okta Red Team's testing showed a 1 GB RAM NGINX instance OOM-killed after roughly 547 MB of fragmented, unreclaimable memory accumulated, and a 16 GB RAM instance had 25% of system memory permanently locked up while the attack traffic also bypassed the server's configured connection-rate limits (because each attacking connection is trivially cheap and short-lived).

This is architecturally the same code family responsible for the 2016 use-after-free in large handshake-message handling (CVE-2016-6309), which originally introduced grow_init_buf() as a replacement for BUF_MEM_grow_clean(). HollowByte does not reintroduce a memory-safety bug (no UAF/overflow) — it is a resource-exhaustion / unbounded-allocation flaw (CWE-789 / CWE-400), which is why OpenSSL's security team classified it as a hardening fix rather than a CVE-worthy vulnerability: no memory corruption, only availability impact, and full remediation ships in every actively maintained branch.

The fix (OpenSSL PRs #30792, #30793, #30794) replaces immediate full-size pre-allocation with incremental buffer growth that expands the init_buf only as bytes are actually received off the wire, capping any single message to roughly 8 incremental reallocations. The change is TLS-only; DTLS was explicitly excluded from the fix due to the added complexity of datagram reassembly, meaning DTLS-terminating services built on the same OpenSSL versions may retain analogous exposure. The patch merged to OpenSSL master and the 4.0 branch on 2026-04-17 after four reviewer approvals, then was backported to the 3.0, 3.4, 3.5, and 3.6 LTS/stable branches.

Because OpenSSL is the default TLS library for NGINX, Apache httpd (mod_ssl), Node.js, Python (ssl module/requests via OpenSSL bindings), Ruby (OpenSSL gem), PHP (openssl extension), MySQL, and PostgreSQL, the practical blast radius spans essentially the entire TLS-terminating web and database tier of a typical Linux server fleet running an unpatched OpenSSL 3.0.x-3.6.x or pre-4.0.1 build. No public exploit code has been released and no in-the-wild exploitation has been observed or claimed; this was disclosed responsibly by Okta's internal red team and patched proactively across all supported branches before public write-up.

MITRE ATT&CK techniques used in TL-2026-1459

Initial Access

T1190 Exploit Public-Facing Application

Execution

T1203 Exploitation for Client Execution

Impact

T1498 Network Denial of Service; T1498.001 Direct Network Flood; T1499 Endpoint Denial of Service; T1499.001 OS Exhaustion Flood; T1499.002 Service Exhaustion Flood; T1499.003 Application Exhaustion Flood

Discovery

T1518 Software Discovery

Resource Development

T1587.004 Exploits; T1588.005 Exploits

Reconnaissance

T1592 Gather Victim Host Information; T1595 Active Scanning; T1596 Search Open Technical Databases

defense-impairment

T1685 Disable or Modify Tools

Affected products and versions in OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

  • OpenSSL Project — OpenSSL
    Vulnerable versions: 3.0.0-3.0.20; 3.4.0-3.4.5; 3.5.0-3.5.6; 3.6.0-3.6.2; 4.0.0
    Fixed in: 4.0.1; 3.6.3; 3.5.7; 3.4.6; 3.0.21
  • F5 / NGINX — NGINX (OpenSSL-linked builds)
    Vulnerable versions: any build linking an affected OpenSSL version
    Fixed in: rebuilt against patched OpenSSL
  • Apache Software Foundation — Apache httpd (mod_ssl)
    Vulnerable versions: any build linking an affected OpenSSL version
    Fixed in: rebuilt against patched OpenSSL
  • Node.js Foundation — Node.js
    Vulnerable versions: builds bundling an affected OpenSSL version
    Fixed in: builds bundling patched OpenSSL
  • Python Software Foundation — CPython ssl module
    Vulnerable versions: builds/wheels linking an affected OpenSSL version
    Fixed in: builds/wheels linking patched OpenSSL
  • Ruby — Ruby OpenSSL gem
    Vulnerable versions: builds linking an affected OpenSSL version
    Fixed in: builds linking patched OpenSSL
  • PHP Group — PHP openssl extension
    Vulnerable versions: builds linking an affected OpenSSL version
    Fixed in: builds linking patched OpenSSL
  • Oracle — MySQL
    Vulnerable versions: builds linking an affected OpenSSL version
    Fixed in: builds linking patched OpenSSL
  • PostgreSQL Global Development Group — PostgreSQL
    Vulnerable versions: builds linking an affected OpenSSL version
    Fixed in: builds linking patched OpenSSL

Remediation for OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

Patches

  • OpenSSL 4.0.1 (fix merged to master 2026-04-17, formally released 2026-06-09)
  • OpenSSL 3.6.3 (backport, released 2026-06-09)
  • OpenSSL 3.5.7 (backport, released 2026-06-09)
  • OpenSSL 3.4.6 (backport, released 2026-06-09)
  • OpenSSL 3.0.21 (backport, LTS, released 2026-06-09)

Immediate actions

  • Upgrade OpenSSL to 4.0.1, or to the backported patch releases 3.6.3, 3.5.7, 3.4.6, or 3.0.21, whichever LTS/stable branch your fleet tracks
  • Rebuild and redeploy any statically-linked binaries (Node.js, Python, Ruby, PHP builds that vendor OpenSSL) against the patched library rather than relying on dynamic linking alone
  • Restart NGINX, Apache/mod_ssl, and any long-running TLS-terminating services after patching, since the vulnerable allocation path is only replaced on process restart, not by config reload
  • Add or tighten TLS handshake timeout and per-IP concurrent-connection limits at a load balancer or WAF ahead of OpenSSL-based origins as a stopgap for hosts pending patch
  • Monitor server memory/RSS and glibc arena fragmentation metrics for anomalous growth correlated with high volumes of short-lived TLS connections

Workarounds

  • Enforce aggressive TLS handshake timeouts and connection-rate limiting at a front-end load balancer/CDN/WAF for internet-facing origins that cannot be patched immediately
  • Reduce OpenSSL's max handshake message size configuration where the application/library exposes such a tunable, to shrink the worst-case per-connection allocation
  • Isolate TLS-terminating services with cgroup/container memory limits so a single amplified process is OOM-killed and restarted rather than exhausting host-wide memory

Longer-term hardening

  • Track OpenSSL security advisories and hardening notes (not just numbered CVEs) as part of routine patch management, since impactful fixes are sometimes shipped without a CVE
  • Adopt automated dependency/SBOM scanning that flags OpenSSL point-release lag across containers, VM images, and statically linked language runtimes
  • Deploy memory-usage anomaly detection / OOM-killer alerting on TLS-terminating edge and database nodes to catch amplification-style DoS earlier
  • Consider deploying a TLS-aware reverse proxy or DPDK/eBPF-based rate limiter that can drop malformed/oversized handshake declarations before they reach OpenSSL
  • Periodically audit third-party base images and vendored language runtimes (Node.js, Python wheels, Ruby gems, PHP builds) for bundled OpenSSL versions that lag upstream

Weaknesses (CWE) in OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

CWE-789, CWE-400, CWE-770, CWE-405

Timeline of OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

  • CVE-2016-6309, a use-after-free in OpenSSL's large handshake message handling, first introduces the grow_init_buf() function later reused in the HollowByte code path (per Fortinet's historical analysis).
  • OpenSSL 4.0.0 is released, cutting deprecated protocols and adding post-quantum support, immediately preceding the branch that receives the HollowByte hardening fix.
  • Okta Red Team's report to the openssl-security team is credited in the PR description as the trigger for the incremental-buffer-growth hardening work.
  • OpenSSL PRs #30792, #30793, and #30794 (incremental init_buf growth for TLS handshake messages) merge to the master/4.0 development branch after four reviewer approvals, crediting Okta Red Team's report as the origin of the fix.
  • The 09 Jun 2026 OpenSSL security advisory bundles the HollowByte hardening fix alongside 17 other listed vulnerabilities/hardening items across the same five simultaneously-released branch versions.
  • OpenSSL formally ships 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 together as part of a coordinated security-advisory release addressing 18 vulnerabilities/hardening items, including the HollowByte init_buf fix (per the OpenSSL project's release-and-advisory timeline, superseding the earlier PR-merge date).
  • Threat tracked into the TL-Intel backlog as TL-2026-1459 despite no CVE assignment, due to breadth of affected stacks and demonstrated PoC impact.
  • The public writeup notes OpenSSL 'quietly' merged the fix into the coordinated release without a dedicated CVE-style advisory naming or publicizing the flaw, meaning affected operators had no CVE identifier to track for patch prioritization between the 09 Jun release and the 17 Jul writeup.
  • Cyber Security News publishes a technical writeup naming the flaw "HollowByte," detailing the 11-byte trigger payload, the 131 KB per-connection allocation, and NGINX OOM test results, roughly five weeks after the coordinated patch release.

Sources cited for OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

Threats related to OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS

Detection coverage for TL-2026-1459

As of 2026-07-17, Threadlinqs Intelligence publishes 9 detection rule(s) for TL-2026-1459 across Splunk SPL, Microsoft KQL and Sigma, covering 16 indicator(s) of compromise. The whole corpus is readable without an account; a free account unlocks full detection query text in Splunk SPL, Microsoft KQL and Sigma; paid tiers add raw indicator values, correlation and the MCP server. Threadlinqs MCP server · View plans.

Threadlinqs Intelligence — Real-Time Threat Detection Platform

[ 0 threats ] [ 0 det ] [ CRIT: 0 ] [ HIGH: 0 ]
// threat_feed
$ sort --newest
Showing all threats

Latest Threats