# OpenSSL "HollowByte" TLS Handshake Memory-Amplification DoS (No CVE Assigned)

> Okta Red Team discovered a memory-exhaustion flaw in OpenSSL's TLS handshake buffer allocation logic where an 11-byte crafted ClientHello-stage payload forces malloc() to reserve up to 131 KB based on an unvalidated attacker-declared size, and glibc heap fragmentation prevents the OS from reclaiming memory after the connection drops. OpenSSL treated it as a hardening improvement rather than a formally numbered CVE, fixing it via incremental buffer growth in 4.0.1 with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21.

- **Published:** 2026-07-17T00:00:00Z
- **Last reviewed:** 2026-07-17T00:00:00Z
- **Canonical:** https://intel.threadlinqs.com/threat/TL-2026-1459
- **ID:** TL-2026-1459
- **Severity:** MEDIUM
- **Category:** VULNERABILITY
- **Status:** PATCHED
- **Detections:** 9 · **IOCs:** 16 (full data via the Threadlinqs MCP server — Purple tier)
- **CVEs:** CVE-2016-6309

## Description

The vulnerability, publicly nicknamed "HollowByte," lives in the TLS handshake message-reading path of OpenSSL's state machine (ssl/statem/statem_lib.c). When OpenSSL reads a handshake message header it extracts a 4-byte, peer-declared payload-length field and immediately grows the connection's init_buf to that full claimed size via grow_init_buf() -> OPENSSL_clear_realloc() -> malloc(), before a single byte of the actual payload has arrived on the wire. Because the length field is attacker-controlled and unvalidated against the bytes actually delivered, an attacker can send an 11-byte packet (the minimal TLS record + handshake header needed to declare an oversized length) and force the server to allocate up to 131 KB of heap per connection while sending little to no follow-on data.

The attack's potency comes from a second-order effect: glibc's allocator does not always return freed heap arenas to the OS when a connection is torn down, and if the attacker randomizes the declared size on each new connection, freed chunks are the wrong size to be reused for the next allocation. This produces persistent heap fragmentation that survives connection closure, so repeated low-bandwidth connection churn compounds memory pressure over time rather than being bounded by the theoretical per-connection maximum. Okta Red Team's testing showed a 1 GB RAM NGINX instance OOM-killed after roughly 547 MB of fragmented, unreclaimable memory accumulated, and a 16 GB RAM instance had 25% of system memory permanently locked up while the attack traffic also bypassed the server's configured connection-rate limits (because each attacking connection is trivially cheap and short-lived).

This is architecturally the same code family responsible for the 2016 use-after-free in large handshake-message handling (CVE-2016-6309), which originally introduced grow_init_buf() as a replacement for BUF_MEM_grow_clean(). HollowByte does not reintroduce a memory-safety bug (no UAF/overflow) — it is a resource-exhaustion / unbounded-allocation flaw (CWE-789 / CWE-400), which is why OpenSSL's security team classified it as a hardening fix rather than a CVE-worthy vulnerability: no memory corruption, only availability impact, and full remediation ships in every actively maintained branch.

The fix (OpenSSL PRs #30792, #30793, #30794) replaces immediate full-size pre-allocation with incremental buffer growth that expands the init_buf only as bytes are actually received off the wire, capping any single message to roughly 8 incremental reallocations. The change is TLS-only; DTLS was explicitly excluded from the fix due to the added complexity of datagram reassembly, meaning DTLS-terminating services built on the same OpenSSL versions may retain analogous exposure. The patch merged to OpenSSL master and the 4.0 branch on 2026-04-17 after four reviewer approvals, then was backported to the 3.0, 3.4, 3.5, and 3.6 LTS/stable branches.

Because OpenSSL is the default TLS library for NGINX, Apache httpd (mod_ssl), Node.js, Python (ssl module/requests via OpenSSL bindings), Ruby (OpenSSL gem), PHP (openssl extension), MySQL, and PostgreSQL, the practical blast radius spans essentially the entire TLS-terminating web and database tier of a typical Linux server fleet running an unpatched OpenSSL 3.0.x-3.6.x or pre-4.0.1 build. No public exploit code has been released and no in-the-wild exploitation has been observed or claimed; this was disclosed responsibly by Okta's internal red team and patched proactively across all supported branches before public write-up.

## MITRE ATT&CK

- T1595 Active Scanning
- T1592 Gather Victim Host Information
- T1596 Search Open Technical Databases
- T1587.004 Exploits
- T1588.005 Exploits
- T1190 Exploit Public-Facing Application
- T1203 Exploitation for Client Execution
- T1685 Disable or Modify Tools
- T1499 Endpoint Denial of Service
- T1499.001 OS Exhaustion Flood
- T1499.002 Service Exhaustion Flood
- T1499.003 Application Exhaustion Flood
- T1498 Network Denial of Service
- T1498.001 Direct Network Flood
- T1518 Software Discovery

## Sources

- [OpenSSL "HollowByte" Vulnerability](https://cybersecuritynews.com/openssl-hollowbyte-vulnerability/)
- [OpenSSL PR #30792 - Incremental init_buf growth for TLS handshake messages](https://github.com/openssl/openssl/pull/30792)
- [OpenSSL 4.0.1 Release](https://github.com/openssl/openssl/releases/tag/openssl-4.0.1)
- [OpenSSL Vulnerabilities Index](https://openssl-library.org/news/vulnerabilities)
- [OpenSSL Release and Advisory Timeline](https://openssl-library.org/news/timeline/)
- [OpenSSL 4.0.0 release cuts deprecated protocols and gains post-quantum support](https://www.helpnetsecurity.com/2026/04/14/openssl-4-0-0-released/)
- [Analysis of OpenSSL Large Message Size Handling Use After Free (CVE-2016-6309)](https://www.fortinet.com/blog/threat-research/analysis-of-openssl-large-message-size-handling-use-after-free-cve-2016-6309)
- [DTLS memory optimization by releasing the init_buf](https://github.com/openssl/openssl/issues/13242)

## Full data

Detection queries (Splunk SPL / Microsoft KQL / Sigma) and IOC values require the Threadlinqs MCP server (Purple tier): https://intel.threadlinqs.com/mcp

Canonical: https://intel.threadlinqs.com/threat/TL-2026-1459