How Cloudflare Found 100 Terabytes Hiding in Tiny Code Decisions

First, what is sitting inside a DNS cache?

A cached item has a key and a value. The key describes the question: the domain name, record type, whether the response was authenticated, and an internal tag. The value carries the response sections—answers, authority records, and additional records—plus metadata such as creation time, time-to-live, hit count, and extended errors.

Big Pineapple starts empty after a restart and fills as queries arrive. Once a location reaches its configured entry limit, older or less popular entries are evicted. The exact capacity varies by data centre. EDNS Client Subnet makes the storage problem harder because authoritative servers can return different answers for the same domain depending on the requesting network, requiring several cached versions of what looks like one question.

That is the first useful lesson: a cache entry is not merely a domain paired with an IP address. Real protocols carry exceptions, metadata, several record sections, and different response types. Good optimization begins by understanding which of those things change, which repeat, and which exist only because a general-purpose type permits them.

At 250 billion cache entries, wasting one byte costs more than 250GB. Cloudflare's story is what happens when tiny abstractions receive internet-scale multiplication.

The benchmark tried to resemble traffic without pretending to be traffic

Cloudflare filled test caches with randomly generated entries shaped around its production mix: 56% IPv4 A records, 25% IPv6 AAAA records, and 19% TXT records. Each entry held between one and four records. TXT values ranged from 64 to 224 bytes and stood in for the many variable-length record types outside A and AAAA.

A custom allocator wrapped Rust's system allocator and counted allocation size and frequency for each entry. The team measured insertion throughput and lookup latency alongside memory, because saving RAM by making the busiest path slower would simply move the bill somewhere else.

The authors are careful about the boundary of the test. Random inputs approximate production rather than reproduce it. Actual resident memory also depends on traffic mix, allocator state, cache occupancy, and everything else inside the process. Cloudflare therefore checked resident memory across deployed Big Pineapple instances during rollout. That combination—a controlled benchmark plus production observation—is stronger than presenting either one as the whole truth.

Optimization one: stop giving frozen data room to grow

Rust's Vec stores a pointer, a current length, and a capacity. Capacity is useful while a list is growing because it prevents an allocation on every push. A String carries the same general idea. But after a DNS response enters this cache, Cloudflare never changes it. The extra capacity field and any reserved heap space become luggage for a journey the data will never take.

Replacing growable vectors with boxed slices and strings with boxed string slices removes the capacity field. Cloudflare counted eight Vec or String fields per entry. At eight bytes saved for each field, the structural saving was 64 bytes per entry before counting unused reserved space on the heap. Across more than 250 billion entries, this stage alone removed over 15 terabytes.

This is not a trick specific to Rust. It is a question about promises. A growable collection promises future mutation, and the machine stores information to keep that promise. If the application knows the value is immutable, choosing an immutable representation lets the data structure tell the same truth as the program.

Optimizations two and three: remove pointers and stop repeating names

DNS responses traditionally separate answer, authority, and additional records. Big Pineapple initially mirrored that structure with three lists. Cloudflare combined them into one list and stored two offsets marking where the later sections begin. Because section record counts fit in 16 bits, each offset needs two bytes. Removing two boxed lists, each carrying an eight-byte pointer and eight-byte length, and adding two two-byte offsets saved 28 bytes per entry.

The team also packed several booleans into one bit field. Small changes can save more than the visible field sizes because Rust aligns values and inserts padding. Remove or rearrange one field, and an empty alignment gap may disappear with it. This is the sort of result that makes memory-layout diagrams more useful than intuition.

The third change concerned record owners. A DNS record normally stores the domain it belongs to, but in most cached responses that owner is identical to the queried domain already present in the cache key. Cloudflare made the owner optional: when it matches, the record stores nothing and response construction restores the name from the key. When a CNAME or another case introduces a different owner, the full name remains available.

The optimization makes each record less self-contained, which would be a poor trade in some systems. Here, every lookup already has the cache key. Reusing context that is guaranteed to be present is cheaper than preserving convenient duplication billions of times.

Optimization four: one rare DNS record made every enum enormous

Rust enums reserve enough inline space for their largest variant. Cloudflare's record-data enum included small A records using four bytes, AAAA records using 16, and much larger types such as NAPTR. NAPTR required 136 bytes; after a variant tag and alignment, every enum value occupied 144 bytes regardless of which record it actually contained.

A and AAAA records account for more than 80% of Cloudflare's traffic. That meant the common four- and 16-byte cases repeatedly paid for the possibility that they might have been a rare 136-byte case. Boxing the larger variants reduced the inline enum to 24 bytes. Cloudflare reports a 120-byte saving for each A or AAAA record, while variable-sized records received heap allocations closer to their actual size.

Boxing was not free. Allocators round requests into size classes, so a 40-byte value may occupy a 48-byte bin. Separate heap allocations also scatter related data, making the CPU follow pointers and fetch more cache lines. Rare NAPTR records became slightly more expensive. The choice still won because common records dominated the workload, but the tradeoff is a reminder that averages should be weighted by real frequency.

Optimization five: meet the wire format halfway

The final design moved away from parsed enum variants and stored record data as a contiguous byte buffer. Each record is encoded with a two-byte length prefix followed by its raw DNS bytes. This removes per-variant enum padding and the individual heap allocations introduced by boxing, while improving memory locality.

Cloudflare did not store the entire DNS response exactly as it travels over the network. Full wire-format messages create complications around DNSSEC selection, per-client fields, and name compression. Instead, the hybrid representation keeps the cache entry's control data structured and stores only record payloads as bytes.

For A, AAAA, TXT, and DNSSEC records, response construction can copy encoded bytes directly instead of serializing fields again. Records containing domain names—such as CNAME, NS, MX, and SOA—still need parsing so the outgoing message can apply DNS name compression. Sequential iteration is also required, but record counts per entry are small enough that Cloudflare found the cost negligible.

A reusable scratch buffer handles serialization during insertion, then one precisely sized boxed byte slice receives the result. This replaced many allocations with one. Cloudflare attributes a 13% insertion-throughput improvement to this change alone and a 5% lookup-latency improvement to direct copying and better locality.

The final numbers are bigger than the cleverness

Across all five changes, benchmarked net memory per entry fell from 953 bytes to 420 bytes, a 56% reduction. Allocated memory fell from about 1.1 kilobytes to 461 bytes, or 58%. Insert throughput increased from 625,000 to 893,000 entries per second. Lookup latency dropped from 828 nanoseconds to 670 nanoseconds.

Production resident memory moved less dramatically because the cache is only part of each process. At the 99th percentile, instance memory fell from 9.3GB to 5.3GB, a 43% reduction. At the 90th percentile, it fell from 6.5GB to 3.8GB, a 42% reduction. Rollout began on May 18 and finished across services on July 6, with memory descending in steps as releases landed and restarted caches refilled.

Cloudflare estimates that the settled fleet uses roughly 100TB less working-set memory. The company plans to spend some of that saving on a larger cache rather than simply running a smaller one. More retained answers should improve hit rates and reduce questions sent upstream. Efficiency, in other words, becomes capacity.

What this pineapple teaches beyond DNS

The fun headline is 100 terabytes. The more transferable lesson is that performance work often begins with data modelling rather than a heroic algorithm. Cloudflare asked whether values truly grow, whether lists need separate ownership, whether repeated information can be inferred, and whether a general representation makes common cases pay for rare ones.

Scale changes what counts as worth fixing. On a laptop, 64 bytes is beneath notice. Multiplied by 250 billion entries, it becomes more than 15TB. One byte becomes 250GB. A pointer stops being punctuation and becomes infrastructure spending.

There is also a pleasant reversal in the result. Engineers are often taught to expect a trade between memory and speed. Here, less memory meant fewer allocations, denser data, better CPU-cache locality, less serialization, faster insertion, and faster lookup. The machine did less work because the representation carried less unnecessary possibility.

None of this means every program should replace vectors with boxes or parsed values with bytes. Big Pineapple has an unusual workload: immutable entries, tiny record counts, immense cardinality, predictable hot paths, and a known traffic distribution. Copying the answer without copying the constraints is how optimization folklore begins. The real pattern is to measure your own data, then make its representation honest.