Saving another 100TB of RAM
TL;DR
Cloudflare cut more than 100TB of RAM from the Pingora Backend Router (PBR), its consistent-hashing load balancer. Two changes did it: 6-byte hash entries instead of 8, and 90% fewer hashes per server. Cloudflare's math showed the extra hashes bought almost no extra balance. Past a point, 32-bit collisions made the balance worse.
The article
- PBR routes cacheable requests with consistent hashing, using the pingora-ketama crate. Some instances used 6GB on hash rings. Each server got 160 base hashes, multiplied by a disk-size weight. Features like compliance and caching repeated this across dozens of rings.
- With one hash per server and 100 servers, a server's share of the ring has a coefficient of variation near 99%. Some servers then get about double their fair load. 160 hashes per server bring that down to about 8%.
- Shrinking the server index from u32 to u16 saved nothing at first. Rust's alignment rules padded the struct back to 8 bytes. Storing hash and index in a raw 6-byte array with getters cut consistent-hashing memory by 25%.
- The author derived an exact error formula for k hashes per server, CV = sqrt((N-1)/(Nk+1)). At an effective 100,000 hashes per server, the last 90,000 bought only a 0.7% reduction in error. Simulations with 2,048 servers showed 32-bit collisions pushing error back up between 10,000 and 100,000 hashes.
- Changing the ring moves cached content. So PBR held the old and new rings in memory at once and picked one per request hash. The rollout went data center by data center to contain cache churn and origin load. It kept a rollback path that needed no redeploy.
- The compact v2 ring, faster sorting and a tunable base hash count ship in pingora-ketama behind an unadvertised cargo feature. The v1 ring is unchanged.
Why it matters
Read it if you run consistent hashing or ketama-style load balancing at scale. The original has the charts for error versus hash count and the collision simulations. It also links the full derivation, which you need to pick a hash count for your own fleet.
The discussion
A warm thread that mostly praised the write-up, then split into two arguments. The longer one asked whether software should be optimized like this at all, or whether shipping fast is the right call. The more technical one asked whether consistent hashing with huge rings was the right design to begin with.
Optimization versus shipping
The top comment welcomed expensive RAM as a push back toward lean software. Developers who lived through scarce memory said the craft was fun, but they preferred spending their time on customers. Others answered that bloated apps come from careless code, not from a tradeoff, and that budgets force better engineering. locknitpicker called most such savings bean counting, and throw10920 said almost no user cares about memory use at this level.
Yes, the challenge was fun, but I really wanted to ship it and get my product in the hands of customers. Now I can spend more time thinking about what they want and less time about what the computer wants.
The wasteful applications are generally not well reasoned about and half assed implementations. That's why they're guzzling resources
Was consistent hashing the right tool?
Several readers proposed alternatives that need no big lookup table: rendezvous or hierarchical rendezvous hashing, a tournament hash, Google's jump hash, or a weighted btree. sfink worked through a plain hash-mod-N design in public, until procaryote pointed out that hashing the servers keeps requests sticky when servers come and go. sfink then named the missing constraint: load balancers never share an exact view of which servers are up. Jump hash drew the same objection, since it can't drop a server from the middle of the list.
You hash the servers because then adding or removing a server doesn't directly affect other servers position on the ring; adding a server just takes some load from som servers.
As I recall, one of the main differences is that jump hash doesn't have a mechanism to remove targets, eg, a server dies and you don't want to route requests to it.
Who this is worth it for
Commenters noted that 100TB is about 1% of Cloudflare's fleet, which pays off only at its scale. A company whose costs are rent and staff would not bother, and would optimize its supply chain instead. A side argument asked whether the RAM was freed for AI inference. It ended with the point that CPU DRAM is a poor fit for inference.
Cloudflare is unusual in that their product is performance and their costs are primarily hardware and networking.
While it's nice to see this focus, while they highlight absolute figures, we are still talking about 1% improvement. For most other software systems, nothing worth putting effort in for.
A human-written Cloudflare post
Many readers said the post read as human after a run of Cloudflare posts they took for LLM output. One pointed to the "Hi, Mom" aside as something no LLM would write. Another ran it through the Pangram detector and got a human verdict. Others called detectors snake oil, and the original poster said Pangram had rated text they had generated as 100% human.
Tried out the first 1000 words in Pangram, and it seemed happy it was human written.
It’s refreshing to actually read English instead of Claudelish.
Silos and layered complexity
One reader asked when a company with optimizations this deep turns into silos nobody can see into. Replies blamed layers like TLS, auth and orchestration for making a single boolean hard to flip. Others said those layers exist because networks are hostile. procaryote held up PBR as a good silo: small, loosely coupled, and unaware of what its weights mean.
The trick is to keep the silos losely coupled and small enough that you can understand it reasonably quickly. A component like the one in the article is pretty good like that.
Consensus
The post was clear and well written. At Cloudflare's scale, a 1% memory cut is real money worth the engineering time.
Dissent
A minority found it overhyped. One said the math exposition was showing off and never explained why consistent hashing is used. Another summed it up as fixing a poorly designed struct. Several argued that a different algorithm would avoid storing huge hash rings at all.