Cloudflare global outage, 2019
On July 2, 2019, a single regular expression pushed out as part of a Cloudflare firewall rule update brought down a large share of the internet’s traffic for 27 minutes. Cloudflare sits in front of millions of websites, and for those 27 minutes a big fraction of the requests passing through its network failed with HTTP 502 errors — the company’s own SRE team measured roughly 80% of traffic lost at its peak.
What it looked like at the time
Minutes after the rule was deployed, CPU usage on Cloudflare’s edge machines hit 100% worldwide and every core serving traffic stalled. Websites behind Cloudflare returned 502 Bad Gateway errors. The Cloudflare dashboard and API — which run through the same edge — also went down, briefly making the incident harder to diagnose and even harder to communicate about.
Root cause
The rule contained a regular expression with a pattern that reduced to the shape .*.*=.* — nested, overlapping wildcards. The regex engine in use (PCRE) allows catastrophic backtracking: on certain inputs it re-tries the same combinations exponentially, with no time limit. One specific real-world request sent the engine into effectively unbounded backtracking, and because the rule ran on every edge server, one bad pattern froze the whole network. A staging step that would normally have caught the CPU spike was skipped during that particular rollout.
How it was found and fixed
Engineers identified the WAF rule as the cause within about eighteen minutes and executed a global “terminate” of the firewall at 14:07 UTC; traffic and CPU recovered by 14:09. The WAF was re-enabled two hours later after testing. Afterwards, Cloudflare re-added a CPU-protection guard that had been accidentally dropped, audited all 3,868 of its managed rules for backtracking risk, began migrating to regex engines with linear-time guarantees, and made staged rollouts mandatory for firewall changes.
The lasting lesson
A regex is a program, and a naive one can run forever. Anything that evaluates user-supplied input with a regex needs a runtime complexity guard or a linear-time engine. And a deployment process that sometimes skips its safety checks will eventually skip them at the worst possible moment.
Practice the skill
A debugging challenge that works the same muscle:
- The Search That Guesses — a debugging challenge in the same discipline
- The Ghost Update — an async race condition to hunt down