How a Typo Took Down AWS S3 (and Half the Internet)
Oct 27, 2026
On February 28, 2017, the internet broke — not all of it, but enough. Slack stopped loading, Trello went blank, Quora, Business Insider, Giphy, and a huge chunk of the web vanished. The cause wasn’t a cyberattack, a fiber cut, or a hurricane. It was a typo.
This is the story of the AWS S3 outage in us-east-1: the day a single wrong character in a command line took down a region and revealed the dependency loop nobody had planned for. We cover S3’s index and placement subsystems, the routine billing playbook that removed far more capacity than intended, the forced full restart that took hours, the health dashboard that couldn’t report health because it depended on S3, the health-check storm that cascaded across AWS services, and the post-mortem fixes AWS committed to afterward.
Chapters
00:00 — The internet breaks
00:32 — Context: S3 and us-east-1
01:23 — The typo: removing too much capacity
01:59 — Forced restart and untested recovery at scale
02:27 — The dependency loop: status page down
02:53 — Health-check storm and cascading failure
03:23 — Recovery timeline: index, then placement
03:52 — Post-mortem and AWS fixes
04:22 — Lessons: routine isn’t routine, test recovery, avoid dependency loops
05:16 — Conclusion: one character between working and broken
Key lessons: A routine command is never truly routine. Test your recovery paths, not just your happy paths. If your monitoring lives inside the thing you’re monitoring, you’re blind when it matters most.
Show More Show Less 