The headlines following the fiasco would make a good tv series.
The CrowdStrike incident takes down the internet. Microsoft counted around 8.5 million Windows devices rendered unusable by the faulty update, which it calls less than one percent of all Windows machines. The official CrowdStrike RCA explains why: a new IPC Template Type defined 21 input fields, the sensor supplied only 20, and the first Template Instance that actually checked the 21st field sent the Content Interpreter reading out of bounds. Microsoft’s analysis of the crash dumps confirms an out-of-bounds read in the CSagent.sys driver and suggests vendors move content parsing out of the kernel into user mode, where a crash is recoverable.
CrowdStrike then said sorry with a $10 Uber Eats gift card, and TechCrunch reports that some recipients found it cancelled because Uber flagged it as fraud over the high usage. Ten dollars, and it still bounced. Delta hired David Boies to seek damages from CrowdStrike and Microsoft, and Microsoft replied that Delta’s ancient IT explains the long outage. The parody site ClownStrike got a trademark infringement notice from CrowdStrike’s representatives, called it protected noncommercial parody and moved away from Cloudflare instead of going offline. And for the season finale, CrowdStrike showed up at DEF CON to accept the Pwnie Award for “most epic fail”.
Some noteworthy additions to add some important historical facts:
The CrowdStrike agent caused kernel panics on Linux a few months earlier. Red Hat documented a kernel panic on RHEL 9.4 caused by the falcon-sensor process loading an eBPF program, and Rocky Linux users reported servers freezing after the 9.4 upgrade in May 2024 until they switched Falcon to its kernel backend.
In 2010, a defective McAfee update caused a worldwide meltdown of XP PCs. CrowdStrike’s CEO Kurtz was CTO of McAfee in 2010.
Obviously, other companies have had incidents too. For example, Google shared results of postmortem analysis in The Site Reliability Workbook.
One pending question is liability for software companies.