CrowdStrike meets Murphy's Law: Anything that can go wrong will
- Reference: 1722018972
- News link: https://www.theregister.co.uk/2024/07/26/crowdstrike_meets_murphys_law/
- Source link:
As a veteran tech journalist, I've seen my fair share of software snafus. Heck, I went hand-to-hand with the grandpa of all network blow-ups – the [1]Morris Worm – in 1988 when I was a sysadmin. Even so, I can't help but marvel at the sheer scale and impact of this blunder. [2]CrowdStrike , a company valued at over $70 billion and trusted by countless organizations to protect their digital assets, inadvertently became the source of one of the largest IT outages in history.
The fallout from this debacle was staggering – thousands of flights canceled, healthcare services disrupted, and 911 systems knocked offline. It's a stark reminder of how deeply intertwined our digital infrastructure has become and how vulnerable it can be to a single point of failure.
[3]
Let's break down the cascade of errors that led to this fiasco.
[4]
[5]
In the beginning, Microsoft enabled CrowdStrike's Falcon security software to run at the zero level of the Windows kernel. [6]Any problem at this low level will likely cause a Blue Screen of Death (BSOD). Meanwhile, [7]Microsoft reportedly wants to blame the European Commission – no, really – for requiring it to grant third-party software vendors this level of access.
You know, I think with all of Microsoft developers and lawyers, they could come up with a better, legal way to avoid this kind of foul-up and let software companies compete equally. It's not rocket science.
[8]
Microsoft doesn't want any of the blame, but it deserves some of it. For far too long, we've placed too many vital IT eggs in the Windows basket. When that basket falls, so does much of the economy.
Returning to CrowdStrike, the company claims a "logic error" in a routine sensor configuration update caused the meltdown. But for a company of CrowdStrike's caliber, such a fundamental mistake is inexcusable. This wasn't some obscure edge case – it was a critical failure in its core functionality.
It wasn't even a code problem. This wasn't a software update per se. The villain of this piece was a Falcon configuration file called a channel file. One simple file containing what should have contained data to update a security setting ended up causing a cascade of one BSOD after another.
[9]The graying open source community needs fresh blood
[10]Windows: Insecure by design
[11]Let's take a look at Oracle's love and hate relationship with open source software
[12]Where do Terraform and OpenTofu go from here?
How did such a catastrophic bug pass quality assurance? CrowdStrike admitted: "Due to a bug in the Content Validator, one of the two [13]Template Instances passed validation despite containing problematic content data [and] were deployed into production." When your software has deep hooks into millions of Windows systems, your testing should be bulletproof. Clearly, CrowdStrike's testing protocols need a massive overhaul.
We also now know, as security expert Kevin Beaumont pointed out on Mastodon: "The key takeaway – [14]channel updates are currently deployed globally, instantly ." I always send major patches to all my customers simultaneously and wait to see what happens next. Doesn't everyone? Who are these people, and why does anyone let them do security work?
[15]
There's a simple concept called [16]canary testing . You may have heard of it. Like the proverbial canary in a coal mine, you first test whether a new space – or program – is safe by trying it on a canary – or a small group of users – and then, if all's well, let everyone else in.
Let's not forget that CrowdStrike's initial response was slow and inadequate. Users were left scrambling for answers while critical infrastructure faltered. Even today, almost a week later, I still have friends having [17]trouble with their Delta flights .
This serves as a sobering wake-up call for the rest of us in the tech industry. As we rush to secure our systems against external threats, we must not overlook the potential for self-inflicted wounds. Rigorous testing, fail-safe mechanisms, and a healthy dose of humility are essential when dealing with critical systems.
In the end, CrowdStrike's Windows fiasco is a textbook example of Murphy's Law in action – anything that can go wrong will go wrong. It's a painful lesson but one that we would all do well to learn from. After all, in cybersecurity, your next big threat might just be an update away. ®
Get our [18]Tech Resources
[1] https://www.theregister.com/2013/11/04/morris_worm_anniversary/
[2] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/patches&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqQchzYv5GyiTrJm84KC4AAAAU8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/patches&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqQchzYv5GyiTrJm84KC4AAAAU8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/patches&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqQchzYv5GyiTrJm84KC4AAAAU8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/07/23/crowdstrike_failure_shows_need_for/
[7] https://www.theregister.com/2024/07/22/windows_crowdstrike_kernel_eu/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/patches&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqQchzYv5GyiTrJm84KC4AAAAU8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/07/15/opinion_open_source_attract_devs/
[10] https://www.theregister.com/2024/06/28/windows_insecure_by_design/
[11] https://www.theregister.com/2024/06/14/oracles_love_and_hate_relationship/
[12] https://www.theregister.com/2024/05/24/opinion_column_terraform/
[13] https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
[14] https://cyberplace.social/@GossiTheDog/112840365900702754
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/patches&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqQchzYv5GyiTrJm84KC4AAAAU8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[16] https://www.techtarget.com/whatis/definition/canary-canary-testing
[17] https://www.theregister.com/2024/07/24/transport_department_delta_probe/
[18] https://whitepapers.theregister.com/
There but for the grace of God go all of us
People, including managers, do get careless
About the first lesson I learnt when I first became a developer, was not to be careless, which covers everthing from TESTING your code adequately to having your designs (at all levels from subroutines to complete systems) criticised by your peers and all levels of users before you even think of implementation. And you keep all the documentation written before and during system development and modification. Fortunately I had all that stuff drummed into me while I was still writing assembler and COBOL modules designed by others in an ICL service bureau. This approach has always served me well at all levels of the system maintenance and development.
I'm also aware that there are still many developers and and code shops where this doesn't happen: special props to Smith's Industries where the systems analysts binned binned documentation once anything had gone live! - but I never dreamt that that this level sloppiness was still commonplace in this century, or that anybody, anywhere, was as sloppy and incompetent as CrowdStrive has turned out to be - except,of course, for Microsloth.
.
The collapse of what little engineering culture existed in IT
It's always been a stretch to apply to the word "engineering" to IT (many of us who tried very hard seem to be found in these forums), but over the last decade or so we've seen the continuous release of new levels of farcical.
Virtually no IT vendors care about stability anymore - they're too busy propping up the house of cards that is IT corporate stock P/E ratios. Upgrade constantly to provide additional "value" to customers (completely ignoring the costs and risks of every change, because that's the customers' problem amirite?>), because if you don't upgrade constantly you can't justify the recurring revenue required to justify the IPO / acquisition / trading price.
Change is not inherently bad, but change is cost and change is risk and if you're adding cost and risk you need to pair it with a corresponding amount of value. This has always been sketchy, and so customers stopped automatically upgrading because the value wasn't there. So now upgrades are forced on us. This will not end well.
Users were left scrambling for answers while critical infrastructure faltered
I have seen this happening for years on a smaller scale within companies, mostly at the level of server applications, despite warning of the stupidity of the concept of continuous deployment. In almost every case, development and support were kept apart as seperate worlds, meaning that when the shit hit the fan, poor lines of communication slowed down the response way more than should have been the case.
I have come to the conclusion that whoever invented / pushed continuous deployment has never worked in IT support for a day of their lives, otherwise they would have understood the risks of that idea on a deep and fundamental level.
Re: Users were left scrambling for answers while critical infrastructure faltered
Proper Continuous Deployment depends on:
(1) A comprehensive automated test suite;
(2) A pipeline which never deploys anything unless the entire test suite passes;
(3) Phased deployment (i.e. canaries)
(4) Instrumentation so you can see no unexpected changes in the behaviour of the canaries
If you're not doing these, you're not doing CD, you're doing crash-and-burn.
I spent the afternoon watching the ticket system fill with alerts saying that crowdstrike had quarantined the dell update utility on every computer at one of our big clients.
The migration of the IT estate from us to the team who mandated that crap was halted because of huge issues caused by CrowdStrike... before last weeks outage...