Delta Sues CrowdStrike Over Software Update That Prompted Mass Flight Disruptions (reuters.com)
- Reference: 0175325651
- News link: https://yro.slashdot.org/story/24/10/26/0035244/delta-sues-crowdstrike-over-software-update-that-prompted-mass-flight-disruptions
- Source link: https://www.reuters.com/legal/delta-sues-crowdstrike-over-software-update-that-prompted-mass-flight-2024-10-25/
> Delta Air Lines on Friday [1]sued cybersecurity firm CrowdStrike in a Georgia state court after a [2]global outage in July caused mass flight cancellations, disrupted travel plans of 1.3 million customers and cost the carrier [3]more than $500 million . Delta's lawsuit filed in Fulton County Superior Court called the faulty software update from CrowdStrike "catastrophic" and said the firm "forced untested and faulty updates to its customers, causing more than 8.5 million Microsoft Windows-based computers around the world to crash." [...]
>
> Delta, which has purchased CrowdStrike products since 2022, said the outage forced it to cancel 7,000 flights, impacting 1.3 million passengers over five days. "If CrowdStrike had tested the faulty update on even one computer before deployment, the computer would have crashed," Delta's lawsuit says. "Because the faulty update could not be removed remotely, CrowdStrike crippled Delta's business and created immense delays for Delta customers." Delta said that as part of its IT-planning and infrastructure, it has invested billions of dollars "in licensing and building some of the best technology solutions in the airline industry."
[1] https://www.reuters.com/legal/delta-sues-crowdstrike-over-software-update-that-prompted-mass-flight-2024-10-25/
[2] https://it.slashdot.org/story/24/07/19/0943232/global-it-outage-linked-to-crowdstrike-update-disrupts-businesses
[3] https://slashdot.org/story/24/07/31/1325235/delta-ceo-says-crowdstrike-microsoft-outage-cost-the-airline-500-million
The Person I Would Hate to Be (Score:3, Interesting)
I would not like to be the person who authored the bad patch at CrowdStrike. Not one bit. I imagine their name will come out at some point during the litigation of this case, and at that point their career and any semblance of a decent life will be over. Never mind that they worked for a company that clearly had just about the worst release process, if you can call it that, on the planet. This situation calls for a scapegoat and that lone coder will be it.
Re: (Score:2)
It's possible that the defendant's lawyers may try to cast shadow on the coder, but the court is likely to consider all aspects of the event, including very poor processes by the company at large. After all, any revision and release processes need to take into account problematic or dangerous updates, and so while you might be able to blame a coder or a code group of buggering it up, the update was green lit, suggesting there are severe systemic issues.
also the lack of update control for that part that (Score:2)
also the lack of update control for that part that crashed.
Re:The Person I Would Hate to Be (Score:4, Insightful)
This guy's name may come out, yes, but I think you are exaggerating the industry's perception of who was responsible for this disaster. This was caused by a systemic problem linked to CrowdStrike's inadequate integration testing, not just the result of one developer's mistake. Based on the information available, most people should understand that there exists no reason to blacklist him from the profession, as you seem to believe.
Re: (Score:2)
> I imagine their name will come out at some point during the litigation of this case, and at that point their career and any semblance of a decent life will be over.
Only truly stupid companies wouldn't hire the person. A person is incapable of making a mistake like this. A mistake like this requires a fundamental lack of quality control systems and processes. There is a name you can directly blame. George Kurtz, CEO and Founder of a company who demonstrated that they don't take any basic QA/QC precautions.
Yes, test it - before putting it into service. (Score:2, Insightful)
I'm talking to you, Delta.
Re:Yes, test it - before putting it into service. (Score:5, Informative)
IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
Re: (Score:2)
[1]https://www.crn.com/news/secur... [crn.com]
[1] https://www.crn.com/news/security/2024/crowdstrike-has-been-doing-updates-this-way-for-many-years-what-went-wrong
Re: (Score:1)
> IIRC part of this issue was that Clownstroke does not let you do controlled updates like that.
That just pushes the fault back a step: Delta should never have gotten into such an arrangement.
Re:Yes, test it - before putting it into service. (Score:4, Informative)
> I'm talking to you, Delta.
That's not how Crowdstrike channel updates work. Not for Delta not for anyone else. That said there was someone who should have tested something before putting it in service, but you're addressing the wrong side of the court room for this.
Valid if it was just Delta that shit itself. (Score:2)
But it was everyone . Crowdstrike was clearly the problem.
Re: (Score:2)
(This was meant in reply to the comment above saying Delta should have tested it)
Re: (Score:2)
Everyone should be testing before deploying. That fact that it hit so many in modern times shows up the bad practice creeping in.
Re: (Score:3)
[1]https://www.crn.com/news/secur... [crn.com]
Bad practices that vendors force on you.
[1] https://www.crn.com/news/security/2024/crowdstrike-has-been-doing-updates-this-way-for-many-years-what-went-wrong
Re: (Score:2)
Testing what? It's not some niche application that fell apart under narrow conditions. All the customers did was "draw water from the well," and you can't expect people to test the well every time they dip a bucket.
EULA (Score:2)
One could guess that the EULA says the software does not have to work and you use it at your own risk.
[1]https://www.crowdstrike.com/en... [crowdstrike.com]
[1] https://www.crowdstrike.com/en-us/software-terms-of-use/
Re: (Score:3)
Which is great! We can strike EULA's out of law, as they should be. A contract you can't negotiate isn't a contract.
Re: (Score:2)
A Contact you can't negotiate is called a "contract of adhesion"
"For a contract to be treated as a contract of adhesion, it must be presented on a standard form on a "take it or leave it" basis, and give one party no ability to negotiate because of their unequal bargaining position."
https://en.wikipedia.org/wiki/Standard_form_contract#Contracts_of_adhesion
This is what we see for most consumer contracts.
That said, if two companies are large enough they will negotiate a custom contract. Usually the largest co
Re: (Score:2)
> 6. No Warranty.
> 6.1 Disclaimer. THE SOFTWARE AND ALL OTHER CROWDSTRIKE OFFERINGS ARE PROVIDED “AS-IS” AND WITHOUT WARRANTY OF ANY KIND. CROWDSTRIKE AND ITS AFFILIATES DISCLAIM ALL OTHER WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE.
Sounds fairly straight forward. Did Delta not read the agreement?
Make Microsoft accountable too. (Score:2)
Forced reboots are why many people use unsupported operating systems as they no longer reboot for updates disrupting workflows.
Re: (Score:2)
> Forced reboots are why many people use unsupported operating systems as they no longer reboot for updates disrupting workflows.
Microsoft doesn't force any reboots. Microsoft displays a warning on the screen that during the next reboot or shutdown an update will apply. Microsoft provides an offline timer to apply updates out of hours when the PC isn't use. Microsoft provide APIs that allow state recovery on reboot, as well as APIs to lock out reboots preventing work being lost.
If you are losing work you definitely have someone to blame, but it's not Microsoft. Start with your own IT department (yes ours forces reboots on our machine
Re: (Score:2)
> Of course, Crowdstrike does not have that kind of money lying around
Crowdstrike is worth $75billion. Not only do they have $4bn in free cash laying around, they could very easily raise more capital.
Re: (Score:2)
Their annual recurring (I.E. subscription) revenue in 2023 was almost 3.5 billion, with a gross profit margin of 75-80% on that income.
They could (maybe with indigestion) stomach a 5.5 billion loss after several years similar to 2023 in recent history.
backups (Score:2)
How did that Clownstrike failure sneak into Delta's backups/snapshots? Asking for a friend.
Problem is lack of transparency? (Score:2)
Maybe the worst thing is that CS's practices came AS A COMPLETE SURPRISE to everyone,
How many other companies out there are you currently relying on? What are their practices?
Delta, like many, took a decision to trust CS and whatever claims CS made about their products. Well, trust works for the good guys. But it's the bad guys that are the problem. And as the old Marx Brothers joke goes, if you can fake that then you've got it made. The bad companies are deceiving you into blindly trusting them.
The only an
The should have read the fine print (Score:2)
Damages never more than the software costs.
Re: The should have read the fine print (Score:2)
These exclusions donâ(TM)t apply to things like gross negligence or fraud, and whether or not the event was one of those is a question of fact for a jury to decide.
Re: (Score:2)
The sticky parts here:
Delta is demanding from its contractor reparations for damages that it's refusing its customers. When they reject claims for which they have responsibility, how can they go after their contractors on the same basis?
Delta is not the only airline to use CrowdStrike, and they all had outages on July 19. Delta, however, is the only one that couldn't recover until July 24.
Crowdstrike's test and deployment processes to me look like gross negligence: their business is having companies ent