CrowdStrike unhappy about Delta's 'litigation threat,' claims airline refused 'free on-site help'
- Reference: 1722861131
- News link: https://www.theregister.co.uk/2024/08/05/crowdstrike_is_not_at_all/
- Source link:
That's according to a letter, seen by The Reg and sent to [1]David Boies , partner at the law firm Delta hired to investigate the airline's legal options after it struggled more than most to bring its systems back online, leading to a sprawling list of flight cancellations.
The Falcon vendor reiterated its apology to Delta and the wider customer base. It then went on to remind Boies, known for his work as special counsel during the 1990s US antitrust trial against Microsoft, that it had been proactive in reaching out to Delta, offering support to the airline "within hours" of the incident unfolding.
[2]
"Delta's public threat of litigation distracts from this work and has contributed to a misleading narrative that CrowdStrike is responsible for Delta's IT decisions and response to the outage," the letter reads.
[3]
[4]
"Should Delta pursue this path, Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions – swiftly, transparently, and constructively – while Delta did not."
CrowdStrike's lawyer, Michael B. Carlinsky, then poked the bear further. He said that among other things, in this hypothetical trial Delta would also need to explain why it took so much longer than competitors to recover from the same issue, why it refused the free on-site help CrowdStrike offered – the support that led to faster recovery times than Delta's, and the operational resiliency of its IT infrastructure.
[5]
Before demanding a swathe of data to be preserved by Delta should the "unfortunate" litigation move forward, Carlinsky said CrowdStrike would "respond aggressively" to protect its customers, employees, and shareholders.
"CrowdStrike's focus remains on its customers, including Delta," the letter went on to say. "CrowdStrike hopes Delta reconsiders its approach and agrees to work cooperatively with CrowdStrike going forward, as the two sides historically have done."
The Register contacted Delta for input, which pointed readers to an [6]interview with its CEO Ed Bastian outlining what he characterized as a "half a billion dollar" damage to the business over five days.
[7]
A Crowdstrike spokesperson told The Register : "The letter speaks for itself. We have expressed our regret and apologies to all of our customers for this incident and the disruption that resulted. Public posturing about potentially bringing a meritless lawsuit against CrowdStrike as a long-time partner is not constructive to any party. We hope that Delta will agree to work cooperatively to find a resolution."
Per our reporting [8]last week of Delta's plans to explore litigation, the airline was also looking into potentially bringing legal action against Microsoft too as systems running on its Windows OS were exclusively affected by the incident.
It made the appointment of David Boies – named partner at the New York firm Boies Schiller Flexner – even more compelling, given the lawyer's history in taking on Microsoft and winning.
Boies was special trial counsel in the Department of Justice's 1998 [9]antitrust case against Microsoft which two years later resulted in Microsoft being [10]found guilty on most charges related to bundling Internet Explorer with Windows, before [11]settling on appeals in 2001.
He has also represented various [12]other Microsoft opponents too, as well as numerous high-profile clients from the [13]tech sphere and beyond.
As things stand
The vast majority of CrowdStrike customers are now back online and fully recovered, although the global mega outage was still being fingered for lingering issues [14]as recently as last week .
Some users took longer to recover than others because booting into safe mode or trying to work through some of the other early recovery options on a Windows device required inputting their Bitlocker recovery key. BitLocker is Microsoft's encryption tool, which makes a Windows device's storage inaccessible without the key.
As for Crowdstrike, the security shop is now battling other issues, namely its share price the value of which continues to fall according to premarket figures today.
Shares are trading at $217.89 which is a hefty tumble compared to the price exactly a month ago – $389.68.
This is understandably not very good news for shareholders, some of whom are assembling to sue the company over its approach to rolling out updates.
The Plymouth County Retirement Association pension fund brought federal legal action against CrowdStrike [15]last week on behalf of savers who held shares in the company between 29 November, 2023, and 29 July this year.
[16]The cybersecurity QA trifecta of fail that may burn down the world
[17]Microsoft whiz dishes the dirt on the Blue Screen Of Death's colorful past
[18]Too late now for canary test updates, says pension fund suing CrowdStrike
[19]DigiCert gives unlucky folks 24 hours to replace doomed certificates after code blunder
The lawsuit alleges CrowdStrike had not properly deployed canary testing before a phased rollout of updates as a means of preventing similar IT calamities in the future, and alleges this contributed directly led to the global outages in July.
"Since the CrowdStrike outage, publicly revealed evidence indicates that CrowdStrike was taking insufficient precautions regarding such updates," the lawsuit
[20]PDF
alleges."For instance, CrowdStrike has promised to take remedial measures to ensure that such a crash does not happen again, including implementing a so-called canary deployment of such updates, meaning a progressive rollout that starts with a subset of users.
"This indicates CrowdStrike was not taking such measures prior to the CrowdStrike outage."
Responding to the development, CrowdStrike told us: "We believe this case lacks merit and we will vigorously defend the company." ®
Get our [21]Tech Resources
[1] https://www.theregister.com/2024/07/30/crowdstrike_delta_microsoft_lawsuit/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZrD3IbabTtlU84sxn3PxGQAAAI4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZrD3IbabTtlU84sxn3PxGQAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZrD3IbabTtlU84sxn3PxGQAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZrD3IbabTtlU84sxn3PxGQAAAI4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.cnbc.com/video/2024/07/31/delta-air-lines-ceo-on-crowdstrike-outage-cost-us-half-a-billion-dollars-in-five-days.html
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZrD3IbabTtlU84sxn3PxGQAAAI4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/30/crowdstrike_delta_microsoft_lawsuit/
[9] https://www.theregister.com/1998/10/20/microsoft_on_trial/
[10] https://www.theregister.com/2000/04/04/judge_finds_against_ms/
[11] https://www.theregister.com/2001/11/02/doj_to_cut_ms_sellout/
[12] https://www.theregister.com/2011/12/01/barnes_noble_microsoft_legal_nemesis/
[13] https://www.theregister.com/2016/04/26/theranos_lumbers_on/
[14] https://www.nytimes.com/2024/08/02/style/mrbeast-beast-games-competition-show.html
[15] https://www.theregister.com/2024/08/01/crowdstrike_lawsuit/
[16] https://www.theregister.com/2024/08/05/opinion_ml_social_media_cybersecurity/
[17] https://www.theregister.com/2024/08/02/who_wrote_windows_bsod/
[18] https://www.theregister.com/2024/08/01/crowdstrike_lawsuit/
[19] https://www.theregister.com/2024/07/31/digicert_certificates_revoked/
[20] https://regmedia.co.uk/2024/08/01/1_24_cv_00857.pdf
[21] https://whitepapers.theregister.com/
Yeah, this is not a good look for Crowdstrike. The perception I get from them is "we messed up, and that's your problem".
We don't know what "on-site help" they offered, or why Delta turned it down - maybe they were wanting to send someone without the proper security clearance into areas that they wouldn't legally be allowed to go? I can imagine that being a legitimate reason for turning down the "help" as not fit for purpose.
"The perception I get from them is "we messed up, and that's your problem"."
Hasn't that been Microsoft's moto since it's creation ? Ans amazingly they are still here. Crowdstrike probably doens't have quite the same funding though..
The article reads more like a lawyers wankfest than an actual resolution to a real world problem.
"Should Delta pursue this path, Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions – swiftly, transparently, and constructively – while Delta did not."
Er, nope.
How about: "Should Delta pursue this path, CrowdStrike will have to explain to the jury why CrowdStrike took no responsibility for its update and clearly didn't test it on even a single hardware platform before release, as evidenced by the diversity of hardware platforms that failed."
It's also hard to see how this isn't gross misconduct, sending something out without testing... and apparently a repeat of what they did for a couple of Linux distros only weeks before.
It also reminds me of when a garage bodged a service on my car leaving it in a dangerous state and then offered me a free service in compensation.... and then were surprised when I didn't ever want to let them near my car again!!
So their argument is thay they're only liable for the first few hours?
Nice of them to capitulate so early in the case.
Oh, I don't think they'll ever admit to liability. They will try to demonstrate that they did what the law required of them which is to provides information and updates to mitigate the situation. If they can do this, they're probably home and dry and I'm sure they'll have a few "expert" witnesses from other airlines who will agree that it worked. Delta is going to have a hell of a job to show that its problems, which were much worse than other companies, were deliberately caused by Crowdstrike and that no mitigation was offered.
Though, personally, I'd love to see the lawyers on both sides forced to fight to the death over the matter, I don't see this being settled in court.
Crowdstrife's PR dept are...
Probably sitting on a top floor ledge about now...
Presumably this Carlinsky lizard gets paid in advance.
"[...] Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions – swiftly, transparently, and constructively – while Delta did not."
Oh, oh, let me try! Ahem. "Dear public, Delta shareholders and members of the Jury. The reason why Delta did not 'take responsibility' for its actions is because it was not Delta's actions that caused the disaster. Delta _have not performed_ any actions that they need to take responsibility for. It was CrowdStrike's actions, and CrowdStrike's actions alone, which caused the damages to my client, and so it is CrowdStrike, and CrowdStrike alone, which must take responsibility for this disaster."
See? Easy!
See, it was Delta's fault because they installed the CrowdStrike software in the first place!
I remember listening to a presentation by an IT guy at one of the world's largest online travel reservation companies. Their allowance for downtime to do upgrades of the production systems was 15 minutes. *Per year*. And they knew exactly how much it would cost them per minute if the downtime exceeded that.
CrowdStrike would "respond aggressively" to protect its customers, employees, and shareholders
Just not in that order....
I would be extraordinarily suprised if the contract between delta and crowdstrike doesn't limit liability. And generally in business to business contracts, limiting liability is enforcible. (and elsewhere it has been pointed out that crowdstrikes default contract says damages are limited to cost of services)
I would be very surprised if the contract actually specifies what is required re testing etc in sufficient detail not to end up with it being dealt with via existing contractual terms. Delta might have negotiated higher penalty clauses, but I doubt they are remotely close to the claimed losses.
Shareholders have a much stronger claim, with a big chunk being they don't have a contract defining the penalty if crowdstrike mess up...
Blame where blame's due
There's a problem on both sides here.
1, vendor releases software without testing it appropriately - that's their problem and they need to address that
2, customer installs software and deploys it to production without testing it appropriately - that's their problem and they need to address that
Of all the people on this site, how many of you have policies that allow for untested software to be deployed into production? And even further, across the entire estate?
I was somewhat incredulous that so many organisations were impacted by this.
Their end-user license will be clear that they do not warrant the software to be bug free... there's a reason they say that.
Re: Blame where blame's due
"customer installs software and deploys it to production without testing it appropriately - that's their problem and they need to address that"
Your downvotes are probably from people who work at companies that DON'T pre stage updates before deployments
Or have dev, stage and prod enviroments
Or have documented change management processes, including roll back
Crowdstrike and Microsoft are responsible for this. But, Delta have to look inwardly at it's own change management processes
The last "help" you want
"Hello, our software just nuked your entire company through a botched update that we couldn't be arsed to test. Now we'd like to have our people put their greasy hands all over your machines, directly."
Am I the only person who would respond to this with "No, thanks, we've had enough of your 'help' already."?
Nope
If you smash my window/s, I'm going to expect you to make me good, even if you do help in picking up the broken glass.
offering support to the airline "within hours" of the incident unfolding
And what's the cost of the disruption caused "within hours"?