Too late now for canary updates, says pension fund suing CrowdStrike
- Reference: 1722537613
- News link: https://www.theregister.co.uk/2024/08/01/crowdstrike_lawsuit/
- Source link:
In what will likely be one of many class-action complaints against the embattled IT security firm, a retirement association has accused CrowdStrike, its CEO George Kurtz, and CFO Burt Podbere of defrauding it and fellow shareholders by making false and misleading statements about the biz's Falcon endpoint defense software.
CrowdStrike and its top execs "repeatedly touted the efficacy of the Falcon platform while assuring investors that CrowdStrike's technology was 'validated, tested, and certified,'" the Plymouth County Retirement Association's lawsuit
[1]PDF
, filed this week in Texas federal court, reads.[2]
But in reality, the security shop's controls and procedures for updating Falcon weren't up to snuff, the lawsuit argued. And this included not properly testing anti-threat updates before pushing them to all of its tens of millions of customers, all at once.
[3]
[4]
"This inadequate software testing created a substantial risk that an update to Falcon could cause major outages for a significant number of the company's customers," the Massachusetts-based association alleged. "Such outages could pose, and in fact ultimately created, substantial reputational harm and legal risk to CrowdStrike."
In the antivirus maker's preliminary post-incident review [5]published after it [6]crashed millions of Microsoft Windows boxes around the world with a bad Falcon sensor update, CrowdStrike [7]promised to improve its software testing and deployment by, among other things, implementing a canary deployment strategy, starting with pushing changes to a small subset of users to see how it goes and then gradually deploying to larger portions of customers.
[8]
Previously CrowdStrike would automatically distribute files that improved or tweaked the operation of its threat-detection system Falcon to all customer installations at once. In July, a corrupted definition file caused CrowdStrike's Windows kernel-level driver to access memory it shouldn't have, bringing down the whole operating system and its applications.
What's worse is that CrowdStrike did have some testing procedures in place for updates prior to release, but in this case, the validation system failed to realize the changes were malformed and allowed them to be deployed at scale.
Following that snafu, the software maker vowed to take a more staggered approach, though the pension fund is still unimpressed.
[9]
"Since the CrowdStrike outage, publicly revealed evidence indicates that CrowdStrike was taking insufficient precautions regarding such updates," the lawsuit stated.
"For instance, CrowdStrike has promised to take remedial measures to ensure that such a crash does not happen again, including implementing a so-called canary deployment of such updates, meaning a progressive rollout that starts with a subset of users.
"This indicates CrowdStrike was not taking such measures prior to the CrowdStrike outage."
When asked about the lawsuit, a CrowdStrike spokesperson told The Register : "We believe this case lacks merit and we will vigorously defend the company."
[10]The months and days before and after CrowdStrike's fatal Friday
[11]Delta Air Lines dials up Microsoft's legal nemesis over CrowdStrike losses
[12]CrowdStrike update blunder may cost world billions – and insurance ain't covering it all
[13]Post-CrowdStrike, Microsoft to discourage use of kernel drivers by security tools
The Falcon update that was heard around the world, and broke IT systems globally, sent CrowdStrike's stock tumbling more than 11 percent, according to the legal complaint, hurting investors including the retirement fund, which is seeking damages.
The association claims it has lost out financially because it was tricked into buying CrowdStrike shares by believing the biz's boasts about itself and its software's reliability. The Falcon-induced outage, caused by a lack of testing, ended up damaging the developer's reputation and stock, and thus the fund's holdings in the firm, it was argued.
A few days after the crash, Congress called on CrowdStrike's [14]Kurtz to testify about the security snafu, and analysts including Guggenheim and BTIG downgraded the biz's rating, both of which allegedly caused CrowdStrike stock to fall even further, dropping more than 13 percent.
And finally, on Monday the news broke that [15]Delta Air Lines hired famed attorney [16]David Boies to potentially seek as much as $500 million in damages from CrowdStrike and Microsoft after the airline was hit hard by the Falcon-caused outage.
This third nail in the coffin by itself caused $CRwD's stock price to drop almost 10 percent, doing further harm to the association's retirement pot, the class-action suit says.
While its legal battles are unlikely to go away anytime soon, CrowdStrike on Wednesday said it's [17]making progress on getting any straggling Windows devices back online.
"Using a week-over-week comparison, ~99% of Windows sensors are online as of July 29 at 5pm PT, compared to before the content update," the update noted. ®
Get our [18]Tech Resources
[1] https://regmedia.co.uk/2024/08/01/1_24_cv_00857.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqwFhwyKk6Q5QOr4FREqQwAAAFY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqwFhwyKk6Q5QOr4FREqQwAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqwFhwyKk6Q5QOr4FREqQwAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/07/24/crowdstrike_validator_failure/
[6] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
[7] https://www.theregister.com/2024/07/25/crowdstrike_timeline/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqwFhwyKk6Q5QOr4FREqQwAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqwFhwyKk6Q5QOr4FREqQwAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/07/25/crowdstrike_timeline/
[11] https://www.theregister.com/2024/07/30/crowdstrike_delta_microsoft_lawsuit/
[12] https://www.theregister.com/2024/07/26/crowdstrike_insurance_money/
[13] https://www.theregister.com/2024/07/29/microsoft_crowdstrike_kernel_mode/
[14] https://www.theregister.com/2024/07/23/crowdstrike_ceo_to_testify/
[15] https://www.theregister.com/2024/07/30/crowdstrike_delta_microsoft_lawsuit/
[16] https://www.theregister.com/2000/05/31/ms_trial_lawyer_in_legal/
[17] https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
[18] https://whitepapers.theregister.com/
Hurting investors
It's possible they had some original involvement, but for most shareholders the likelihood is that they didn't "invest", but bought someone else's lottery stake. It is possible for shareholders to influence the behaviour of companies, but most of them seem to find that too much of an effort.
It's a shame the only people that stand a chance of suing ClownStrike are the investors and not the affected customers.
According to the article one of their customers (Delta Airlines) is suing them and Delta has a pretty good chance of succeeding, especially with a top lawyer.
In which case the Crowdstrike shareholders should be suing Delta for devaluing the Crowdstrike stock?
If 99% of Deltas computers were disabled by a virus because they hadn't had AV updates pushed out to them during the canary phase, I expect they'd sue in that case as well.
Vultures circling vultures (with all due respect to vultures).
WTF did I just read?
"pushing changes to a small subset of users to see how it goes"
Wot?! Can't they afford a test lab? Or maybe use the C-suite's personal computers then roll it out to the shareholders personal computers.that'll do it!
Re: WTF did I just read?
I suspect they may not get the chance
It's highly likely they'll be sued out of existence, and the carcass picked up by one of the AV names.
Re: WTF did I just read?
They can spin up a Windows VM for testing, but the real cost would be hiring human QA testers to take responsibility for making the calls to delay a release. Other tech companies hire QA, but to maximize profit for the next financial quarter, you need to cut down on expenses like paying for employees. This is the strategy for optimizing for the short term over the long term.
I hate to defend Microsoft...
Unless you are apple the diversity of hardware, bios, OS and other configurations make a test lab difficult even if you can afford it trying to ensure that you have every possible configuration represented and the updates that will be applied by other vendors like Microsoft if you are CrowdStrike or OEMs like HP, Dell, Lenovo... if you are Microsoft have been tested with the patch you are putting out is a large undertaking, and a large part of the reason even when hardware vendors are onside getting full compatibility for new hardware in free open source software maintained by volunteers can take time. So whilst Microsoft have dumped a number of dud releases on us they have not quite messed up like this and being the vendor of a whole OS and security product for the same OS their record doesn't look so bad compared to the Crowd now.
Still given the over 660 million shareholders and a major fraction of the global population being customers, over 200 billion in revenue in 2023 one might expect Microsoft to keep improving.
Re: I hate to defend Microsoft...
Sorry, I disagree, you can't give Microsoft a pass like that, exactly BECAUSE they have the resources to do it right - all they do now is pass the burden to the customer.
First off, hardware diversity is something that you can address by layering the software. It's exactly because they sought to give their own software a competitive advantage that they broke that layering, so issues there are their own fault.
Secondly, their code is brutally insufficient, they got lazy through Intel using that to flog new, more powerful gear.
Thirdly, good testing would have avoided the lack of functionality, missing features and other issues that bedevil their 'new' version of Outlook and Teams - no hardware dependency there, it's good old UI and software design. Instead, customers were forced to waste massive amount of staff time yet again working around problem created by practically every new MS software release ever.
I've said it before and I'll say it again: if anyone was honest enough to also consider the OPEX expenditure associated with the use of Microsoft products they would not have gotten far.
Unfortunatly your canary is really a (dead) parrot
Crowdstrike might have figured out that they really need to have organized testing but its really too late. This fiasco calls their entire software engineering capability into question -- until proved otherwise what they appear to be vending is a bright idea that was sloppily implemented backed by a first class marketing and sales effort.
Re: Unfortunatly your canary is really a (dead) parrot
Considering the diversity of hardware setups around the world that crashed, the update was likely not tested on any real-life machine at all, just pushed out.
validated, tested, and certified
Sounds like weasel words with built in get outs from crowdstrike.
Validated: is it the right product? No reference to quality.
Tested: anything can tested but what tests? (and did it even pass?)
Certified: according to what?
Could pass all those checks and still be a buggy thing that brings down global IT.
Well.....
There were/are configuration options that would have delayed the rollout and let the customers perform their own canary tests. Recommended or not, they were not exercised by the victims' IT teams. This is a good reason to TEST TEST TEST and stop firing the folks that would have prevented this from happening at your company! The vendor is not the one signing up for the risk here... it is the plaintiffs. I don't see lawsuits going much of anywhere. But maybe the point is only to bleed Crowd strike a little?
If it's mission critical..
... don't run it on Windows.
Is it ok to say...
Fry the fucks?