CrowdStrike president cheered after accepting 'Epic Fail' Pwnie award
- Reference: 1723461914
- News link: https://www.theregister.co.uk/2024/08/12/crowdstrike_president_cheered_following_acceptance/
- Source link:
CrowdStrike hires outside security outfits to review troubled Falcon code [1]READ MORE
The award's recipient was never really in doubt. The list of nominations for each Pwnie was released five days after the incident began and as such, there [2]could only ever be one winner of this in 2024 .
While every other award entry has a shortlist of nominations, each with detailed descriptions of the reasons for their inclusion, the Epic Fail category simply lists "lol" and "lmao even" as the contenders.
CrowdStrike has quite rightly attracted a flood of criticism for its role in what is likely to be the [3]IT event of the year . However, its president Michael Sentonas took the stage at the annual awards ceremony hosted at DEF CON this weekend, confidently gripped the ironically large trophy, and faced the embarrassment head-on.
"Definitely not the award to be proud of receiving," Sentonas [4]said to conference delegates in his acceptance speech. "I think the team was surprised when I said straight away that I'd come and get it because we got this horribly wrong. We've said that a number of different times and it's super important to own it when you do things well. It's super important to own it when you do things horribly wrong, which we did in this case.
[5]
"The reason why I wanted the trophy is: I'm heading back to headquarters. I'm going to take the trophy with me. It's going to sit pride of place because I want every CrowdStriker who comes to work to see it because our goal is to protect people and we got this wrong, and I want to make sure that everybody understands these things can't happen and that's what this community is about.
[6]
[7]
"So, from that perspective, I will say thank you, and I will take the trophy, and we'll put it in the right place, and make sure everybody sees it. So, thank you."
The speech and Sentonas' candor were met with rousing applause, and the occasional lone woos that emanated from the crowd during the president's acceptance, growing louder with each clap. It was a showing of appreciation from a room full of people whose weeks were most likely [8]ruined by CrowdStrike last month .
[9]
While DEF CON-ers were perhaps appeased by the words, in the background CrowdStrike customers like Delta [10]continue to rage about the ordeal that it claims cost the airline half a billion dollars.
In fact, Delta recently accused the embattled security vendor of trying to "shift the blame" for its IT outage – a takeaway opposing that of the DEF CON crowd who attended the prez's speech.
It follows a three-way standoff between Delta, figuratively wielding a gun each at Microsoft and CrowdStrike, both barrels loaded with threats of litigation, accusing both of wrongdoing after it was affected especially by the outage.
[11]
Microsoft and CrowdStrike have fired back at Delta. Satya Nadella and Microsoft's legal rep [12]claimed the company offered Delta IT support every day between 19-24 July but was ignored, before alleging Delta's IT environment was in need of modernization.
[13]Techie told 'Bill Gates' Excel is rubbish – and the Microsoft boss had it fixed in 48 hours
[14]Delta: CrowdStrike's offer to help in Falcon meltdown was too little, too late
[15]Study backer: Catastrophic takes on Agile overemphasize new features
[16]Et tu, Brute? Then fail, Caesars: When it's hotel staff, not online attackers, invading folks' privacy
CrowdStrike also [17]went on the offensive following Delta's accusations of gross negligence, rejecting these claims, and saying it would aggressively defend any case brought against it.
Similarly to Microsoft, the letter CrowdStrike's lawyer sent to Delta also alleged that the airline's IT might not be up to scratch and said if Delta were to proceed with a trial, it would have to lay bare exactly why that might be the case.
Like Microsoft, CrowdStrike also claimed Delta refused its offers of support – something Delta inturn claimed [18]came too little, too late . ®
Get our [19]Tech Resources
[1] https://www.theregister.com/2024/08/07/crowdstrike_full_incident_root_cause_analysis/
[2] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
[3] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[4] https://x.com/singe/status/1822324795645575263
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZroxofFDMakn8JOTZBM7KQAAABI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZroxofFDMakn8JOTZBM7KQAAABI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZroxofFDMakn8JOTZBM7KQAAABI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/19/crowdstrike_windows_kettle/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZroxofFDMakn8JOTZBM7KQAAABI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/07/30/crowdstrike_delta_microsoft_lawsuit/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZroxofFDMakn8JOTZBM7KQAAABI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2024/08/07/microsoft_delta_fight/
[13] https://www.theregister.com/2024/08/09/on_call/
[14] https://www.theregister.com/2024/08/08/delta_crowdstrikes_offer_for_help/
[15] https://www.theregister.com/2024/08/07/agile_catastrophes_risk_undermining_the/
[16] https://www.theregister.com/2018/08/20/blackhat_defcon_chaos/
[17] https://www.theregister.com/2024/08/05/crowdstrike_is_not_at_all/
[18] https://www.theregister.com/2024/08/08/delta_crowdstrikes_offer_for_help/
[19] https://whitepapers.theregister.com/
No, the correct solution is to hire at least a pair of professional assholes, like me, to perform user acceptance testing. If no one had previously asked "What happens if we feed the application garbage?" then they don't have a QA department...
At one company I worked:
"Our customers are our QA"
Of course if you are going to do this your development team should get feedback from the helpdesk, not wait for the bug reports to show up in their morning news feed...
They've added internal canary testing and customer control for staging release versions of template instances.
The content validator with mitigation improvements is scheduled to have deployed on August 19th.
Probably enough done to stop Crowdstrike from being replaced by CTOs.
https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
and root cause analysis PDF
"Significant work remains for the Windows ecosystem to support a robust security product that doesn't rely on a kernel driver for at least some of it's functionality. We are committed to working directly with Microsoft on an ongoing basis as Windows continues to add more support for security product needs in userspace."
Bottom of page 11 on the PDF showing the array of 20 / 21 pointers, one of the pointer addresses is repeated (bottom of first stack, top of the 2nd stack).
I wonder even if they had the number of pointers correct, would there have been an issue with consuming the same elements twice?
Root causes should be at the root
From the [1]PDF :
> As we enter this code, the address of the 20-input pointer array is held in register rax, and register r11 indicates that the input to be retrieved is at index 0x14, i.e., the 21st element.
Surprise! The 21st element does not point to valid memory, and global misery ensues.
However, that does not seem to be me to be adequate for a document calling itself Root Cause Analysis. How did r11 come to hold an out of bounds index? That would seem to be a rootier cause than jumping through the pointer that doesn't exist.
Disclaimer: the last time I tried to work out what the hell was going on by means of referring to assembler listing, it was MCS 6502 code. I'll get me coat, if I can remember where I left it.
[1] https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf
We all know it's a big PR stunt to accept it, it allows them to pretend to be humble while continuing the PR push to try to get themselves out of the spotlight. Still I'll grudingly give the guy a little credit for going in person to get the award.
I'd give them the benefit of the doubt - for now.
"We all know it's a big PR stunt to accept it"
OTOH you've got to ask yourself if Musk would have done such a thing.
No need to ask, he wouldn't.
And it was a PR masterclass in how to handle awards like this.
Dear El Reg
"As such" is not a posh way of saying "therefore" and should be left to the sort of people who refer to "myself".
A Pedant
I've found English-language pedantry to be a lonely, unappreciated endeavor. Not recommended.
This workplace has been incident free for X days
Guessing the trophy will sit next to such a sign?
Re: This workplace has been incident free for X days
We used to have a "purple plunger of doom", awarded to the last person to fsck it up (3ft handle, plunger on the end). No one wants to work next to that for long.
We also used to have a huge clock, about 2ft diameter, awarded to the last one who was late for work. These things are jokey ways to keep people focussed and at least aware of consequences so I hope it works for him.
Re: This workplace has been incident free for X days
We had a flag that sat on your desk till you could prove that it was someone else not your code that screwed up. A flag transfer was the ultimate shaming, so much so that everyone would be scouring their code to see if the flag might be coming their way. Occasionally someone feeling guilty would run over, grab the flag and head back to their desk. It was a silly game but it did very quickly result in extremely robust code.
"Michael Sentonas hopes trophy will remind staff that failure is unacceptable"
If failure were unacceptable he'd be fired already.
Re: "Michael Sentonas hopes trophy will remind staff that failure is unacceptable"
And I still wonder what execs mean when they say they want their software teams to "take risks".
Re: "Michael Sentonas hopes trophy will remind staff that failure is unacceptable"
Failure is not an option, it's a certainty for all of us that we will all fail in some way at some point. It's what you do to mitigate the consequences of failure and what you do after the inevitable failure occurs that matters.
Re: "Michael Sentonas hopes trophy will remind staff that failure is unacceptable"
Admitting the failure and not pretending it didn't happen is arguably the most important step.
Aides-mémoire
In his long-ago youth, my brother totaled our parents' car. For years afterward, he kept that car's now-useless key on his keyring as a reminder to himself of how not to drive.
"[The award is] going to sit pride of place because I want every CrowdStriker who comes to work to see it..."
It seems as though this guy has the same idea. If that Pwnie award gets put in a glass case in CrowdStrike's lobby, it will be serving its purpose.
Time will tell, of course, whether the lesson will be learned...
Re: Aides-mémoire
Everyone makes mistakes. It happens.
What we do to minimise those mistakes, what we to to correct those mistakes, and what we do as a result of those mistakes is very telling about us.
So yes: Lessons learned. The question is: What did they learn and how will they act upon those lessons. After all, learning to hide their mistakes better isn't the same as learning how to spot them better.
Re: Aides-mémoire
The first step should always be to take ownership of ones mistake, it is the adult/responsable thing to do....
Admitting to the mistake in in such a public forum is the first step. The real task is for CrowdStrike to re-write their software.