Administrators have update lessons to learn from the CrowdStrike outage
- Reference: 1721755629
- News link: https://www.theregister.co.uk/2024/07/23/crowdstrike_lessons_to_learn/
- Source link:
One of the reasons the CrowdStrike update caused so many problems was that administrators assumed the faulty update would have been pulled and fixed long before it troubled their systems. Many were cheerfully running on N-2 or N-1, meaning they were set to use a release two or one version behind the latest.
On Friday, July 19, a security expert fielding an influx of calls from alarmed customers told us: "Every managed CrowdStrike customer is N-2. It's even in their support documents as a recommendation."
[1]
Yet Windows systems around the world began experiencing Blue Screen of Death boot loops as a CrowdStrike update made its global update.
[2]
[3]
The problem for many users was understanding that the version policy only applied to part of the CrowdStrike system. One [4]posted : "We learned the N-1 policy we had in place only applies to agent updates, and not signature files."
"As far as we can tell there is not a good way to delay what signature files get pushed, hence everybody receiving the 7/18 23:09 (central time) signature file that blew up the world over the next hour."
[5]
Others [6]complained : "Crowdstrike overrode client settings on N-1/N-2 deployments (i.e. staged 1 or 2 releases back from current) for this release."
[7]Another user , having noted that they had CrowdStrike set to be one version behind on non-critical infrastructure and two versions behind on critical infrastructure, glumly said: "We got hit anyway because it was a 'content file' and so ignored our auto update restrictions."
While phrases such as "betrayal of trust" have been thrown around, it seems that not all administrators understood the update cadence applied to the software and not the channel used for signatures. There are, after all, good reasons why a customer would want the most up-to-date signature files, considering the speed at which malware evolves.
[8]EU gave CrowdStrike the keys to the Windows kernel, claims Microsoft
[9]CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
[10]Cybercriminals quickly exploit CrowdStrike chaos
[11]Life, interrupted: How CrowdStrike's patch failure is messing up the world
Sharon Martin, CEO of Managed Nerds, was in the process of spinning up a CrowdStrike evaluation to determine if it was something to offer customers just as the Blue Screen of Death wave began. As it became clear that critical systems were at risk from the wayward signature file regardless of update cadence, Martin said: "If CrowdStrike was the only EDR solution left in the world, I would choose ransomware over it.
"It's a tool in the security stack that completely broke the affected systems with no warning. There was no indication in the beginning of how or when recovery might be obtained. The information was so slow flowing, some were unsure if they should wait for a fix or just go on and reimage the machine.
[12]
"Most of the information that did flow was either to their largest partners or behind their login-walled documentation portal which is not publicly searchable. That's why some organizations didn't know whether to send their people home for the day or not on Friday."
Jamil Ahmed, Distinguished Engineer at Solace, noted the dilemma facing administrators. "n-2 for software means you are taking a conservative and cautious view to not run the most latest version.
"However ... it is in your interest to get the latest definitions of new threats and viruses as soon as you can without delay. This is the heart of the struggle here. It is the processing of that new definition that had the bug and caused the BSOD."
Planning for disaster and staging updates is critical. Using deployment rings to apply updates to a few devices first is commonplace. Analysts including [13]Directions on Microsoft's Mary Jo Foley have listed the practice as a lesson to be learned from the CrowdStrike meltdown.
However, staging will only go so far when there is a separate channel that pushes signature updates on a different cadence.
Just ask any of the administrators who have had to deal with the consequences. ®
Get our [14]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqAn@ZU7C0V72M0l2qDXMQAAAMA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@ZU7C0V72M0l2qDXMQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@ZU7C0V72M0l2qDXMQAAAMA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.reddit.com/r/sysadmin/comments/1e7w0q6/comment/le2zl98/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@ZU7C0V72M0l2qDXMQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.reddit.com/r/delta/comments/1e85b42/comment/le6sujq/
[7] https://www.reddit.com/r/technology/comments/1e6wtlq/comment/ldy5eif/
[8] https://www.theregister.com/2024/07/22/windows_crowdstrike_kernel_eu/
[9] https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
[10] https://www.theregister.com/2024/07/19/cyber_criminals_quickly_exploit_crowdstrike/
[11] https://www.theregister.com/2024/07/19/life_interrupted_how_crowdstrikes_patch/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@ZU7C0V72M0l2qDXMQAAAMA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://www.directionsonmicrosoft.com/blog/four-lessons-microsoft-customers-can-learn-from-the-crowdstrike-meltdown/
[14] https://whitepapers.theregister.com/
Considering the root cause was a total lack of file content validation (or a staggeringly shitty one if a file of all nulls is "valid"!), not even one of the "genuine antimalware boo-boos" (like the classic "false positive quarantining a key system exe") perhaps this case should prompt the Sys Admins to ask - demand - for the right to test the system for themselves.
I.e. to know precisely what files are capable of being updated, their update method (humans running installers, auto-installers that still show up in "Programs and Features" or "just data we silently grab from our servers". Which then allows them to run a test system and actually, um, test the software before they install it. By feeding it gibberish files.
Yes, you *ought* to be able to trust your vendor has run fuzzing tests, zero byte files etc, but if you are knowingly going to install software - well, really any software you install fleet-wide, but especially stuff you *know* can fiddle at a low level - and you are prevented from doing your own (double) checks that at least the most basic, trivial, well known and bleeding obvious protections are present then perhaps you should be raising alarm bells.
Development location
And I’m sure moving their development effort to India in Feb 2024 had nothing to do with it.
If you can't block it, it's malware.
Even if you paid for it.
I think the single biggest take-away from this is "How can I recover a bricked machine?"
Crowdstrike have been spectacularly bad here, but it is only a matter of time before another screw up, or another form of malware, does the same. So, if the machine won't boot, what is your plan?
You do have a plan, don't you?
The plan is to spin up my disaster recovery site! Which is also running CrowdStrike! For safety and compliance purposes!
Running N-2 in signature files might not mean much
They could be updated several times in a day, so it might not buy you much.
Everyone choosing the same "N-2" makes it worse than useless though, because they're all getting the updates at the same time. You'd want some sort of randomly time delayed application. Companies that want maximum protection against 0 day threats could choose "zero delay", those who want protection against what happened last week could choose "random delay of 1 to 3 days" and there'd be a few options to choose between the two.
That would mean some of the zero delay folks getting hit next time something like this happens, and some of the 1-3 days folks getting hit with 0 day virus/malware attacks. But that's better than everyone go down to one or the other, I suppose.
Mad and it didn't even affect us
But I only read this bit "If administrators have learned anything from the CrowdStrike chaos, it's to understand exactly what delayed updates mean – or don't mean – in the anti-malware world."
and thought "Oh fuck off. Are you a c-suit? You do realise there is no control over CrowdStrike's cocking updates, that's where the problem lies. Being forced to use CrowdStrike and it giving no option to control updates. As from what I've read, it updates every 2hrs to prevent attacks.
EDIT- Read the whole article now :) you do know, but my rant is still valid.
And
I'll also add, when you get management that is now wondering why us moving fully to the cloud isn't saving us money, that is the fuck problem. You might want to put in place a test system to control something like this, but when they won't fucking pay for it, shit like this happens.
Let's face it, all of the AV and anti malware vendors have had their dodgy and duff updates in the past, with varying impacts. Crowdstrike's duff update was more spectacular than most but most of the vendors could face the same issue one day.
So what is the industry to do? It's a constant battle to keep the baddies out of our systems and services, and keeping signatures up to date multiple times a day along with a dose of behavioural analysis (recently renamed to AI) so far has been the thing to do. We currently have a dose of whataboutery, we could do some update rings on signatures, but that seems to defeat the point somewhat. So what are the practical options to do something different?
When everyone is N-2, nobody is.
It only achieves anything if you have some of your own canaries on N and N-1.
Also, it's utterly pointless if the vendor won't tell you exactly what you're delaying - which was clearly the case.
it's not that hard to understand
N, -1, -2, code, data, signatures .... no one cares
What they are asking for is production Stability, to NOT be on the bleeding g edge. It's up to crowdstrike to manage it that way, as a service and with an sla.
Things like Crowdstrike are just a piece of your security posture. Delayed updates shouldn't cause a major concern. Maybe some consideration should be given to using a single vendor to provide this capability across your entire company.