CrowdStrike IT Outage Explained by a Windows Developer

https://www.youtube.com/watch?v=wAzEJxOo1ts

For those unwilling or unable to youtube;

In the YouTube video titled "CrowdStrike IT Outage Explained by a Windows Developer," retired software engineer and Windows developer Dave explains the cause of the recent CrowdStrike IT outage. The issue was due to a bad update to CrowdStrike software that resulted in blue screens on various machines worldwide. Dave discusses the significance of CrowdStrike being on machines in the first place and the consequences of a kernel driver failure. He also shares his experience as a Microsoft developer in the 1990s and the importance of understanding the differences between kernel mode and user mode. The speaker then delves into the concept of kernel mode and user mode, explaining that only a few things, such as thread scheduling, Heap manager, and device drivers, run in kernel mode due to its access to hardware. CrowdStrike's Falcon security product requires kernel-level access to function effectively, and writing a device driver for Falcon allows it to reside in kernel mode and access system data structures and services. However, the use of dynamic definition files instead of Microsoft's WHQL certification for drivers could potentially contain unsigned and unknown code that runs in kernel mode, posing a security risk. The precise cause of the IT outage was a null pointer issue in a dynamic data file downloaded as a Cy file, which contained only zeros instead of pcode or malware definitions. The CrowdStrike driver that processes and handles these updates is not very resilient and lacks adequate parameter validation, leading to the entire system crashing and depositing users into the recovery blue screen. Windows does offer facilities to boot without certain drivers, but CrowdStrike marked their driver as a boot driver, requiring physical access to the machine to delete the problematic file and fix the issue.

via [1]www.summarize.tech

[1] https://www.summarize.tech/

So it is down to the shitty Windows environment once again. No surprise there.

The issue might be more in the fact that it seems to be very difficult to isolate the kernel in an OS while granting security access to protective measures, all the while keeping the hoi polloi at bay and the system secure.

Too late to think about a redesign of the OS architecture. We'll just have to make do with what we have.

Any OS, from a kernel component will have this problem.

Eg. RedHat: having a kernel panic due to CloudStrike: https://access.redhat.com/solutions/7068083

Edit: note I make no comment on whether the use of definitions + driver in kernel mode is a good idea[1] rather than the driver just being a bridge to user mode worker where everything happens[2].

[1] I don't think it is.

[2] The chance of a company like CrowdStrike getting this right (handling the worker failing...) seems low.

difficult to isolate the kernel in an OS while granting security access to protective measures

It's impossible, because well designed malware could hide inside the most privileged part of the OS and remain undetectable by security software. We see even hardware enclaves don't really help for that, and their capabilities fall well short of preventing blue screens. A microkernel or whatever you're thinking wouldn't solve that.

One big reason Windows security is a huge problem is that it's such a ubiquitous OS that any threat actor producing malware for it can deploy and re-deploy that malware many times against most targets they come across, making it a cheap target to attack. If they also have to contend with an unknown OS-level security product on each target then it does raise the bar. Just not very much if that unknown security product is invariably 'Crowdstrike'.

Would be interesting to find out how the faulty .cy file got out into the update. Presumably Crowdstrike had tested with a good version, but the bad one sneaked into the package for release.

From what others have said crowdstrike bypassed their own rollout procedure to force the update straight onto production networks, bypassing staging.

So failures all round.. not only did they not test internally (testing with a different version than you send out is not proper testing) they bypassed measures that would have caught this before it did damage.

And of course crowdstrike are able to do this with no consequences because the companies all signed contracts absolving them of liabilities.. the millions spent on the cleanup will be borne by others.

"no consequences" ??

First let's see what the market has to say about it, they have had a huge amount of stock value lost.

> From what others have said crowdstrike bypassed their own rollout procedure to force the update straight onto production networks, bypassing staging.

Exactly: this isn't the failure of an individual, but an organisational operations failure where a choice to "push and be damned" led to being damned.

Deeper is the clear failure to do fuzz, and similar, testing within an integration test suite in the publication pipeline. As this would quickly lead to seeing the driver's lack of input validation with respect to the definitions.

"The CrowdStrike driver that processes and handles these updates is not very resilient"

And there's the real problem. Anything with that privilege needs to be very resilient.

Does the driver require signing by Microsoft to be allowed this access? If so then Microsoft need to exert some strict QA before doing so. And, yes, I recognise that there might be a slight problem/irony (choose according to personal preference) there.

"Null pointer issue" in Windows driver is the silver lining in this tragedy. It's a poster-child-in-waiting for getting Rust used in future drivers. https://techcommunity.microsoft.com/t5/surface-it-pro-blog/open-source-rust-driver-development-platform/ba-p/3974222

This vid is worth a watch

https://www.youtube.com/watch?v=wAzEJxOo1ts

The crowdstrike driver was WHQL tested and signed. There are some checks and balances on what can run at that level. It just seems to be at this time that the kernel driver comes up a little short on data validation.

But the driver executes code from a different file that isn't WHQL tested, and you can replace that file without installing a new driver, but the driver still behaves differently when you do.

I can't watch your video link right now, but if you are linking to the David Plummer video, I watched that last night, and he explains that.

That is the slightly shocking bit that MS approved of this setup. You should not be able to feed new code to something running in the kernel space from the user space and certainly not without huge amounts of checking.

Yeah, its Dave's video.

Without ensuring the driver performs adequate testing it should not have been signed at all.

Remember WHQL validates that a driver calls APIs correctly doesn't make a mess of interrupt handling, etc.

That there was a dependency on a definition file would not be in scope, because that is not the kind of thing that drivers for hardware drives have done historically. It would be great to see WHQL being updated to broaden tests.

Even better would be MS saying new "no third party drivers in the kernel" (Windows has long supported non-kernel drivers). However that would depend on MS being willing to fight through US and European counts that it isn't an anti-competitive measure (ie. third party software writer's laziness is an insufficient reason to allow kernel access).

If an AV is running anywhere else than around the kernel, can it still guarantee that it can do it's job ? If it was in another non-kernel tier would it not lose the capacity to protect itself from the dark side.

And wasn't the EUs request done in order to push against MS's monopolistic nature ?

You can have a proper kernel-user split putting a service which is able to do the privileged parts but is essentially simple in ring 0. But that's hard, so AV companies don't do that.

Doing hard stuff is what's expected of AV companies.

You need a hook up in kernel-space, but the rest of it can run in user-space. There are some performance drawbacks going that route, but it does lend additional stability to the system.

On a side note, Minix is a good example of an OS that runs as few things in kernel-space as possible.

Running it outside of Kernel mode isn't the answer no matter what the question is. If the code calls out to User mode and crashes, what does the Kernel code that made the call do? Does it fail open (no security) or does it fail closed (hard crash).

Question remains the same whether you're dealing with Kernel mode or not. Microsoft could have put the equivalent of "ON ERROR RESUME NEXT" in when calling third party Kernel libraries, but you've got the same problem then.

And I can bet everyone would be up in arms if Microsoft left things insecure by default...!

The answer is test what you're shipping and roll it out slowly.

Easy answer that, it should flag the problem in the event viewer and with Crowdstrike, continue with the previous good configuration, and not lose its shit and cause a bootloop.

Also nothing the EU said obliged Microsoft to put or keep ropey software architecture in Windows. They're just finding someone else to blame and if it can be that horrible organisation that dares to regulate them a bit then so much the better.

So failing open but with logging.

There's no such thing as a last known good configuration if something updates itself outside of the normal Windows process.

I don't think that continuing to run using definitions which were fine until 04:08 UTC is the same as "no security".

Imagine that the process on 8.5 million Windows PCs logged the definition error with Crowdstrike. That would have set alarm bells ringing and I'm pretty sure that another corrected update could have been pushed by Crowdstrike on the same day without taking out airports, airlines, trains, hospitals, GPs, etc...

But if the PC is bluescreened then how is the AV software going to phone home to Crowdstrike?

By catching the error previously with better validation of new definition files.

If you are outside the kernel you restart the user process (there are well established approaches to this).

Dropping back to previous definitions after n failures is also an option (works very well with good telemetry).

What are these "definitions"? Which API are your referring to that loads "definitions" in to the kernel?

Windows was loading the exact same driver it loaded last boot, which has not modified... As to what that third party code does when it runs, that's outside of Microsoft's control.

If I make a program for that loads some external data, I don't want windows going "naa, I don't want to give you the data you asked for, so here's some different data instead"

Windows was loading the exact same driver it loaded last boot, which has not modified...

The driver crashed repeatedly. How did Windows deal with it? Badly. If a driver fails often enough it should be put on the naughty step by the OS.

If I make a program for that loads some external data, I don't want windows going "naa, I don't want to give you the data you asked for, so here's some different data instead"

I don't understand. Crowdstrike's driver gets Crowdstrike's data but crashes. I don't care how the driver crash happens, but if it happens repeatedly I do want Windows to eventually disable the driver so I can get on with my day.

Crowdstrike would have to push out a driver update with Windows Update after explaining to Microsoft what went wrong and what they did about it, but that shouldn't be my concern.

That's a good suggestion, but that's not Microsoft's job. The whole reason you're running Crowdstrike is because you think they're doing a better job than Microsoft. It's a black box as far as Microsoft is concerned. There's no concept of definitions, there's nothing to roll back, there's no notification that a change to your system has been made.

Crowdstrike needed to detect that their driver had sh*t the bed and done their own rollback. It was their code that threw the blue screen. There's plenty wrong with Microsoft software without blaming them for others too.

Crowdstrike's job is better validation in the driver or preferably in a userland subprocess to reject corrupt definition files so it continues working in a reasonable way until new definition files are issued in a matter of hours.

Microsoft's job is to make sure repeated driver crashes are not fatal to the entire OS. If Crowdstrike's driver repeatedly crashes in a short space of time, then Windows should disable the driver.

We know the answer because we know it was a faulty AV update and, in that particular situation, disabling the AV was best. The kernel has none of that hindsight. Stopping a random driver would likely leave the machine useless; for example, booting without a working graphics driver, or a hard disk driver. And I have no idea the consequences of losing the "PCI-to-PCI Bridge" or the "High precision event timer", but I bet it's not good.

And the trickle down could make the situation worse: or do real damage. Even in this case, if it wasn't a faulty update but malware, then stopping the driver could allow the malware free run. (And Microsoft would get it in the neck if they disable AV, and opportunistic malware takes advantage of AV being down.) And do we know if CrowdStrike have just one driver, or multiple that interact? Has the system been tested with one down?

Anyway, the correct response to an unknown error is always to stop, do nothing more, and wait for help. If there's any solution, it's that data files need to be registered as part of kernel state so a rollback can be attempted to last known good.

"That's a good suggestion, but that's not Microsoft's job."

If Microsoft sign the driver in order to gain that access they do have a job to do which is to require it to be able to roll back and do so. Microsoft are a gatekeeper here. If they say that in order to gain access a third party has to meet quality requirements then that third party has to meet those requirements or stay outside.

The only basis for a regulator to quibble with that would be if Microsoft gave itself a free pass not to meet those requirements itself.

"There's no such thing as a last known good configuration if something updates itself outside of the normal Windows process."

If it updates itself it can revert itself to its last known good configuration if it has maintained a copy of that. If it can't then either the kernel should then fail it or, if it isn't designed to do that, it goesn't get the signature to allow it into the kernel at all.

"Easy answer that, it should flag the problem in the event viewer and with Crowdstrike, continue with the previous good configuration, and not lose its shit and cause a bootloop."

That is the default for faulty kernel drivers, but the Crowdstrike driver had flagged itself as being required for boot.

The whole point of the boot flag in kernel drivers is to tell Windows that halting is safer than continue booting without them.

Really after three failed boots due to one driver, it should be obvious that trying to boot without that driver is safer than continuing to try and boot with it.

I am not sure I would categorise automatically disabling stuff (in the context of security software!) and carrying on as if everything is normal as being "safer" than demanding someone fix the broken system.

Then make it a group policy. Let the customer have the choice of what the best action to take... for the PC to continue bashing its head against a brick wall and the company to grind to a halt or for the driver to be temporarily disabled.

A lot of people seem to think the way it's done now in Windows is the only way but it's obvious that the IT world has to come up with something better.

"t's obvious that the IT world has to come up with something better."

And that receives downvotes? No wonder the IT world is in a mess.

What if we're dealing with a system with two drivers. One. Starts a centrifuge spinning and the other checks the status of all the components and applies the brake when something goes wrong. Problem lies in the 2nd driver, so Windows goes "fine, I'll start and ignore that one".

Centrifuge starts, problem occurs, system physically destroyed.

Current method has less danger as centrifuge won't even start.

I'm sure even Microsoft's programmers can maintain separate counters for separate drivers.

But crowdstrike had labelled their driver absolutely required for boot.

The system had no choice. It can't make value judgments.. it has to use the information it has.

Crowdstrike had been allowed to label their driver as absolutely required for boot. Once installed the system may not be able to make value judgements. Whoever allowed it to be so labelled could and should have made such judgements.

... I remember with Win10, (I am currently using 11 and have not had to see this yet) when you tried to booth and it failed,it auto restarted and tried again, 3 times, then it knew it was failing somewhere and instead shut off completely or booted into startup options.

So there is precedent for this kind of behaviour in windows already, even if not exactly as you described.

So it should be possible, just need a low enough running watchdog that can catch boot errors at, well, boot time.

And I can bet everyone is up in arms because Microsoft leaves things insecure by default...!

There, FTFY

"Question remains the same whether you're dealing with Kernel mode or not"

I'd say the question remains "Was Friday's event acceptable or not?"

We should be asking why something like ClownStrike was considered necessary in the first place. It does not seem to be to just address the weaknesses in MS Windows as there are versions available for Linux and macOS. I have been hearing things about it being required by insurance and compliance.

It would be interesting to see a cost benefit analysis: what does it cost to install vs what does it prevent - especially since it does have a history of causing outages.

I suppose from the PHB point of view this ticks a box as otherwise getting security done right takes time and expertise that a business often does not want to pay for.

"We should be asking why something like ClownStrike was considered necessary in the first place. "

There is no such thing a fully protected system, we all know this and there are bunch of bastards that make their living from encrypting all your files, it is not easy job to continually protect yourself against them. Crowdstrike like all of the EDRs give you at least some hope, even though it is only a a part of a larger solution, and yes user training is part of the solution... but nothing is perfect so we make do with what we can.

It depends on the company, but a lot of them that use MS products and also Microsoft 365 are already getting AV as part of that license, whilst also having policies not to use a single vendor for everything or having a distrust of Microsoft from a security POV (where, lets be honest, they don't have the most sterling record).

I made this point at my current contract as they were complaining about costs whilst double-paying for AV (and other things), the rationale was that CrowdStrike was "better" than the MS products they were already paying for. I suspect that particular argument won't wash any more though.