News: 1722234614

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Microsoft admits 8.5 million CrowdStruck machines estimate was lowballed

(2024/07/29)


Microsoft has admitted that its estimate of 8.5 million machines crashed by CrowdStrike's faulty software update was almost certainly too low, and vowed to reduce infosec vendors' reliance on the kernel drivers at the heart of the issue.

Redmond [1]posted an incident response blog on Saturday – titled "Windows Security best practices for integrating and managing security tools" – in which veep for enterprise and OS security David Weston explained how Microsoft measured the impact of the incident: by accessing crash reports shared by customers.

But of course, as Weston noted, not every Windows customer shares crash reports.

[2]

"It's worth noting the number of devices which generated crash reports is a subset of the number of impacted devices previously shared by Microsoft," he wrote. Which means the 8.5 million crashed machine estimate Redmond shared on July 20 for over a week was not entirely accurate. It was also advantageous to Microsoft, which was criticized for the fragility of its OS in the wake of the incident – especially in mainstream media, which often identified crashes caused by CrowdStrike as a Microsoft mess.

[3]

[4]

Weston's post justifies how Windows performed, on the grounds that kernel drivers like those employed by CrowdStrike can improve performance and prevent tampering with software in ways that enhance security.

He noted, however, that infosec vendors must rationalize those benefits against potential negative impacts on resilience.

[5]

"Since kernel drivers run at the most trusted level of Windows, where containment and recovery capabilities are by nature constrained, security vendors must carefully balance needs like visibility and tamper resistance with the risk of operating within kernel mode," he wrote.

Weston observed that security vendors can find the right balance.

"For example, security vendors can use minimal sensors that run in kernel mode for data collection and enforcement, limiting exposure to availability issues," he wrote. "The remainder of the key product functionality includes managing updates, parsing content, and other operations can occur isolated within user mode where recoverability is possible."

[6]

That arrangement, he suggested, "demonstrates the best practice of minimizing kernel usage while still maintaining a robust security posture and strong visibility."

Are you taking notes, CrowdStrike?

[7]CrowdStrike meets Murphy's Law: Anything that can go wrong will

[8]CrowdStrike update blunder may cost world billions – and insurance ain't covering it all

[9]Windows Patch Tuesday update might send a user to the BitLocker recovery screen

[10]How a cheap barcode scanner helped fix CrowdStrike'd Windows PCs in a flash

Weston also reminded readers that Redmond runs an industry forum called the Microsoft Virus Initiative (MVI) in which security vendors and the OS giant work together to "define reliable extension points and platform improvements, as well as share information about how to best protect our customers."

The Microsoft veep listed the many security-related enhancements Microsoft has made over the years, and revealed the software megalith now plans "to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability."

That work will involve four efforts, namely:

Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products;

Reducing the need for kernel drivers to access important security data;

Providing enhanced isolation and anti-tampering capabilities with technologies like recently announced VBS enclaves;

Enabling zero trust approaches like high integrity attestation which provides a method to determine the security state of the machine based on the health of Windows native security features.

Point two seems aimed at ensuring a CrowdStrike-like event becomes less likely in future.

Weston didn't explain how that reduced dependence will be delivered – some re-jigging of Windows will likely be needed to make it happen.

Microsoft and Windows have a long and inglorious history of security snafus. If Redmond's changes go awry, it won't have CrowdStrike to blame for any new problems. ®

Get our [11]Tech Resources



[1] https://www.microsoft.com/en-us/security/blog/2024/07/27/windows-security-best-practices-for-integrating-and-managing-security-tools/

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[7] https://www.theregister.com/2024/07/26/crowdstrike_meets_murphys_law/

[8] https://www.theregister.com/2024/07/26/crowdstrike_insurance_money/

[9] https://www.theregister.com/2024/07/24/windows_update_bitlocker/

[10] https://www.theregister.com/2024/07/25/crowdstrike_remediation_with_barcode_scanner/

[11] https://whitepapers.theregister.com/



A Non e-mouse

Whilst I'm sure there are things MS could do to minimize the impact of a repeat ClownStrike performance, let's not forget that ClownStrike made two fundamental errors here: No testing of the files before release and no client-side sanity checking of the data before being used in the kernel. If ClownStrike has done ONE of those, we wouldn't be in this mess. (And let's not forget the "Don't deploy on a Friday" either)

"Windows Security best practices for integrating and managing security tools"

simonlb

This isn't rocket science, but if you're going to write a kernel driver which is parsing data from an external file for it to use, then you have it check the data is in the correct format and validate each data object as it is read in. You cannot assume the file has the correct data format and values. If anything being read in is out-of-scope or just plain wrong, reject it, flag it up, add an entry to your logfiles and gracefully move on. Do NOT , whatever you do, leave open the possibility of crashing.

That said, my own POV, which I've said before, is that the whole design of Windows is fundamentally flawed with far too many hacks, kludges and compromises that have been left unaddressed, or fudged to 'fix' over the years, and which now are almost impossible to fix without a complete rewrite. I mean, how can plugging in a USB device such as a keyboard, mouse or headset require a reboot of the machine to get it to work once the OS has installed the drivers? I've seen this numerous times over the years, but only on Windows machines, never on any of my Linux PC's and I've never heard anyone who uses a Mac ever mention it. That and 'forgetting' where the printer drivers have disappeared to overnight, amongst many other things.

Re: "Windows Security best practices for integrating and managing security tools"

A Non e-mouse

the whole design of Windows is fundamentally flawed with far too many hacks, kludges and compromises that have been left unaddressed, or fudged to 'fix' over the years, and which now are almost impossible to fix without a complete rewrite

To Microsoft, backwards compatibility is almost sacrosanct: This includes supporting software that did very dodgy things (e.g. abusing bugs)

To be fair, this isn't too far different to the policy for the Linux Kernel of "Don't break the user space".

Re: "Windows Security best practices for integrating and managing security tools"

Doctor Syntax

Backward compatibility is fine provided that what it's compatible with is good practice and documented as what's supposed to happen. That doesn't include things like use-after-free. It also doesn't include using a few features for their own applications but not documenting them for vendors of competing products.

Reboot of the machine to get it to work ...

abend0c4

I just unearthed my old Toshiba Libretto 50 which miraculously sprang to life running Windows 95. Merely to change the DNS server it needed to install software from the installation disks (or at least the copy on the hard disk, the installation disks being a couple of dozen floppies...) and reboot.

It's not that Windows hasn't changed (considerably) over the years, but the emphasis on backwards compatibility has led to an accumulation of deprecated features that still require (and complicate) support and maintenance. But that's a significant reason people continue to buy Windows - their software will continue to run. It's an unhappy compromise, but probably a commercial necessity.

Malware authors must be salivating

Pascal Monett

Hey guys ! Here's an entire new vector for crashing a Windows PC. Then you can sell the "solution" and take over the computer.

Those scum are intelligent scum. We've not heard the last of this.

Tom Chiverton 1

How hard can it be to flag at boot "loading module X" and then "finished running module X". If last boot has one but not the other, do not load the module, inform the user/app.

Plus a hard reboot interrupt timer after 5 many mins

If Windows had this basic protection, wouldn't be an issue.

A Non e-mouse

The driver loaded fine at boot. The issue was when the driver came to load a data file from disk once the system had booted.

John Robson

Because as a "security" product clownstrike was tagged as "essential for boot", so couldn't be simply disabled.

Doctor Syntax

The instructions for fixing it implied it wasn't essential for booting into safe mode but that wasn't possible without manual entry of the key if Bitlocker was used. It seems there's a gap there between safely fetching keys from a server and not opening up networking to a degree that would be unacceptable without services such as CloudStrike.

Blue Screen

Anonymous Coward

Blue Screen, you saw me standing alone

Without a dream in my heart

Without a love of my own

Re: Blue Screen

ComputerSays_noAbsolutelyNo

You're having the Blue Screen Blues?

I've got the blues, the blue screen blues.

Repeated failure to boot, make my weekend plans moot.

I've tried turning it off and on, Oh happy weekend, you're gone.

Sic transit gloria mundi.
[So passes away the glory of this world.]
-- Thomas `a Kempis