Microsoft admits 8.5 million CrowdStruck machines estimate was lowballed
- Reference: 1722234614
- News link: https://www.theregister.co.uk/2024/07/29/microsoft_crowdstrike_incident_report/
- Source link:
Redmond [1]posted an incident response blog on Saturday – titled "Windows Security best practices for integrating and managing security tools" – in which veep for enterprise and OS security David Weston explained how Microsoft measured the impact of the incident: by accessing crash reports shared by customers.
But of course, as Weston noted, not every Windows customer shares crash reports.
[2]
"It's worth noting the number of devices which generated crash reports is a subset of the number of impacted devices previously shared by Microsoft," he wrote. Which means the 8.5 million crashed machine estimate Redmond shared on July 20 for over a week was not entirely accurate. It was also advantageous to Microsoft, which was criticized for the fragility of its OS in the wake of the incident – especially in mainstream media, which often identified crashes caused by CrowdStrike as a Microsoft mess.
[3]
[4]
Weston's post justifies how Windows performed, on the grounds that kernel drivers like those employed by CrowdStrike can improve performance and prevent tampering with software in ways that enhance security.
He noted, however, that infosec vendors must rationalize those benefits against potential negative impacts on resilience.
[5]
"Since kernel drivers run at the most trusted level of Windows, where containment and recovery capabilities are by nature constrained, security vendors must carefully balance needs like visibility and tamper resistance with the risk of operating within kernel mode," he wrote.
Weston observed that security vendors can find the right balance.
"For example, security vendors can use minimal sensors that run in kernel mode for data collection and enforcement, limiting exposure to availability issues," he wrote. "The remainder of the key product functionality includes managing updates, parsing content, and other operations can occur isolated within user mode where recoverability is possible."
[6]
That arrangement, he suggested, "demonstrates the best practice of minimizing kernel usage while still maintaining a robust security posture and strong visibility."
Are you taking notes, CrowdStrike?
[7]CrowdStrike meets Murphy's Law: Anything that can go wrong will
[8]CrowdStrike update blunder may cost world billions – and insurance ain't covering it all
[9]Windows Patch Tuesday update might send a user to the BitLocker recovery screen
[10]How a cheap barcode scanner helped fix CrowdStrike'd Windows PCs in a flash
Weston also reminded readers that Redmond runs an industry forum called the Microsoft Virus Initiative (MVI) in which security vendors and the OS giant work together to "define reliable extension points and platform improvements, as well as share information about how to best protect our customers."
The Microsoft veep listed the many security-related enhancements Microsoft has made over the years, and revealed the software megalith now plans "to work with the anti-malware ecosystem to take advantage of these integrated features to modernize their approach, helping to support and even increase security along with reliability."
That work will involve four efforts, namely:
Providing safe rollout guidance, best practices, and technologies to make it safer to perform updates to security products;
Reducing the need for kernel drivers to access important security data;
Providing enhanced isolation and anti-tampering capabilities with technologies like recently announced VBS enclaves;
Enabling zero trust approaches like high integrity attestation which provides a method to determine the security state of the machine based on the health of Windows native security features.
Point two seems aimed at ensuring a CrowdStrike-like event becomes less likely in future.
Weston didn't explain how that reduced dependence will be delivered – some re-jigging of Windows will likely be needed to make it happen.
Microsoft and Windows have a long and inglorious history of security snafus. If Redmond's changes go awry, it won't have CrowdStrike to blame for any new problems. ®
Get our [11]Tech Resources
[1] https://www.microsoft.com/en-us/security/blog/2024/07/27/windows-security-best-practices-for-integrating-and-managing-security-tools/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/oses&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqdoQzYv5GyiTrJm84JjzAAAAUQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/07/26/crowdstrike_meets_murphys_law/
[8] https://www.theregister.com/2024/07/26/crowdstrike_insurance_money/
[9] https://www.theregister.com/2024/07/24/windows_update_bitlocker/
[10] https://www.theregister.com/2024/07/25/crowdstrike_remediation_with_barcode_scanner/
[11] https://whitepapers.theregister.com/
"Windows Security best practices for integrating and managing security tools"
This isn't rocket science, but if you're going to write a kernel driver which is parsing data from an external file for it to use, then you have it check the data is in the correct format and validate each data object as it is read in. You cannot assume the file has the correct data format and values. If anything being read in is out-of-scope or just plain wrong, reject it, flag it up, add an entry to your logfiles and gracefully move on. Do NOT , whatever you do, leave open the possibility of crashing.
That said, my own POV, which I've said before, is that the whole design of Windows is fundamentally flawed with far too many hacks, kludges and compromises that have been left unaddressed, or fudged to 'fix' over the years, and which now are almost impossible to fix without a complete rewrite. I mean, how can plugging in a USB device such as a keyboard, mouse or headset require a reboot of the machine to get it to work once the OS has installed the drivers? I've seen this numerous times over the years, but only on Windows machines, never on any of my Linux PC's and I've never heard anyone who uses a Mac ever mention it. That and 'forgetting' where the printer drivers have disappeared to overnight, amongst many other things.
Re: "Windows Security best practices for integrating and managing security tools"
the whole design of Windows is fundamentally flawed with far too many hacks, kludges and compromises that have been left unaddressed, or fudged to 'fix' over the years, and which now are almost impossible to fix without a complete rewrite
To Microsoft, backwards compatibility is almost sacrosanct: This includes supporting software that did very dodgy things (e.g. abusing bugs)
To be fair, this isn't too far different to the policy for the Linux Kernel of "Don't break the user space".
Re: "Windows Security best practices for integrating and managing security tools"
Backward compatibility is fine provided that what it's compatible with is good practice and documented as what's supposed to happen. That doesn't include things like use-after-free. It also doesn't include using a few features for their own applications but not documenting them for vendors of competing products.
Reboot of the machine to get it to work ...
I just unearthed my old Toshiba Libretto 50 which miraculously sprang to life running Windows 95. Merely to change the DNS server it needed to install software from the installation disks (or at least the copy on the hard disk, the installation disks being a couple of dozen floppies...) and reboot.
It's not that Windows hasn't changed (considerably) over the years, but the emphasis on backwards compatibility has led to an accumulation of deprecated features that still require (and complicate) support and maintenance. But that's a significant reason people continue to buy Windows - their software will continue to run. It's an unhappy compromise, but probably a commercial necessity.
Malware authors must be salivating
Hey guys ! Here's an entire new vector for crashing a Windows PC. Then you can sell the "solution" and take over the computer.
Those scum are intelligent scum. We've not heard the last of this.
How hard can it be to flag at boot "loading module X" and then "finished running module X". If last boot has one but not the other, do not load the module, inform the user/app.
Plus a hard reboot interrupt timer after 5 many mins
If Windows had this basic protection, wouldn't be an issue.
The driver loaded fine at boot. The issue was when the driver came to load a data file from disk once the system had booted.
Because as a "security" product clownstrike was tagged as "essential for boot", so couldn't be simply disabled.
The instructions for fixing it implied it wasn't essential for booting into safe mode but that wasn't possible without manual entry of the key if Bitlocker was used. It seems there's a gap there between safely fetching keys from a server and not opening up networking to a degree that would be unacceptable without services such as CloudStrike.
Blue Screen
Blue Screen, you saw me standing alone
Without a dream in my heart
Without a love of my own
Re: Blue Screen
You're having the Blue Screen Blues?
I've got the blues, the blue screen blues.
Repeated failure to boot, make my weekend plans moot.
I've tried turning it off and on, Oh happy weekend, you're gone.
Whilst I'm sure there are things MS could do to minimize the impact of a repeat ClownStrike performance, let's not forget that ClownStrike made two fundamental errors here: No testing of the files before release and no client-side sanity checking of the data before being used in the kernel. If ClownStrike has done ONE of those, we wouldn't be in this mess. (And let's not forget the "Don't deploy on a Friday" either)