News: 1721411647

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

CrowdStrike Windows patchpocalypse could take weeks to fix, IT admins fear

(2024/07/19)


Kettle If you're an IT administrator with Windows boxes on your network, Friday can't have been a lot of fun. What's likely millions of systems were or still are stuck in blue-screen boot loop hell, mostly requiring manual intervention to fix.

It's due to a [1]broken file pushed out by CrowdStrike to Microsoft Windows systems, causing them to [2]crash and stay down [3]around the world . It's hit airports, hospitals, businesses ... you name it.

On this week's [4]Kettle episode to discuss the news – see below – we have our knowledgeable enterprise tech vulture Richard Speed in the UK, our cybersecurity editor Jessica Lyons and IT reporter Brandon Vigliarolo in the US, and your host Iain Thomson. Nicole Hemsoth Prickett produced the show.

[5]

[6]Youtube Video

[7]

For those who prefer just the audio, the Kettle is available via [8]RSS and MP3 , [9]Apple , [10]Amazon , and [11]Spotify . Let us know in the comments how you're coping and what networks you've seen knackered by this snafu. ®

Get our [12]Tech Resources



[1] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/

[2] https://www.theregister.com/2024/07/19/crowdstrike_update_nhs_it_outages/

[3] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/

[4] https://www.theregister.com/Tag/Kettle

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zprh9vkkjl4jixvUPC1n5QAAAhE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[6] https://youtu.be/4bgQoneymPQ

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zprh9vkkjl4jixvUPC1n5QAAAhE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[8] https://feeds.simplecast.com/Vnvf8Fkd

[9] https://podcasts.apple.com/us/podcast/the-register-kettle/id1713589041

[10] https://music.amazon.com/podcasts/fe62a659-3451-496b-b064-f30744c2ad65/the-register-kettle

[11] https://open.spotify.com/show/3SkH00VOX02KYNaAbYzajj

[12] https://whitepapers.theregister.com/



Anonymous Coward

> If you're an IT administrator with Windows boxes on your network, Friday can't have been a lot of fun

Wait, do some people in IT actually enable auto-updates and carelessly assume they will work with zero testing? I thought that was just a joke. That is a terrible strategy in terms of resilience!

I actually don't blame Microsoft or Crowdstrike. I blame slackers who don't apply updates responsibly. Fun to see just how large a proportion of the IT industry are incompetent.

> could take weeks to fix, IT admins fear

The longer the better. It gives people more time to go back and learn their trade.

Anonymous Coward

You're an idiot.

Given the choice between an occasional glitch and a zero-day, I'll take the glitch.

Turning auto updates for security patches on is the ONLY responsible choice in 2024.

In this case, the mistake was attempting to 'manage' the updates with a broken 3rd party tool. Don't do that. Just turn on the auto updates.

NONE of my clients were hit by this, because NONE of them use this crapass 'management' garbage software. If you feel like you absolutely must 'manage' something, stick to open source software.

Anonymous Coward

> Given the choice between an occasional glitch and a zero-day, I'll take the glitch

You are an idiot.

You do understand updates don't need to be "automatic" to be installed right? haha.

Perhaps you don't know how to install updates properly? Hint: You don't go around all machines with a floppy disk anymore ;)

> stick to open source software

If you carelessly install updates (many of them are irrelevant I may add), even open-source won't save you.

Nate Amsden

I'd wager 85% of the times that auto updates are either enabled, or that updates are completely disabled(or perhaps the software version is obsolete and can't get updates). Which is worse? Most places likely lack the skills to perform proper testing, I know this from experience working for companies that have built their own software for 24 years now. Talking internal QA failures, which of course MS and obviously Crowdstrike has as well. The safest bet is to just delay the update by a bit see if others get bit by it first.

Problem is, for security software like this I suspect 95%+ is updated from "the cloud" anyway(and likely updates are checked for multiple times a day). Likely large numbers of systems running in isolated areas not connected to main corporate networks so no easy way to slip in some kind of intermediary update management platform. Not to mention remote employees who may almost never login to a VPN to do their work. I'm sure there are some systems that can work around those kinds of things but it adds more cost and complexity and more often than not the organizations don't want to pay for it(and may not have the staff skilled to handle it adding to more costs). Same can be said for going "multi cloud" (in the truest sense of the phrase), extra cost, complexity. I recall at the last org I was at they used Sophos for their IT security endpoint solution. I recall at one point I asked Sophos a question about something and they said something along the lines of, "do you know you don't have ransomware protection enabled? you just need to go into this setting and click this check box". The IT staff at the company never paid attention to some of the most basic things, which I think is the norm rather than the exception. After the "network engineer" quit I found out that he had not applied any software updates to their ASA firewalls in ~4 years and I counted 120+ known security vulnerabilities in it. He didn't ever put them on software support because "the devices never fail" .... .... ...

For me, I feel sorry for those folks impacted myself I don't have any real suggestions. Glad I don't really have to deal with corporate IT endpoints, my work has been on internet facing linux stuff for the past 21 years. Though I do deal with windows servers as well, just is a tiny fraction of my routine.

On my Windows 10 1809 LTSC VM that I use for work stuff I only apply updates there manually, by using local security policy or whatever it's called to disable the auto updates(apparently disabling the windows update service in Win10 wasn't sufficient like it was in Win7 which I used till late 2022). I get updates till 2029 I believe so don't have to worry about Win11 for a while, by 2029 even Win11 should be to a decent point of stability. I haven't had a known security incident on my home systems since the [STONED] virus in the early 90s. Though I did have AV software flag some malware in some pirated game stuff I did back in the late 90s(none if it appeared to be actually harmful as far as I could tell).

That and I moved my org out of the cloud 12 years ago, so I don't have to worry about that aspect of things either, my co-lo runs smooth as butter in their super conservative configurations.

Anonymous Coward

> The safest bet is to just delay the update by a bit see if others get bit by it first.

Yep, it is as simple as that. First step is to turn off the compusive auto-updates and show a tiny bit of due diligence.

Let the 12 year old gamers be the first to test the random Windows updates. That is what they are there for!

Anonymous Coward

I'm doing neither video nor audio. Just gimme something to read.

Written word

diodesign

We got ya covered - see the links. We've written a ton so far. Some people like to hear from us hacks direct. This is that.

C.

Re: Written word

Anonymous Coward

Will the links have THIS STORY in writing too?

That's the goal, here: my phone has no headphone jack so 1) its shite and 2) I don't carry headphones, and thus on the bus or just when life's too short we don't want to die inside while someone speaks loudly and slowly. Written words allow us to take in info - and diagrams - in a way that ensures our fellow passengers doesn't hate us more than we do already.

In short, there's value. And when people spend so much effort to produce a story or article, why limit its reach?

-..

It's a video

diodesign

Nah it's a video/audio discussion by Reg staff. It's partly to show we are smart, nice but sarcastic humans putting this site together, not some bots or humorless suits. You don't have to watch it; it complements our written coverage. Some people like to listen and don't have time to read through pages of text.

Their needs are as important as your needs, as important everyone else's needs. This is classic 'you can't please all the people all the time'. We're at peace with it.

C.

(BTW YouTube does auto-generate a transcript that replays in real-time, which you could follow on mute, tho it might not be to your liking.)

Re: Written word

cyberdemon

I appreciate the coverage. Both written and spoken.

I've often railed against the morphing of journalism into podcast chatter and u-bend narcissism, but this time we had about 5 written articles and one audio summary (i didn't watch the heads move) which is a good balance IMO, and it was a good summary.

CloudStrife (for that is what I will call them from now on) have perfectly demonstrated how there is "no silver bullet" for infosec, and they have provided me with a plentiful supply of schadenfreude for a Friday afternoon.

Cav

Who can afford to mix and match security across their systems? People usually buy in bulk, from one supplier, because you get a better price. Go with a few sites on supplier A and others on supplier B etc and you might not be able to afford the contracts. Only the biggest organizations could afford to do that.

And if you mean limit the numbers of customers per product then how will you enforce that? It's customer choice. If half the world chooses to use one supplier then there's little you can do about it. What are you going to do, issue quotas per supplier? Who will maintain that on a global scale?

Just uninstall the update…

Anonymous Coward

…or can’t you do that anymore? ;) ;) ;)

Eleventh Law of Acoustics:
In a minimum-phase system there is an inextricable link between
frequency response, phase response and transient response, as they
are all merely transforms of one another. This combined with
minimalization of open-loop errors in output amplifiers and correct
compensation for non-linear passive crossover network loading can
lead to a significant decrease in system resolution lost. However,
of course, this all means jack when you listen to Pink Floyd.