News: 1721393226

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Azure VMs ruined by CrowdStrike patchpocalypse? Microsoft has recovery tips

(2024/07/19)


Updated Did the CrowdStrike patchpocalypse knock your Azure VMs into a BSOD boot loop? If so, Microsoft has some tips to get them back online.

It's [1]believed that a bad channel file for CrowdStrike's endpoint security solution Falcon caused its Sensor active detection agent to attack its host. That's caused Windows machines around the world to become even less useful and wreaked havoc at airports, [2]hospitals , emergency services and in countless unexpected places.

It's not believed that the CrowdStrike failure was related to [3]the other Azure outage yesterday, so if you're recovering from one hopefully you didn't have to deal with the other. If your VMs were borked by Falcon, however, then read on.

Just keep booting

We'd tell you it's a joke, but it's not: Microsoft's top piece of advice to fix your broken Azure VMs is to turn them off and on again - repeatedly. No, even more than that.

"We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines," Microsoft said on its [4]Azure status page as of writing. "Several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage."

[5]

Microsoft says affected users can reboot their VMs in the Azure portal, or by using Azure CLI or Azure Shell.

[6]

[7]

It's always a great situation when mitigation starts with "reboot and pray."

Of course, that's not going to help everyone, and from there the steps are largely similar to what's been reported by other people, like CrowdStrike's head of threat hunting, Brody Nisbet: You [8]gotta do it manually.

[9]

First, if you have a backup from before 1900 UTC yesterday, just restore that. If your backup habits are lax, then you're going to have to [10]repair the OS disk offline , which will be more difficult [11]for those with encrypted disks .

Once you've successfully attached a recovery disk, Microsoft says customers need to delete Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys, the same recommendation Nisbet made for other affected users.

Unfortunately, even that might not work, Nesbit said - here's hoping your systems don't fall into that category.

[12]

Rebooting has been recommended as a solution largely to give the machine a chance to try to contact CrowdStrike servers and retrieve the fix. Unfortunately, when you're stuck in a boot loop this isn't very feasible. For those unable to boot into Windows, be it on a VM or physical machine, the Internet Storm Center has recommended booting into safe mode with networking, and then following the steps to delete the offending file. ®

Updated at 15.51 on July 19, 2024, to add:

CrowdStrike's [13]notice page for the outage has been updated to add more recovery options, as well as specific steps for AWS users and those whose Windows VMs are secured via Bitlocker.

Get our [14]Tech Resources



[1] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/

[2] https://www.theregister.com/crowdstrike_update_nhs_IT_outages

[3] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/

[4] https://azure.status.microsoft/en-us/status

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZpqNl1Mia3rX@daYBF-44AAAAVU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpqNl1Mia3rX@daYBF-44AAAAVU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpqNl1Mia3rX@daYBF-44AAAAVU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[8] https://x.com/brody_n77/status/1814185935476863321

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpqNl1Mia3rX@daYBF-44AAAAVU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[10] https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/unmanaged-disk-offline-repair

[11] https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/unlock-encrypted-disk-offline

[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpqNl1Mia3rX@daYBF-44AAAAVU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[13] https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

[14] https://whitepapers.theregister.com/



Security solution attacks its own host

t245t

Sounds like that biological mRNA “ vaccine ” that attacks its own host.

“ CrowdStrike's endpoint security solution Falcon ”

You mean an Anti Virus solution working off of a blacklist of potential malware.

--

“ overall feedback is that reboots are an effective troubleshooting step at this stage ” :o

Someone is going to get their ass kicked

Noonoot

No comment

Re: Someone is going to get their ass kicked

heyrick

Everybody who authorised (*) having unverified (by their company's IT) updates self-installing on mission critical systems ?

* - careful wording as it's often the bean counters rather than omissions by IT.

Re: Someone is going to get their ass kicked

Anonymous Coward

Or because your Windows updates were migrated over to your MSP by the powers that be, where you lost all control. Despite asking for filtering you get told "No, as then we'd have to do it for all companies we support" and then when asked "Well do you at least delay the updates a few weeks to make sure they're tested?" and told "Yes" but this turns out to be a lie.

Luckily we never got hit.

Anonymous Coward

good luck booting in to safe mode for VM's in Azure! A big shitty issue with VM's in Azure is the lack of a proper console you're stuck with the serial console when things to titsup at bootup which is pi$$ poor! On prem with esx, hyper-V, AHV or whatever hypervisor you have, and you have a connection issue to a VM just bat up the console

Anonymous Coward

I heard deleting the bios resolves the problem. If that fails remove all physical drives then submerge it in water, 30 minutes should do the trick. Only use fire as a last resort.

I jest ofc but I do love the old turn it off and on again. The multiple and up to 15 times just makes it even crazier. I feel sorry for those that are having to deal with this right now. Who would have thought an update pushed out on a Friday (or just before) could cause so much chaos? When will humanity ever learn not to do this?

Flywheel

I heard deleting the bios resolves the problem.

Wow! I read that as "deleting the boss" - maybe that would be a better bet?

Doctor Syntax

Always.

Doctor Syntax

"The multiple and up to 15 times just makes it even crazier."

It makes the disclaimer easier - if it hasn't righted itself yet then you just haven't tried enough times.

richardcox13

> The multiple and up to 15 times just makes it even crazier.

But does make sense. The theory is that eventually the network stack gets enough time before the next BSOD to have updated to the latest files, which don't trigger the BSOD.

I can also imagine this very much depends on the machine's internet connectivity being very low latency and very high bandwidth.

Anonymous Coward

is this not in the cloud? Should that be an issue? If I was to guess it would be something to do with the propagation of the update.

Ya know?

Omnipresent

I really don't know why we are so concerned about the tinkering monkeys stupidly destroying themselves with technology. The world will carry on without them.

Re: Ya know?

John Savard

I wasn't aware that anyone else but us humans was able to understand English and post on these forums. We humans are concerned about the survival of the human race because it's the one we belong to, and thus its survival is required for our own personal survival.

Re: Ya know?

heyrick

I'm old enough and cynical enough that I'd happily drink a toast to the end of the shitshow that passes for humanity and raise another glass to our new feline overlords.

Re: Ya know?

Khaptain

"I'm old enough and cynical enough that I'd happily drink a toast to the end of the shitshow that passes for humanity and raise another glass to our new feline overlords."

You mean the Furries ?

Re: feline overlords

TimMaher

No, no! It’s not them. It’s the white mice.

Mines the one with the towel in the pocket.

Re: Ya know?

TheMaskedMan

"We humans are concerned about the survival of the human race because it's the one we belong to, and thus its survival is required for our own personal survival."

True, but I can think of quite a few individuals that the race could easily manage without.

Human intervention is needed.

glennsills

The problem here is that people who own/lease Windows systems, including VMs, are automatically updating software that can bring those systems down. This automated updating system saves tons of money in payroll expenses, but a simple, "Let's try this release on a sandbox and see how things work before we roll it out to every system" would be prudent. If you are an airline, and airport, a healthcare system (NHS, really?), or any other system that must run continuously, you shouldn't be trusting your vendors to be perfect. I understand that business want to save money, but until businesses that own software take some responsibility for maintenance of that software, this kind of problem will continue to happen.

And no, having AI check out the software is not a solution. :-)

Re: Human intervention is needed.

Anonymous Coward

True but places like the NHS, you have people in charge in IT that won't listen to their engineers. They appear to think they know best. Like the directors who took bribes from HP back in 2008 to go with HP laptops. The managers who ignored my requests for a HDD crusher of our own, then there was a big breach with sold HDDs. Its always someone else fault when things go wrong, but their idea when things go right so they can get their bullshit promotion.

Peter Principle comes to mind.

Re: Human intervention is needed.

hoola

Yet on other topics people are screaming that systems are not updating. The trouble with AV, particularly a nebulous cloud based pile of shite is you have no control. That is what a "modern " solution is sold on. Always up to date, no need to interact or manage stuff.

alcomatt

I guess it detected windows, and blocked this malware. Product working as intended.... ;-)

hoola

Looking at it more objectively there are several possibilities:

1. Very little is deployed on Linux

2. Simple luck that it was not affected.

I am always stunned at how few people use AV on Linux server or desktop and Mac. They are not invulnerable and the attact vectors are moving to software not the OS.

Life imitates art

502 bad gateway

Flippantly I was quoting Roy: "Hello, IT. Have you tried turning it off and on again?" earlier today... how perverse it is that I was unintentionally providing official support advice.

Thank Frick it's Friday, we can have a nice cool beer on the way home

Tut tut ... letting your real feeling out !!!

Anonymous Coward

" ... That's caused Windows machines around the world to become even less useful"

Meeeow !!!

:)

P.S. I wonder if the person responsible for the $12.5bn (as at market opening) slipup is still working for Crowdstrike !!!???

P.P.S. Good time to buy Crowdstrike cheap as they will recover because there is no quick/easy/cheap alternative !!!

Also good time to get into Coffee, Pharma Companies (particularly Migraine Meds) and Keyboards (lots are going to be worn out !!!)

If this market play pays good, please let me know !!! :)

Doctor Syntax

"First, if you have a backup from before 1900 UTC yesterday, just restore that."

And accept that you lost any work, orders taken, despatches made, whatever, since then.

Anonymous Coward

depends if you have anything on the box that changes state, for a large number of bog stand application servers it probably doesn't matter

richardcox13

Equally if it is a database server you then restore the most recent DB backup.... if your DB is not been backed (could be incremental and/or log) up every few minutes your business data is very vulnerable to loss.

Doctor Syntax

Yes. If you have incremental backups you'll be fine. But my point, which I maybe didn't make plain, was that just restoring a pre-1900hrs backup will, on its own, take you back to the state of work then. It will take further action to recover subsequent transactions, whether by restoring incremental backups or by manual re-entry. If, however, data was entered, say from a website, and there was no separate backup of that then it will have been lost. Is there a chance of recovering information from email acknowledgements or did the restore overwrite outbound emails? The system may even start handing out order numbers duplicating those issued between backup and outage.

Restoring a backup image is only the start of getting back.

Life is a game. Money is how we keep score.
-- Ted Turner