Life, interrupted: How CrowdStrike's patch failure is messing up the world
- Reference: 1721401089
- News link: https://www.theregister.co.uk/2024/07/19/life_interrupted_how_crowdstrikes_patch/
- Source link:
CrowdStrike shares sink as global IT outage savages systems worldwide [1]READ MORE
The issue at CrowdStrike that put millions of Windows systems into perpetual boot loops has been blamed on a [2]faulty channel file , necessitating [3]manual intervention by an IT professional to get each system back online.
And it's not like CrowdStrike is a niche product - according to the company's [4]website , 298 of the Fortune 500, 43 out of 50 US states and eight of the top 10 food and beverage, auto, financial services and tech companies all use CrowdStrike products.
[5]
Yes, but for how long? - click to enlarge
What that means for IT professionals is clear: A long weekend. What it means for everyone else is that day to day life isn't going to feel quite normal today - and it [6]might take some time for things to get back that way.
In other words, it's going to be an interesting Friday.
The day the world BSODed still
Reactions to the outage have been pouring in from around the world. With such a wide reach, outages like these quickly show how fundamental to day-to-day life Crowdstrike (and [7]Microsoft ) have become.
Take, for example, in the city of London, where taxi drivers haven't been able to reliably take card payments. One driver speaking to the BBC [8]said around 95 percent of his customers use cards, and that he's already had to turn people away who didn't have cash.
Employees at Starbucks have taken to Reddit to [9]warn customers that their systems were down too, and those that have managed to get them back online have been left with partially-working machines, no ability to print tickets to track orders and no way to take online orders for pickup.
Depending on how quickly things are fixed, this weekend's Hungarian Grand Prix could also be affected - at least for one racing outfit. Mercedes' F1 team, of whom CrowdStrike is a [10]major sponsor , has [11]reported that their systems have been disrupted by the outage, making it unclear whether the team would be ready for the first practice later today.
[12]CrowdStrike shares sink as global IT outage savages systems worldwide
[13]Second NHS IT system confirmed to be affected by CrowdStrike issues
[14]Azure VMs ruined by CrowdStrike patchpocalypse? Microsoft has recovery tips
[15]Firms skip security reviews of major app updates about half the time
Even the news anchors at Bloomberg Radio have had their lives disrupted, with one saying this morning that the elevator in their building wasn't working due to the CrowdStrike patchpocalypse.
While this issue doesn't appear to be Microsoft's fault, it does raise the specter of [16]recent cybersecurity troubles the company has experienced, and how the Windows-maker's massive reach into critical infrastructure has turned it into a [17]potential national security threat .
[18]
CrowdStrike will have rightfully earned a spot next to Microsoft after this incident, and here's hoping the world takes [19]a different tack - ignoring Mirosoft's problems as has largely been the case at the governmental level so far.
[20]
So, how has CrowdStrike had a non-IT impact on your life today? ®
Get our [21]Tech Resources
[1] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
[2] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[3] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[4] https://www.crowdstrike.com/platform/
[5] https://regmedia.co.uk/2024/07/19/crowdstrike-reach.jpg
[6] https://www.nbcnews.com/news/world/live-blog/live-updates-it-outage-flights-banks-businesses-microsoft-crowdstrike-rcna162669#:~:text=recovery%20%27could%20be%20some%20time%27
[7] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/
[8] https://www.bbc.com/news/live/cnk4jdwp49et?post=asset%3A22fd80de-28c5-45df-8d39-7eff22ee11c0#post
[9] https://www.reddit.com/r/starbucks/comments/1e71gya/not_just_starbucks_down/
[10] https://www.sportcal.com/data-insights/top-10-sponsor-partners-mercedes-amg-f1-team/
[11] https://www.sportcal.com/data-insights/top-10-sponsor-partners-mercedes-amg-f1-team/
[12] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
[13] https://www.theregister.com/2024/07/19/crowdstrike_update_nhs_it_outages/
[14] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[15] https://www.theregister.com/2024/07/18/security_review_failure/
[16] https://www.theregister.com/2024/04/05/microsoft_government_contracts/
[17] https://www.theregister.com/2024/04/21/microsoft_national_security_risk/
[18] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZpqNlnIyC1bVQnhhOL7kvgAAAAs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[19] https://www.theregister.com/2024/06/04/pentagon_doubling_down_on_microsoft/
[20] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpqNlnIyC1bVQnhhOL7kvgAAAAs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[21] https://whitepapers.theregister.com/
Impact...
So, how has CrowdStrike had a non-IT impact on your life today?
Personally:
The total non-IT impact: 0 (zero)
The total IT-impact: 0 (zero)
I've been watching this slow moving disaster and can only say that it is a perfect example how not to design (any) infrastructure you want to rely on.
But, I'm rather convinced that, after the shock has worn off, the beancounters all over the world will prevent any real improvement from happening.
Re: Impact...
Unfortunately, I am inclined to agree with you.
After all, there are those pesky contract clauses, etc etc, and finding another AV supplier is going to be a major pain and probably require humongous amounts of work from IT personnel everywhere, so why not just listen to the siren call of "lessons have been learned" and stay the course ?
After all, it works so well for Capita in the UK . . .
WTF?
"the elevator in their building wasn't working due to the CrowdStrike patchpocalypse. "
People do know that Windows isn't certified for safety of life applications, don't they?
Re: WTF?
An elevator that doesn’t work is safe. It’s pretty inconvenient but safe. I’d expect an elevator to go to the next floor and open its doors, whatever happens on the outside.
Re: WTF?
I’d expect hope an elevator to would go to the next floor and open its doors, whatever happens on the outside.
FTFY
Re: WTF?
As long as it doesn't try moving sideways.
Re: WTF?
People might.
Companies don't.
So where do we go from here?
What is going to be the flavor of the day for next years auditors when they prod our resilience policies. What are they going to be looking for?
Still amazes me that MS allows a third-party driver to simply pull the system down and stop it getting back up, with no soft landing to allow remediation (aside from safe boot etc). Surely Windows should still start but with a huge warning that "service X couldn't start".
Windows is too fragile (no shit, Sherlock) such that it can be crippled by a third party component - although MS are pretty capable of doing that themselves.
Chaos at the Docs today
My GP had problems issuing prescriptions today, finally got a "sort of" prescription to take to a local pharmacy, but they were unable to confirm its validity due to systems down. They ended up phoning docs to check. Turns out my normal online pharmacy has now also double issued it too, which I understand from the local pharmacist has left them technically breaking the law. All fun and games.
Staggered releases?
I worked at a place writing software that could be quite essential for some countries. So we did releases timed at Monday, 7am in one country two hours ahead of us. The idea was that early birds and auto-updaters would get the changes at 9am their time, two hours later they had plenty of time to find serious problems and talk to support, and at 9am we had all the information and could start fixing things. Worldwide release was on the next day.
I learned that once in a while you can call Apple and tell them you need an urgent review for something that is on the AppStore broken, and it can get reviewed and on the store within an hour of submitting a new version.
If God would have been an IT guy
We would have been extinct from plague, or any other pandemic preceding it, due to similar genes, offering a great attack surface for the plague microbes.
Diversity in IT landscape costs a bit more, but like human genes, a single attack vector won't wipe it completely out in a short period of time.
Maybe Resilience will become a factor in IT security as well, until now the scope of IT security is very much focused on "keeping bad actors out".
SECURE NO BOOT
Sorry boss. My car had an BSOD today.
What the experts are saying
“Such bad, many downtime.”
-Doge, IT Security Spiritual Analyst
“With the security cameras off-line, I was able to loot an entire pie… with sausage”
-Pizza Rat, Survivalist NYC
What's worse than...
...Being of one of the most trusted companies in IT?
Being a single point of failure.
Luckily for me I don't have to fix any of it. Then again, this might be a good time to do some moon lighting over the weekend. I bet there will be plenty of places desperate for extra boots on the ground.