CrowdStrike shares sink as global IT outage savages systems worldwide
- Reference: 1721382817
- News link: https://www.theregister.co.uk/2024/07/19/crowdstrike_shares_sink_as_global/
- Source link:
At the time of writing, the share price is down more than 19 percent as the security shop to some of the biggest organizations in the world continues to work through issues with its customers.
[1]
CrowdStrike shares tumble - click to enlarge
Those hoping to catch a flight today might encounter some problems as the Federal Aviation Adminstration (FAA) grounded major airlines until further notice. This includes United, Delta, and American Airlines, while Brits' go-to budget airline Ryanair is also experiencing disruption.
"Affected passengers will be notified and any passengers traveling across the network on Fri 19 July should check their Ryanair app for the latest updates on their flight," a Ryanair statement said. "We advise passengers to arrive at the airport 3 hours in advance of their flight to avoid any disruptions.
"We regret any inconvenience caused to passengers by this third-party IT issue, which is outside of Ryanair's control and affect all airlines operating across the network."
[2]
In statement sent to The Register, an FAA spokesperson said:
[3]
[4]
"The FAA continues to work closely with airlines as they work to resume normal operations. Ground stops and delays will be intermittent at various airports as the airlines work through residual technology issues. Please contact the airlines for more information and monitor [5]www.fly.faa.gov for real-time airspace updates."
From a regional perspective, Edinburgh Airport owes longer waiting times to the CrowdStrike incident, with wider reports suggesting departure boards are down across the site. Check-in services at Berlin airport are also impacted.
[6]
The list of victims is growing faster than our fingers can type. Other airports also confirmed to be affected include: Heathrow, Gatwick, Manchester, Stansted, Luton, airports across the entirety of Spain, and Swissport.
As if the UK's health service hasn't taken enough of a [7]battering in recent weeks, reports from individuals in Bristol writing into BBC Radio 4's Today program this morning, said they were unable to book doctor's appointments at their local general practitioner's (GP) office.
Putneymead GP surgery in West London has updated its website to say its core Medical Management System is affected and online requests are restricted. The phone lines are also down temporarily but the practice remains open.
[8]
Putneymead Group GP website outage message – click to enlarge
The Register asked the National Health Service (NHS) about this and in a statement it explained that IT issues have hit the EMIS system used by practices across the country.
"The NHS is aware of a global IT outage and an issue with EMIS, an appointment and patient record system, which is causing disruption in the majority of GP practices," it said.
[9]
"The NHS has longstanding measures in place to manage the disruption, including using paper patient records and handwritten prescriptions, and the usual phone systems to contact your GP.
"There is currently no known impact on 999 or emergency services, so people should use these services as they usually would.
"Patients should attend appointments unless told otherwise. Only contact your GP if it's urgent, and otherwise please use 111 online or call 111."
Speaking of critical services, various train lines are experiencing disruptions in the UK – namely those operated by Govia Thameslink, the UK's largest rail franchise which operates the Thameslink, Southern, Gatwick Express, and Great Northern lines.
A spokesperson said: "We apologize to customers for the disruption they're experiencing this morning. This is due to a worldwide IT issue affecting multiple companies and industries. Our advice to customers is to check our websites for the latest travel information and to check before they travel."
Other train lines affected include: Avanti West Coast, Great Western Railway, Hull Trains, Lumo, TransPennine, and West Midlands Rail.
Over in the US, reports suggest that some states' 911 emergency services are down. So far, it's believed that Ohio, Alaska, Arizona, Minnesota, Indiana, and New Hampshire are affected. Scary stuff.
And who could forget about the poor investment bankers unable to access their news service. Trading at the London Stock Exchange (LSE) is all okay – no worries there – but its regulatory news service (RNS) is down, so companies can't even hide their data breach reports behind the biggest news of the day.
So, what's going on?
The cause of the outages isn't entirely confirmed, but the prevailing explanation is that a faulty channel file in CrowdStrike Falcon – the vendor's flagship EDR solution trusted by organizations the world over – is to blame.
The [10]full advisory is available to paying CrowdStrike customers, however, the director of the vendor's managed hunting service OverWatch, Brody Nisbet, said the dodgy content update has been reversed.
It means there should be no new BSODs going forward, but it won't reverse the damage that's already been done – that'll be a job for IT admins who are sure to have the very worst of Fridays. Hopefully the work doesn't bleed into the weekend.
CrowdStrike hasn't yet responded to our requests for additional information, but Nisbet published a workaround via the vendor's dedicated Reddit page, a thread in which is [11]teeming with furious customers .
However, that workaround will not work for every customer, he [12]Xeeted while also calling the situation "a mess." The company is still working on the issue.
Speaking to BBC Radio 4 this morning, former CEO at the UK's National Cyber Security Centre, Ciaran Martin, concurred with the current explanations for the outage.
"[CrowdStrike] have a range of products under this brand they call Falcon, and their Falcon sensor update, which a lot of companies will use to detect threats and so forth, seems to have been misconfigured in such a way that it wrecks Windows.
"And so if a company is using both CrowdStrike and Windows for its operating system, it seems that they get what people in the trade call the blue screen of death, and Windows doesn't work, and that's why airlines aren't able to process, presumably, why Sky [News] hasn't been able to broadcast. It's also why, just simply for times and reasons, it seems to be emerging first in Australia.
"These complex systems always operate interdependently, so [for] the cybersecurity to do its job, the cybersecurity tool still has to be able to interact with Windows, so companies spend a lot of time, money, and effort on both sides of that equation, making sure that they're compatible when you're deploying these things. You have to make sure you don't destabilize other parts of the network, and most of the time that works. Occasionally, it doesn't – it appears that that's not the case. It's very rare for it to be as serious as this."
[13]CrowdStrike code update bricking Windows machines around the world
[14]Microsoft 365 remains 'degraded' as Azure outage resolved
[15]Elexon's Insight into UK electricity felled by expired certificate
[16]Internet Archive blames 'environmental factors' for overnight outages
CrowdStrike, not Microsoft
Early reporting from national media organizations misattributed the IT issues to Microsoft, which itself battled its own [17]outage hours earlier on its Azure cloud platform , but this appears unrelated to the underlying problem causing the widespread IT issues across the world.
This particular Azure issue was affecting Microsoft 365 subscription services but was already resolved by the time disruption scuppered global systems.
Microsoft found the time to respond to us briefly today, but continued to ignore our repeated requests for an explanation over [18]allegations that it failed to properly notify customers of a Russia-attributed data breach.
A spokesperson said: "We're aware of an issue affecting Windows devices due to an update from a third-party software platform. We anticipate a resolution is forthcoming."
Crowdstrike said in an update that is was "actively working with customers impacted by a defect found in a single content update for Windows hosts.
"Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
"The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
"We further recommend organizations ensure they're communicating with CrowdStrike representatives through official channels.
"Our team is fully mobilized to ensure the security and stability of CrowdStrike customers." ®
Updated to add:
Crowdstrike has added:
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
Windows hosts which are brought online after 0527 UTC will also not be impacted
Hosts running Windows 7/2008 R2 are not impacted
This issue is not impacting Mac- or Linux-based hosts
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Get our [19]Tech Resources
[1] https://regmedia.co.uk/2024/07/19/screenie.jpg
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZpqNmHKU1D9L6fr73UFd9QAAAhc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpqNmHKU1D9L6fr73UFd9QAAAhc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpqNmHKU1D9L6fr73UFd9QAAAhc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.co.uk/2024/07/19/crowdstrike_shares_sink_as_global/www.fly.faa.gov
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpqNmHKU1D9L6fr73UFd9QAAAhc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/06/20/qilin_our_plan_was_to/
[8] https://regmedia.co.uk/2024/07/19/putneymead_group.jpg
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpqNmHKU1D9L6fr73UFd9QAAAhc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://supportportal.crowdstrike.com/s/login/?ec=302&startURL=%2Fs%2Farticle%2FTech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19
[11] https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/
[12] https://x.com/brody_n77/status/1814187346641990010
[13] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[14] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/
[15] https://www.theregister.com/2024/07/09/elexons_insight_expired_cert/
[16] https://www.theregister.com/2024/07/08/internet_archive_suffers_a_wobble/
[17] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/
[18] https://www.linkedin.com/posts/kevin-beaumont-security_check-your-email-logs-including-exchange-activity-7215355395878305793-K8n_/
[19] https://whitepapers.theregister.com/
Quiet in the comments - is everyone busy firefighting?
The Reddit Crowdstrike thread has gone nuts with horror stories, poor BOFHs looking at 1000s of endpoints in boot loops. Glad we don't use it, bullet dodged even for our small footprint...
My condolences to those looking at RSI from entering zillions of bitlocker keys.
Had a call with a client; "can we reschedule to next week, we use Crowdstrike." = Weekend ruined
Re: Quiet in the comments - is everyone busy firefighting?
> My condolences to those looking at RSI from entering zillions of bitlocker keys.
Wait, the workaround trips BitLocker? Argh
So for some, CloudStrike has turned Microsoft into an inept but large ransomware gang?
Where's my BitLocker key? It's somewhere in AzureAD.. Which one is for this server? Err ..
Probably nothing. That was a medium term forecast.No forecast of a sudden drop.
There's something familiar about all of this...
I remember McAfee deploying an update that basically removed a key boot file from all windows machines (around the 2000's).
That was *not* a good day to be in IT.
McAfee have been on my shit list ever since.
Re: There's something familiar about all of this...
I Missed that one, but around that time McRappy put out a DAT that deleted excel.exe
Easily fixed when i got in, but they will always be McRappy to me for that.
Re: There's something familiar about all of this...
To be fair, deleting excel.exe is probably a good measure to improve organisational efficiency. I was literally only just having a conversation with someone about why Excel should be destroyed with fire, and then the ashes launched into the sun (and only then because there aren't any convenient black holes close enough by to dispose of it there).
Re: There's something familiar about all of this...
[1]https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-if-you-use-that-spreadsheet/
Anyone who disagrees with me can go along for the trip. I'm sorry it has come to this, but it's for the good of humanity.
[1] https://ludic.mataroa.blog/blog/i-will-fucking-dropkick-you-if-you-use-that-spreadsheet/
Re: There's something familiar about all of this...
Technically the Sun is our nearest, most available blackbody. Which for the required result would be just as effective.
Re: There's something familiar about all of this...
I want something with a Schwartzchild Radius to make sure there's no chance of information escape. You know, just to be on the safe side, just in case there's some massive solar prominence in the future that looks for all the world like it spells out "VLOOKUP" in the sky.
Re: There's something familiar about all of this...
Have an up vote and a pint on me for the Schwartzchild Radius quote.
Made me laugh and God knows I need it just now with a seven month old Malinois puppy running me ragged.
Oh, and I'm very glad that I am retired and outside the blast zone for this balls-up.
Re: There's something familiar about all of this...
Seems visionary. I bet there was no safer day for Windows machines than that.
I'm guessing CrowdStrike's share price is not the only thing that is going to be tanking in the coming weeks.
The fallout from this is starting to look like it will draw blood. CloudStrike's client list is going to shrink worse than a banana in the desert sun.
> CloudStrike's client list is going to shrink worse than a banana in the desert sun.
And the Client List on the class-action suit will be growing like ... an infestation of Tribbles in the cargo hold
.. Tribbles all-round for the lawyers
Re: There's something familiar about all of this...
I remember McAfee deploying an update that basically removed a key boot file from all windows machines (around the 2000's).
I was working for Logica at the time. It bricked most of the PCs and laptops in the company.
Fortunately, I was out of the office and offline that day, so I missed the update. And I made damn sure that the issue had been corrected before I reconnected to the company network!
Re: There's something familiar about all of this...
At some point any software that has rapid, regular automatic updates is going to screw up, it is only a matter of time.....and how bad it is.
Re: There's something familiar about all of this...
It depends on whether or not you take precautions. One might be to test before deploying, another might be to wait a day to see if any adverse reports roll in. I guess any Cloudstrike customers who adopted either approach won't be rolling it out today.
Re: There's something familiar about all of this...
Wonderful day... McAfee decided the NT Kernal was malware... mad rush round 000s of servers to put it back before someone rebooted
things that are running
are on Unix or Mainframes (remember them)!!
funny enough at £dayjob all the Mac users are able to carry on, with the exception that our files are stored on OneDrive/SharePoint due to IT locking our machines w/ JAMF so we're screwed too.
time to head to the garage to put new spark plugs in our very analogue Triumph car
Re: things that are running
Until 2038… when it will be a surprise for everybody (and nobody here).
Re: things that are running
2038 won't break mainframes running proprietary mainframe OSes like z/OS.
Re: things that are running
.. or z/VM (The mother of all Hypervisors) or indeed zLinux - still around.
Re: things that are running
Yes, 2038 is going to be really annoying for everybody still running Mac OS X 10.5 and earlier.
And 2040 is going to be really annoying for everybody still running Mac OS X 10.13 and earlier.
So... me (assuming I'm not dead yet) and other vintage Mac geeks. I mean, it's not that the machines won't boot, they just won't have correct dates. But most folks aren't likely to be running pre-2017 Mac OS in 2040 or pre-10.6 in 2038.
(It uses 64-bit dates now. So there's a Y292,277,026,596 problem. Humanity is unlikely to have to worry about it.)
Re: things that are running
"Humanity is unlikely to have to worry about it." That's the attitude that got us into the Y2K problems. /S
Re: things that are running
Well, I plan to be around in Y292,277,026,596 to find out one way or the other.
FWIW, I'd still be waiting for Beardienet customer service to answer the phone then.
Re: things that are running
"Until 2038… when it will be a surprise for everybody"
Not everybody - just you if don't realise that time_t has mostly - if not entirely - been upgraded to 64-bits already.
Re: things that are running
Triumph Herald, Stag, TRx?
Icon - Former grey haired Triumph owner wants to know.
> Early reporting from national media organizations misattributed the IT issues to Microsoft
Isn't this always the case? Even going way back in the 90s people would blame Windows for constantly BSOD'ing when in reality they decided to get the cheapest eMachines computer they could buy, made with capacitors that decided for themselves if they wanted to work that day or not.
Sure, let's pretend that it was eMachines hardware and not Windoze. Even though the BSOD was incredibly common on EVERY Windoze computer, it was all because of crappy eMachines capacitors.
"Isn't this always the case?"
It's the default assumption. What does that tell us?
If the shares were part-paid
the stock price would have the potential to go negative,
Re: If the shares were part-paid
It should be dropping even faster, it is all based on the ARR of subscriptions.
Payback time for automation
and an inevitable side-effect of centralization. Maybe not that costly, considering long term savings.
Clusterf*ck or what?
I find it both amusing and ironic that on the main page of CrowdStrike's website they have a pic of some SciFi baddy (looks like it is taken from some game art) along with the prophetic words:
"62 minutes could bring your business down.
That’s the average time it takes an adversary to land and move laterally through your network. When your data, reputation, and revenue are at stake, trust the pioneer in adversary intelligence."
Oh how right they are!
https://www.crowdstrike.com/en-gb/
Perhaps they should have replaced "an adversary" with "us".
Re: Clusterf*ck or what?
I mean, to be fair, if the computer is down and inaccessible then the hacker can't access your data...
Re: Clusterf*ck or what?
What's that old phrase that used to be about?
Ah, I know, "Single Point of Failure."
Surely we've got a bit better at IT since then.
The fault's with Microsoft
Yes the root cause is Crowdstrike. But let's be clear on this, the OS is Microsoft's. Why do we put up with an OS that can be felled by one program with a problem? Why can't the OS have a fallback to automatically roll back during a boot loop?
We've probably all seen Windows 10 or 11 fail to boot and then say "repairing" and more often than not just fail. This isn't really acceptable any longer. Microsoft needs to up their game and start making their OS reliable and resilient. It should not be possible for security software to break the system like it seems to have done for countless desktops and servers around the world.
Re: The fault's with Microsoft
It has to be possible for security software to break the system -- that's what it's supposed to do (in specific and limited ways) when it detects malware.
Without the ability to filter low-level system operations it can't detect and disable malware.
With that ability, any software defects can be disastrous.
Re: The fault's with Microsoft
Agreed, mostly... But to be fair I have seen Linux machines brought down by botched updates loads of times (often due to insufficient space in the /boot partition). I have also have many brushes with boot loops in both Android and iOS phones.
Re: The fault's with Microsoft
I can only refer you to Joe Tidy, the BBC "Cyber correspondent". He says They have "god-like" access to all the inner workings of an IT system for obvious reasons. .
Mind you, I don't think Joe is really a techie. His next paragraph tells us that "End Point Protection" programmes have to be able to monitor the inner workings of computers . Back in the last century it was not unknown for reactionary UK crusties to try to insist on spelling "program" that way, but I don't think I've seen it for at least 40 years.
Re: The fault's with Microsoft
Joe Tidy is a BBC Correspondent. He Writes for the Lusers. So, yeah, he's vaugewashing the reporting which is a bit infuriating but I'm not his audience.
Re: The fault's with Microsoft
But who watches the BBC any more?
Opps, sorry, I forgot. Mostly pensioners.
Re: The fault's with Microsoft
People who normally watch Sky?
Re: The fault's with Microsoft
"But who watches the BBC any more?"
People who want their news [1]as un-slanted as possible and as verified as possible . Yes, that makes it a bit boring (I go elsewhere to find out the rumours they're not telling me), but it is the most reliable and unbiased source we've got. And you – yes you, spitting your tea at the screen and laughing with incredulity – I challenge you to point me to a more unbiased general news source in the UK. It's not perfect by any means, but it's a start.
I expect downvotes.
[1] https://adfontesmedia.com/bbc-bias-and-reliability/
Re: The fault's with Microsoft
(And I'll count downvotes that aren't backed up by a post actually pointing me to a more unbiased general news source in the UK as a direct admission that you can't. Downvotes on this post doubly so.)
Re: The fault's with Microsoft
"Downvotes on this post doubly so"
With an invitation like that, what choice did I have?
Re: The fault's with Microsoft
GB News.
And you – yes you, spitting your tea at the screen and laughing with incredulity – I challenge you to wake your ass up.
Re: The fault's with Microsoft
Like you have? Spouting all the latest shit from twitter.
Re: The fault's with Microsoft
I'm going to assume this is deep trolling. There's simply [1]no other logical explanation .
[1] https://mediabiasfactcheck.com/gb-news-uk-bias/
Re: The fault's with Microsoft
When I were but knee high to a grasshopper a program was the new fangled computer thingy, and a programme was something you got at a theatre laying out the various acts. Similary, a programme was also a collection of activities that made up overarching initituative or agenda.
I blame the left-pondians for adopting simplified English as usual.
A programme can be applied to the various parts of a circus show... so perhaps it might be useful for the US to relearn this one following their election.
When I searched google news for Crowdstrike a couple of hours go i came across this [1]CrowdStrike stock could drop to $275 amid valuation concerns, analyst warns from yesterday - so what did Redburn Atlantic know?
[1] https://invezz.com/news/2024/07/18/crowdstrike-stock-could-drop-to-275-amid-valuation-concerns-analyst-warns/