CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
- Reference: 1721605878
- News link: https://www.theregister.co.uk/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
- Source link:
Red Hat in June [1]warned its customers of a problem it described as "Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process" that impacted some users of Red Hat Enterprise Linux 9.4 after (as the warning suggests) booting on kernel version 5.14.0-427.13.1.el9_4.x86_64.
A [2]second issue titled "System crashed at cshook_network_ops_inet6_sockraw_release+0x171a9" advised users "for assistance with troubleshooting potential issues with the falcon_lsm_serviceable kernel module provided from the CrowdStrike Falcon Sensor/Agent security software suite." Red Hat also advised that "disabling the CrowdStrike Falcon Sensor/Agent software suite … will mitigate the crashes and provide temporary stability to the system in question while the issue is investigated." The issue was "Observed but not limited to release 6 and 7."
[3]
We've also spotted reports of CrowdStrike being suspected of [4]causing problems in [5]Debian and [6]Rocky Linux .
[7]
[8]
Linux Kernel panics and Windows Blue Screens of Death are broadly comparable. The occurrence of kernel panics mere weeks before CrowdStrike broke many Windows implementations therefore hints at wider issues at the security vendor.
The Register has asked CrowdStrike to comment on the issues identified by Red Hat, and will update this story if we receive substantial information.
CrowdStrike's CEO oversaw very similar McAfee meltdown
In 2010, PCs around the world [9]crashed after antivirus vendor McAfee pushed a bad update that left PCs in an endless reboot cycle.
At the time, McAfee's chief technology officer was George Kurtz – who now serves as CEO of CrowdStrike.
Kurtz therefore has the possibly unique and almost-certainly-unwanted distinction of having presided over two major global outage events caused by bad software updates.
Rapid restore tool on the way
CrowdStrike on Sunday [10]teased a rapid recovery tool for the mess it made.
"Together with customers, we tested a new technique to accelerate impacted system remediation," the security vendor stated on LinkedIn, adding "We're in the process of operationalizing an opt-in to this technique. We're making progress by the minute."
[11]
That progress will likely be of great interest, as Microsoft veep for enterprise and OS security David Weston on Saturday [12]estimated that 8.5 million Windows machines had been laid low by the problem. That's less than one percent of all Windows devices in operation, though a lot of the ones affected obviously were in critical environments.
Microsoft also created a repair tool that runs from a bootable USB storage device and can be found [13]here , along with instructions for use. Those instructions were modified on Sunday to require a full wipe of the USB device "so it doesn't error out when used in the recovery process."
CrowdStrike [14]published technical details of the incident. It has also offered [15]guidance on how to recover Windows machines encrypted with BitLocker.
[16]
Former Microsoft operating system developer David Plummer has shared his dissection of the flawed CrowdStrike update [17]here .
[18]CrowdStrike shares sink as global IT outage savages systems worldwide
[19]Cybercriminals quickly exploit CrowdStrike chaos
[20]Angry admins share the CrowdStrike outage experience
[21]Life, interrupted: How CrowdStrike's patch failure is messing up the world
Up in the air
The extent of disruption caused by CrowdStrike remains uncertain, but we've read accounts of [22]over 6,800 flights cancelled last Friday alone, and of some airlines only restoring systems [23]on Sunday evening .
The British Medical Association has [24]warned that "normal service cannot be resumed immediately" by UK doctors, at least, due to the backlog caused by the outage.
Australia's home affairs minister Claire O'Neill has warned that remediation could take weeks.
This remains a developing story: The Register will update this item, or write others, as further info emerges. ®
Get our [25]Tech Resources
[1] https://access.redhat.com/solutions/7068083
[2] https://access.redhat.com/solutions/7068083
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zp3ZX7r7VCRZOW9TskdtogAAAsU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://news.ycombinator.com/item?id=41005936
[5] https://lists.debian.org/debian-kernel/2024/04/msg00202.html
[6] https://forums.rockylinux.org/t/crowdstrike-freezing-rockylinux-after-9-4-upgrade/14041
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zp3ZX7r7VCRZOW9TskdtogAAAsU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zp3ZX7r7VCRZOW9TskdtogAAAsU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2010/04/21/mcafee_false_positive/
[10] https://www.linkedin.com/posts/crowdstrike_falcon-content-update-remediation-and-guidance-activity-7220896892623220737-lq8Y/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zp3ZX7r7VCRZOW9TskdtogAAAsU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/
[13] https://techcommunity.microsoft.com/t5/intune-customer-success/new-recovery-tool-to-help-with-crowdstrike-issue-impacting/ba-p/4196959
[14] https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/
[15] https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zp3ZX7r7VCRZOW9TskdtogAAAsU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.youtube.com/watch?v=wAzEJxOo1ts
[18] https://www.theregister.com/2024/07/19/crowdstrike_shares_sink_as_global/
[19] https://www.theregister.com/2024/07/19/cyber_criminals_quickly_exploit_crowdstrike/
[20] https://www.theregister.com/2024/07/19/admin_crowdstrike_update_mess/
[21] https://www.theregister.com/2024/07/19/life_interrupted_how_crowdstrikes_patch/
[22] https://simpleflying.com/6855-flights-canceled-it-outage/
[23] https://www.jetstar.com/au/en/travel-alerts#Update%20on%20global%20software%20issue
[24] https://www.bma.org.uk/bma-media-centre/gps-will-take-time-to-recover-from-crowdstrike-outage-says-bma
[25] https://whitepapers.theregister.com/
Gluttons for punishment
I would ask "I wonder how many customers they're going to lose because of this..."
But then I remember the fact people still use Windows and realize that answer would be "zero, or maybe one or two"
> But then I remember the fact people still use Windows and realize that answer would be "zero, or maybe one or two"
My own organisation doesn't use Crowdstrike, but from many other comments I've seen, many (especially local government, state government) orgs use it for compliance/insurance reasons. And it appears that Crowdstrike is the biggest player in this particular market. Therefore many orgs have little choice in whether they use Crowdstrike or not, it seems to be about the only player in town who can tick a compliance checkbox - which in itself to me seems to be a significant issue.
Of course, this may be a leg-up for other, smaller players ...
There are other options but a lot of DMs drink the cloud first, cloud native kool aid and think that these types of cloud back end applications are infallible.
Truth is that anything can get taken down whether by the elements, by malicious intent or by human error.
Always have a contingency plan to run your business without computing resources because you never know when the feces will collide with the thermantidote.
Should rebrand themselves as ClownStrike.
Why use Crowdstrike when you have SELinux?
SELinux is the ultimate security product for Linux-based machines. It was developed by the National Security Agency (NSA) to secure Government computers. It is maintained by Red Hat and available on most distros.
The SELinux philosophy is: if it's not explicitly allowed then it's blocked.
SELinux has pretty good threat detection as well. Every blocked action is logged and available to log monitors to start reacting in seconds if not milliseconds
CrowdStrike seems to have the philosophy of: letting everything run and if I hear of a problem I'll provide a fix
SELinux needs you to understand your systems and processes and enable functionality only when required
CrowdStrike lets you treat systems as black boxes and believe the man behind the curtain will make it all good.
The downside of SELinux is that you must know about the processes and systems you are administering and be patient while developing an optimal configuration. Funny that! I thought that was a basic requirement for systems security administrators. Never mind. With CrowdStrike, the man behind the curtain will make it all good, so you can hire cheap helpdesk staff to set up your systems.
Re: Why use Crowdstrike when you have SELinux?
Is there a Whinedoze version?
And this, ladies & gentlemen, is how you DDoS the entire world.
Several hacking groups would also like to thank CrowdStrike for the file (that caused the BSoD) -- It will be very handy.
This (demonstration) makes Petya & WannaCry(pt) look "pedestrian".
Yeah, this was fun...
Just drove from San Francisco to Dallas in two days flat to make it home in time for work. There were no flights to be had once mine canceled, and the drive was faster. Yuck.
“Kurtz therefore has the possibly unique and almost-certainly-unwanted distinction of having presided over two major global outage events caused by bad software updates.”
“Once is happenstance. Twice is coincidence. Three times is enemy action”
At the very least he should be on Oprah to explain himself.
Well just great
FUBAR all around!