How did a CrowdStrike config file crash millions of Windows computers? We take a closer look
(2024/07/23)
- Reference: 1721767932
- News link: https://www.theregister.co.uk/2024/07/23/crowdstrike_failure_shows_need_for/
- Source link:
Analysis Last week, at 0409 UTC on July 19, 2024, antivirus maker CrowdStrike released an update to its widely used Falcon platform that caused Microsoft Windows machines around the world to crash.
The impact was extensive. Supply chain firm Interos [1]estimates 674,620 direct enterprise customer relationships of CrowdStrike and Microsoft were affected. Microsoft [2]said 8.5 million Windows machines failed. The results beyond a [3]massive amount of IT remediation time included global flight and shipping delays due to widespread Windows system [4]failures .
The cause, to the extent so far revealed by CrowdStrike, was "a logic error resulting in a system crash and blue screen (BSOD) on impacted systems."
[5]
That crash stemmed from quite possibly mangled data that somehow found its way into a Falcon configuration file called a Channel File, which controls the way CrowdStrike's security software works.
[6]
[7]
Channel Files are updated over time by CrowdStrike and pushed to systems running its software. In turn, Falcon on those machines uses information in the files to detect and respond to threats. This is part of Falcon's behavioral-based mechanisms that identify, highlight, and thwart malware and other unwanted activities on computers.
In this case, a configuration file was pushed to millions of Windows computers running Falcon that confused the security software to the point where it crashed the whole system. On rebooting an affected box, it would almost immediately start up Falcon and crash all over again.
[8]
According to CrowdStrike, Channel Files on Windows machines are stored in the following directory:
C:\Windows\System32\drivers\CrowdStrike\
The files use a naming convention that starts with "C-" followed by a unique identifying number. The errant file's name in this case started with "C-00000291-", followed by various other numbers, and ended with the .sys extension. But these are not kernel drivers, according to CrowdStrike; indeed, they are data files used by Falcon, which does run at the driver level. That is to say, the broken configuration file was not a driver executable but it was processed by CrowdStrike's highly trusted operating-system-level code, and when the bad file caused that code to go off the rails, it brought down the whole operating system – Microsoft Windows in this saga.
[9]
"Channel File 291 controls how Falcon evaluates named pipe execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows," CrowdStrike explained in [10]a technical summary published over the weekend.
The configuration update triggered a logic error that resulted in an operating system crash
"The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash."
Translation: CrowdStrike spotted malware abusing a Windows feature called named pipes to communicate with that malicious software's command-and-control (C2) servers, which typically instruct the malware to perform all sorts of bad things. CrowdStrike pushed out a configuration file update to detect and block that misuse of pipes, but the config file broke Falcon.
While there has been speculation that the error was the result of null bytes in the Channel File, CrowdStrike insists that's not the case.
"This is not related to null bytes contained within Channel File 291 or any other Channel File," the cybersecurity outfit said, promising further root cause analysis to determine how the logic flaw occurred.
Specific details about the root cause of the error have yet to be formally disclosed – CrowdStrike CEO George Kurtz has just been asked to testify before Congress over this matter – though security experts such as Google Project Zero guru [11]Tavis Ormandy and Objective-See founder [12]Patrick Wardle , have argued convincingly that the offending Channel File in some way caused Falcon to access information in memory that simply wasn't present, triggering a crash.
It appears Falcon reads entries from a table in memory in a loop and uses those entries as pointers into memory for further work. When at least one of those entries was not correct or present, as a result of the config file, and instead contained a garbage value, the kernel-level code used that garbage as if it was valid, causing it to access unmapped memory.
That bad access was caught by the processor and operating system, and sparked a BSOD because at that point the OS knows something unexpected has happened at a very low level. It's arguably better to crash in this situation than attempt to continue and scribble over data and cause more damage.
Wardle told The Register the crash dump and disassembly make it clear that the crash arose from trying to use uninitialized data as a pointer – a wild pointer – but further specifics remain unknown.
"We still don’t have the exact reason, though, why the channel file triggered that," he said.
The Register spoke with cybersecurity veteran Omkhar Arasaratnam, general manager of [13]OpenSSF , about how things fell apart.
Arasaratnam said the exact cause remains a matter of speculation because he doesn't have access to the CrowdStrike source code or the Windows kernel.
[14]CrowdStrike CEO summoned to explain epic fail to US Homeland Security
[15]CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
[16]CrowdStrike Windows patchpocalypse could take weeks to fix, IT admins fear
[17]Angry admins share the CrowdStrike outage experience
CrowdStrike's Falcon software, he said, has two components: A digitally signed, Microsoft-approved driver called [18]CSAgent.sys and the Channel Files used for updating the software with the latest security information.
"What CrowdStrike did is they essentially have a driver that's signed that then loads a bunch of these channel configurations," said Arasaratnam. "We don't know what the channel configuration file actually entails. It's a combination of what's in the file, as well as how CSAgent.sys interprets that."
Based on one stack trace, he explained, CSAgent.sys gets terminated for performing what's known as a bad pointer dereference. It tried to access memory from the address 0x000000000000009c , which didn't exist.
"It was an area of memory that it shouldn't have had access to," said Arasaratnam.
"Now, the Catch-22 you get into when you have a very low-level program like this, is the kernel overall is supposed to be responsible for the operating system doing many low-level things, including allocating memory," Arasaratnam said.
"So if the kernel is trying, in essence, is trying to access memory that it shouldn't access, the appropriate thing to do, just from an operating system theory perspective, is to assume that none of the memory that's been allocated is safe, because if the kernel doesn't know, who the heck does, and basically halt the system."
Do you know more about how the crash happened? Contact us [19]directly here or anonymously via the [20]instructions here .
The situation was complicated by the way the Windows driver architecture works, Arasaratnam explained.
"The way that it works is that drivers can set a flag called boot-start," he said.
"So normally, if you've got a driver that's acting kind of buggy and causes a failure like this, Windows can auto resume by simply not loading the driver the next time. But if it is set as boot-start, which is supposed to be reserved for critical drivers, like one for your hard drive, Windows will not eliminate that from the startup sequence and will continue to fail over and over and over and over again, which is what we saw with the CrowdStrike failure."
(We believe the reason why Microsoft [21]recommended people reboot affected Windows virtual machines on Azure as many as 15 times to clear the problem is because there was a small chance each time that the errant config file would be automatically updated to a non-broken one before the CSAgent.sys driver started parsing it. After multiple reboots, you would eventually win that race condition.)
Arasaratnam said that beyond that, we won't know how the Channel File update that told CSAgent.sys to do a bad pointer dereference managed to pass quality assurance (QA).
"It seems obvious that something slipped past QA given the frequency with which the crash occurred," he said. "It seems like even a trivial amount of QA would have caught this. This isn't some edge case where it's like one in 1,000 machines, right?"
Arasaratnam said there are several best practices that should have been observed. One is not to run software in kernel mode if you can help it. Second, is to ensure that QA is more thorough. Third, is to do what Google does by deploying incremental Canary releases.
He explained, "One of the techniques employed by Google, which we used when I was there, is to do what's called Canary releases – gradual or slow rollouts – and observe what's occurring rather than crashing what Microsoft estimated were 8.5 million machines." ®
Get our [22]Tech Resources
[1] https://www.interos.ai/blog-crowdstrike-impact-analysis-at-least-674000-enterprise-customers-at-risk-of-disruption-globally/
[2] https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/
[3] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[4] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/
[11] https://x.com/taviso/status/1814762315587461495
[12] https://threadreaderapp.com/thread/1814343502886477857.html
[13] http://www.openssf.org/
[14] https://www.theregister.com/2024/07/23/crowdstrike_ceo_to_testify/
[15] https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
[16] https://www.theregister.com/2024/07/19/crowdstrike_windows_kettle/
[17] https://www.theregister.com/2024/07/19/admin_crowdstrike_update_mess/
[18] https://www.virustotal.com/gui/file/fc17c021f18ec73d1544ad46dde6a1f1949f126bf3e75f97e241f982e2b07c86/details
[19] mailto:tclaburn@theregister.com
[20] https://www.theregister.com/Profile/contact/
[21] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[22] https://whitepapers.theregister.com/
The impact was extensive. Supply chain firm Interos [1]estimates 674,620 direct enterprise customer relationships of CrowdStrike and Microsoft were affected. Microsoft [2]said 8.5 million Windows machines failed. The results beyond a [3]massive amount of IT remediation time included global flight and shipping delays due to widespread Windows system [4]failures .
The cause, to the extent so far revealed by CrowdStrike, was "a logic error resulting in a system crash and blue screen (BSOD) on impacted systems."
[5]
That crash stemmed from quite possibly mangled data that somehow found its way into a Falcon configuration file called a Channel File, which controls the way CrowdStrike's security software works.
[6]
[7]
Channel Files are updated over time by CrowdStrike and pushed to systems running its software. In turn, Falcon on those machines uses information in the files to detect and respond to threats. This is part of Falcon's behavioral-based mechanisms that identify, highlight, and thwart malware and other unwanted activities on computers.
In this case, a configuration file was pushed to millions of Windows computers running Falcon that confused the security software to the point where it crashed the whole system. On rebooting an affected box, it would almost immediately start up Falcon and crash all over again.
[8]
According to CrowdStrike, Channel Files on Windows machines are stored in the following directory:
C:\Windows\System32\drivers\CrowdStrike\
The files use a naming convention that starts with "C-" followed by a unique identifying number. The errant file's name in this case started with "C-00000291-", followed by various other numbers, and ended with the .sys extension. But these are not kernel drivers, according to CrowdStrike; indeed, they are data files used by Falcon, which does run at the driver level. That is to say, the broken configuration file was not a driver executable but it was processed by CrowdStrike's highly trusted operating-system-level code, and when the bad file caused that code to go off the rails, it brought down the whole operating system – Microsoft Windows in this saga.
[9]
"Channel File 291 controls how Falcon evaluates named pipe execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows," CrowdStrike explained in [10]a technical summary published over the weekend.
The configuration update triggered a logic error that resulted in an operating system crash
"The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash."
Translation: CrowdStrike spotted malware abusing a Windows feature called named pipes to communicate with that malicious software's command-and-control (C2) servers, which typically instruct the malware to perform all sorts of bad things. CrowdStrike pushed out a configuration file update to detect and block that misuse of pipes, but the config file broke Falcon.
While there has been speculation that the error was the result of null bytes in the Channel File, CrowdStrike insists that's not the case.
"This is not related to null bytes contained within Channel File 291 or any other Channel File," the cybersecurity outfit said, promising further root cause analysis to determine how the logic flaw occurred.
Specific details about the root cause of the error have yet to be formally disclosed – CrowdStrike CEO George Kurtz has just been asked to testify before Congress over this matter – though security experts such as Google Project Zero guru [11]Tavis Ormandy and Objective-See founder [12]Patrick Wardle , have argued convincingly that the offending Channel File in some way caused Falcon to access information in memory that simply wasn't present, triggering a crash.
It appears Falcon reads entries from a table in memory in a loop and uses those entries as pointers into memory for further work. When at least one of those entries was not correct or present, as a result of the config file, and instead contained a garbage value, the kernel-level code used that garbage as if it was valid, causing it to access unmapped memory.
That bad access was caught by the processor and operating system, and sparked a BSOD because at that point the OS knows something unexpected has happened at a very low level. It's arguably better to crash in this situation than attempt to continue and scribble over data and cause more damage.
Wardle told The Register the crash dump and disassembly make it clear that the crash arose from trying to use uninitialized data as a pointer – a wild pointer – but further specifics remain unknown.
"We still don’t have the exact reason, though, why the channel file triggered that," he said.
The Register spoke with cybersecurity veteran Omkhar Arasaratnam, general manager of [13]OpenSSF , about how things fell apart.
Arasaratnam said the exact cause remains a matter of speculation because he doesn't have access to the CrowdStrike source code or the Windows kernel.
[14]CrowdStrike CEO summoned to explain epic fail to US Homeland Security
[15]CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes
[16]CrowdStrike Windows patchpocalypse could take weeks to fix, IT admins fear
[17]Angry admins share the CrowdStrike outage experience
CrowdStrike's Falcon software, he said, has two components: A digitally signed, Microsoft-approved driver called [18]CSAgent.sys and the Channel Files used for updating the software with the latest security information.
"What CrowdStrike did is they essentially have a driver that's signed that then loads a bunch of these channel configurations," said Arasaratnam. "We don't know what the channel configuration file actually entails. It's a combination of what's in the file, as well as how CSAgent.sys interprets that."
Based on one stack trace, he explained, CSAgent.sys gets terminated for performing what's known as a bad pointer dereference. It tried to access memory from the address 0x000000000000009c , which didn't exist.
"It was an area of memory that it shouldn't have had access to," said Arasaratnam.
"Now, the Catch-22 you get into when you have a very low-level program like this, is the kernel overall is supposed to be responsible for the operating system doing many low-level things, including allocating memory," Arasaratnam said.
"So if the kernel is trying, in essence, is trying to access memory that it shouldn't access, the appropriate thing to do, just from an operating system theory perspective, is to assume that none of the memory that's been allocated is safe, because if the kernel doesn't know, who the heck does, and basically halt the system."
Do you know more about how the crash happened? Contact us [19]directly here or anonymously via the [20]instructions here .
The situation was complicated by the way the Windows driver architecture works, Arasaratnam explained.
"The way that it works is that drivers can set a flag called boot-start," he said.
"So normally, if you've got a driver that's acting kind of buggy and causes a failure like this, Windows can auto resume by simply not loading the driver the next time. But if it is set as boot-start, which is supposed to be reserved for critical drivers, like one for your hard drive, Windows will not eliminate that from the startup sequence and will continue to fail over and over and over and over again, which is what we saw with the CrowdStrike failure."
(We believe the reason why Microsoft [21]recommended people reboot affected Windows virtual machines on Azure as many as 15 times to clear the problem is because there was a small chance each time that the errant config file would be automatically updated to a non-broken one before the CSAgent.sys driver started parsing it. After multiple reboots, you would eventually win that race condition.)
Arasaratnam said that beyond that, we won't know how the Channel File update that told CSAgent.sys to do a bad pointer dereference managed to pass quality assurance (QA).
"It seems obvious that something slipped past QA given the frequency with which the crash occurred," he said. "It seems like even a trivial amount of QA would have caught this. This isn't some edge case where it's like one in 1,000 machines, right?"
Arasaratnam said there are several best practices that should have been observed. One is not to run software in kernel mode if you can help it. Second, is to ensure that QA is more thorough. Third, is to do what Google does by deploying incremental Canary releases.
He explained, "One of the techniques employed by Google, which we used when I was there, is to do what's called Canary releases – gradual or slow rollouts – and observe what's occurring rather than crashing what Microsoft estimated were 8.5 million machines." ®
Get our [22]Tech Resources
[1] https://www.interos.ai/blog-crowdstrike-impact-analysis-at-least-674000-enterprise-customers-at-risk-of-disruption-globally/
[2] https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/
[3] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[4] https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqAn@UoNkbdkUgDzsMBs@wAAAEw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.crowdstrike.com/blog/falcon-update-for-windows-hosts-technical-details/
[11] https://x.com/taviso/status/1814762315587461495
[12] https://threadreaderapp.com/thread/1814343502886477857.html
[13] http://www.openssf.org/
[14] https://www.theregister.com/2024/07/23/crowdstrike_ceo_to_testify/
[15] https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
[16] https://www.theregister.com/2024/07/19/crowdstrike_windows_kettle/
[17] https://www.theregister.com/2024/07/19/admin_crowdstrike_update_mess/
[18] https://www.virustotal.com/gui/file/fc17c021f18ec73d1544ad46dde6a1f1949f126bf3e75f97e241f982e2b07c86/details
[19] mailto:tclaburn@theregister.com
[20] https://www.theregister.com/Profile/contact/
[21] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[22] https://whitepapers.theregister.com/
doublelayer
This article explains, if you didn't already know, why Windows has to go down when code which is running as part of the kernel breaks this badly. Guess what would happen if a kernel module I loaded into Linux, Mac OS, or any other operating system had a memory violation. That's right, it would panic. It is required to panic. If it did not panic, that kernel has a serious reliability problem.
Until people understand that, the attempts to find a reason why Microsoft is to blame here will not work. Maybe you or someone else can actually find a thing that Microsoft should be doing differently related to this, but while people continue to post comments trying to blame it for doing something both standard and necessary, you will fail to make any case because it appears that you have a gap in important systems knowledge.
david 12
RedHat had exactly the same problem with a CrowdStrike causing a kernel panic after loading a bad channel file.
If you want a resilient system that can't be taken down by drivers that are marked as required for boot, then you want a different kind of machine architecture, not a different OS.
Canary releases?
W.S.Gosset
Is that the official new name? I just used to call them phased releases, back in the day. Or just common sense.
Re: Canary releases?
Doctor Syntax
The term's been around for a while, if not in relation to releases, in other contexts. About 10 years ago it became a practice fo post assurances that a business had not been serves a subpoena by a given date. Failure to update it was an indication that it had received one without breaching any terms the subpoena may have contained forbidding an announcement that it had. The origin, of course, is a comparison with the coal-miner's canary which would be more susceptible to carbon monoxide poisoning than the miner - not a close analogy with the warrant canary but it fits well with a sacrificial S/W instance which can be exposed to a pending update.
Re: Canary releases?
david 12
The canary releases were one part of the phased release system. Canary was the name used for that phase, somebody thought it was more informative than just "phase 3" or whatever.
How did this happen? It seems to me that it happened because CrowdStrike’s quality engineering and release procedures are nowhere near fit for purpose. But it also happened because Microsoft still only pays lip service to security on Windows. CrowdStrike shouldn’t have released what they did - but equally it shouldn’t have been able to take down Windows.