Security vendor SolarWinds says product updates were subverted by nation-state, Fireye says exploit is rampant
(2020/12/14)
- Reference: 1607911847
- News link: https://www.theregister.co.uk/2020/12/14/solarwinds_fireeye_cozybear/
- Source link:
UPDATE Security vendor SolarWinds’ “Orion” IT monitoring platform has been compromised, and speculation is swirling that it was used in attacks on major US government agencies that could also be linked to last week’s [1]revalation that security vendor FireEye’s top hacking tools have been accessed.
A statement from Kevin Thompson, SolarWinds president and CEO says the company is "aware of a potential vulnerability which if present is currently believed to be related to updates which were released between March and June 2020 to our Orion monitoring products."
"We believe that this vulnerability is the result of a highly-sophisticated, targeted and manual supply chain attack by a nation state. We are acting in close coordination with FireEye, the Federal Bureau of Investigation, the intelligence community, and other law enforcement to investigate these matters. As such, we are limited as to what we can share at this time.”
As we report in our update to this story below, FireEye says it found the flaw in a compromised .dll file that was posted to the downloads section of SolarWinds' site.
The Register has asked SolarWinds for further detail, but evidence of updates in the relevant timeframe is not hard to find: here’s a [2]June 2020 patch to the company’s remote monitoring agent for Windows.
If you’re a SolarWinds customer, assume compromise and immediately activate your incident response team.
News of the SolarWinds hack was broken by newswire Reuters , which also reports that US government agencies, among them Treasury and the Department of Commerce, have been hit with a hack so serious that the National Security Council met to discuss it on Saturday.
The Washington Post has [3]reported that the government hacks were made possible by flaws in SolarWinds products and that the attack was perpetrated by Russian hacking group APT29, aka Cozy Bear. US government officials have acknowledged the incidents, but have not offered further details.
This situation is properly scary because a supply chain attack that poisons product updates issued by a major security vendor suggests that Cozy Bear could be deep inside all sorts of systems and vendors. If that doesn’t scare you, maybe SolarWinds’ [4]customer list will, as it mentions the following organisations are users.
More than 425 of the US Fortune 500
All ten of the top ten US telecommunications companies
All five branches of the US Military
The US Pentagon, State Department, NASA, NSA, Postal Service, NOAA, Department of Justice, and the Office of the President of the United States
All five of the top five US accounting firms
While the prospect of Cozy Bear rummaging around inside the abovementioned organisations is scary, security experts aren't panicking.
Security analyst Jake Williams has posted a [5]Twitter thread pointing out that products like Orion are a fine jumping-off point for an attack but points out that many such products are implemented to observe IT infrastructure performance rather than actively change configurations. He therefore urges readers not to assume the attack automatically translates to an ability to control systems.
[6]Former US Cybersecurity and Infrastructure Security Agency head Chris Krebs suggested the attack has likely been under way for months, but that it should be possible to contain.
"If you’re a SolarWinds customer & use the below product, assume compromise and immediately activate your incident response team," he advised. "Odds are you’re not affected, as this may be a resource intensive hack. Focus on your Crown Jewels. You can manage this."
As news breaks about what looks to be a pretty large-scale hack, I have the utmost confidence in the [7]@CISAgov team and other Federal partners. I'm sorry I'm not there with them, but they know how to do this. This thing is still early, I suspect. Let's let the pros work it. — Chris Krebs (@C_C_Krebs) [8]December 13, 2020
Hopefully, Krebs and Williams are correct. But even if they are, the fact remains that two big security vendors – FireEye and SolarWinds – have been revealed to be cracked and something appears to have taken a bite out of the US government. And all of these organisations boast of having strong defences against such attacks. ®
UPDATE: 03:50 UTC, Monday December 14th.
FireEye has posted an [9]analysis of the flaw in SolarWinds code that says the problem is present in a file called SolarWinds.Orion.Core.BusinessLayer.dll that it describes as a "digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third party servers."
FireEye says that once the .dll reaches a machine it remains dormant for up to two weeks, but then comes to life and "retrieves and executes commands, called 'Jobs', that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services."
"The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files allowing it to blend in with legitimate SolarWinds activity. The backdoor uses multiple obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers."
FireEye continues: "The trojanized update file is a standard Windows Installer Patch file that includes compressed resources associated with the update, including the trojanized SolarWinds.Orion.Core.BusinessLayer.dll component. Once the update is installed, the malicious DLL will be loaded by the legitimate SolarWinds executable SolarWind.BusinessLayerHost.exe or SolarWindws.BusinessLayerHostx64.exe (depending on system configuration)."
The malware then goes dormant for another fortnight before attempting to resolve a subdomain of avsvmcloud[.]com. "The DNS response will return a CNAME record that points to a Command and Control (C2) domain. The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications."
FireEye says it has "detected this activity at multiple entities worldwide."
"The victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East. We anticipate there are additional victims in other countries and verticals."
Long story short, this is a bad one and made worse by the fact that SolarWinds offers infrastructure monitoring but appears not to have been able to keep its own website and APIs clean.
Get our [10]Tech Resources
[1] https://www.theregister.com/2020/12/09/fireeye_tools_hacked/
[2] https://status.solarwindsmsp.com/2020/06/15/solarwinds-rmm-security-notice-regarding-an-agent-vulnerability-pre-v10-8-9/#more-19721
[3] https://www.washingtonpost.com/national-security/russian-government-spies-are-behind-a-broad-hacking-campaign-that-has-breached-us-agencies-and-a-top-cyber-firm/2020/12/13/d5a53b88-3d7d-11eb-9453-fc36ba051781_story.html
[4] https://www.solarwinds.com/company/customers
[5] https://twitter.com/MalwareJake/status/1338278185692246016
[6] https://www.theregister.com/2020/11/18/trump_fires_krebs/
[7] https://twitter.com/CISAgov?ref_src=twsrc%5Etfw
[8] https://twitter.com/C_C_Krebs/status/1338249463949893634?ref_src=twsrc%5Etfw
[9] https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
[10] https://whitepapers.theregister.com/
A statement from Kevin Thompson, SolarWinds president and CEO says the company is "aware of a potential vulnerability which if present is currently believed to be related to updates which were released between March and June 2020 to our Orion monitoring products."
"We believe that this vulnerability is the result of a highly-sophisticated, targeted and manual supply chain attack by a nation state. We are acting in close coordination with FireEye, the Federal Bureau of Investigation, the intelligence community, and other law enforcement to investigate these matters. As such, we are limited as to what we can share at this time.”
As we report in our update to this story below, FireEye says it found the flaw in a compromised .dll file that was posted to the downloads section of SolarWinds' site.
The Register has asked SolarWinds for further detail, but evidence of updates in the relevant timeframe is not hard to find: here’s a [2]June 2020 patch to the company’s remote monitoring agent for Windows.
If you’re a SolarWinds customer, assume compromise and immediately activate your incident response team.
News of the SolarWinds hack was broken by newswire Reuters , which also reports that US government agencies, among them Treasury and the Department of Commerce, have been hit with a hack so serious that the National Security Council met to discuss it on Saturday.
The Washington Post has [3]reported that the government hacks were made possible by flaws in SolarWinds products and that the attack was perpetrated by Russian hacking group APT29, aka Cozy Bear. US government officials have acknowledged the incidents, but have not offered further details.
This situation is properly scary because a supply chain attack that poisons product updates issued by a major security vendor suggests that Cozy Bear could be deep inside all sorts of systems and vendors. If that doesn’t scare you, maybe SolarWinds’ [4]customer list will, as it mentions the following organisations are users.
More than 425 of the US Fortune 500
All ten of the top ten US telecommunications companies
All five branches of the US Military
The US Pentagon, State Department, NASA, NSA, Postal Service, NOAA, Department of Justice, and the Office of the President of the United States
All five of the top five US accounting firms
While the prospect of Cozy Bear rummaging around inside the abovementioned organisations is scary, security experts aren't panicking.
Security analyst Jake Williams has posted a [5]Twitter thread pointing out that products like Orion are a fine jumping-off point for an attack but points out that many such products are implemented to observe IT infrastructure performance rather than actively change configurations. He therefore urges readers not to assume the attack automatically translates to an ability to control systems.
[6]Former US Cybersecurity and Infrastructure Security Agency head Chris Krebs suggested the attack has likely been under way for months, but that it should be possible to contain.
"If you’re a SolarWinds customer & use the below product, assume compromise and immediately activate your incident response team," he advised. "Odds are you’re not affected, as this may be a resource intensive hack. Focus on your Crown Jewels. You can manage this."
As news breaks about what looks to be a pretty large-scale hack, I have the utmost confidence in the [7]@CISAgov team and other Federal partners. I'm sorry I'm not there with them, but they know how to do this. This thing is still early, I suspect. Let's let the pros work it. — Chris Krebs (@C_C_Krebs) [8]December 13, 2020
Hopefully, Krebs and Williams are correct. But even if they are, the fact remains that two big security vendors – FireEye and SolarWinds – have been revealed to be cracked and something appears to have taken a bite out of the US government. And all of these organisations boast of having strong defences against such attacks. ®
UPDATE: 03:50 UTC, Monday December 14th.
FireEye has posted an [9]analysis of the flaw in SolarWinds code that says the problem is present in a file called SolarWinds.Orion.Core.BusinessLayer.dll that it describes as a "digitally-signed component of the Orion software framework that contains a backdoor that communicates via HTTP to third party servers."
FireEye says that once the .dll reaches a machine it remains dormant for up to two weeks, but then comes to life and "retrieves and executes commands, called 'Jobs', that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services."
"The malware masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files allowing it to blend in with legitimate SolarWinds activity. The backdoor uses multiple obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers."
FireEye continues: "The trojanized update file is a standard Windows Installer Patch file that includes compressed resources associated with the update, including the trojanized SolarWinds.Orion.Core.BusinessLayer.dll component. Once the update is installed, the malicious DLL will be loaded by the legitimate SolarWinds executable SolarWind.BusinessLayerHost.exe or SolarWindws.BusinessLayerHostx64.exe (depending on system configuration)."
The malware then goes dormant for another fortnight before attempting to resolve a subdomain of avsvmcloud[.]com. "The DNS response will return a CNAME record that points to a Command and Control (C2) domain. The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications."
FireEye says it has "detected this activity at multiple entities worldwide."
"The victims have included government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East. We anticipate there are additional victims in other countries and verticals."
Long story short, this is a bad one and made worse by the fact that SolarWinds offers infrastructure monitoring but appears not to have been able to keep its own website and APIs clean.
Get our [10]Tech Resources
[1] https://www.theregister.com/2020/12/09/fireeye_tools_hacked/
[2] https://status.solarwindsmsp.com/2020/06/15/solarwinds-rmm-security-notice-regarding-an-agent-vulnerability-pre-v10-8-9/#more-19721
[3] https://www.washingtonpost.com/national-security/russian-government-spies-are-behind-a-broad-hacking-campaign-that-has-breached-us-agencies-and-a-top-cyber-firm/2020/12/13/d5a53b88-3d7d-11eb-9453-fc36ba051781_story.html
[4] https://www.solarwinds.com/company/customers
[5] https://twitter.com/MalwareJake/status/1338278185692246016
[6] https://www.theregister.com/2020/11/18/trump_fires_krebs/
[7] https://twitter.com/CISAgov?ref_src=twsrc%5Etfw
[8] https://twitter.com/C_C_Krebs/status/1338249463949893634?ref_src=twsrc%5Etfw
[9] https://www.fireeye.com/blog/threat-research/2020/12/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor.html
[10] https://whitepapers.theregister.com/
Don't Panic Too Much, They're Surely Insured. Successfully Sue Bill, or George or Mike or Whoever?
amanfromMars 1
If you’re a SolarWinds customer, assume compromise and immediately activate your incident response team.
However, if you’re a SolarWinds customer, and they are your active security and incident response team and they have failed to protect servering of your crown jewels, with their value now decimated/reduced to single figure cents on the dollars, what else can you do other than panic a lot and hope they have a case to answer and you can still afford to sue them for whatever can be conjured up and thrown at them?
After all, is that not the American Way and the normal route to get to the root of such that matters?
Could be scarier than one might expect
Where I've seen Solarwinds implemented, the tool was often given administrative credentials to not just the networking gear to pull credentials (and restore them if they change without authorization), but also to perform discovery on Linux systems. Despite having well documented "these are the only sudo permissions needed on Linux", I see many shops just gave it full root.
Often, I was asking admins to choke down the access and there was surprise that the tool worked with less than full root.
So, yes, this is a very scary thing to me.