Emergency services call-handling provider: Ransomware forced it to pull servers offline
- Reference: 1660309583
- News link: https://www.theregister.co.uk/2022/08/12/advanced_confirms_ransomware_forced_it/
- Source link:
The incident was spotted on 4 August and efforts to contain it resulted in server and network connections being taken offline, causing the loss of service on products used by Health & Care customers. Affected hosted products include Adastra, Caresys, Odyssey, Carenotes, Crosscare and Staffplan.
Some 36 customers from the UK's National Health Service (NHS) use services provided by Advanced, including NHS 111, which provides round-the-clock support such as health information. Adastra, for example, is said to work with 85 percent of NHS 111 Services, and call operators were forced to use pen and paper to keep things running.
[1]
The turn of events bore all the [2]signs of a serious security strike and in its [3]latest update on 10 August , Advanced confirmed it fell victim to "ransomware."
[4]
[5]
Third party forensic specialists at Mandiant and Microsoft DART teams are working with Advanced's techies to "ensure our systems are back online securely with enhanced protections."
Advanced said communication is also being maintained with the NHS, the National Cyber Security Centre (NCSC) and UK data watchdog the ICO.
[6]
"We want to stress that there is nothing to suggest that our customers are at risk of malware spread and believe that early intervention from our Incident Response Team contained this issue to a small number of servers," the update says.
No further issues have since been detected, the company added.
As for the way forward? Sources told us on 5 August they were informed that services may resume on 9 August but that was seemingly overly optimistic.
[7]
Advanced's update says: "We are rebuilding and restoring impacted systems in a separate and secure environment. To help all customers feel confident in reconnecting to our products once service is restored, we have implemented a defined process by which all environments will be systematically checked prior to securely bringing them online.
This process includes:
Implementing additional blocking rules and further restricting privileged accounts for Advanced staff;
Scanning all impacted systems and ensuring they are fully patched;
Resetting credentials;
Deploying additional endpoint detection and response agents; and
Conducting 24/7 monitoring.
Following this, Advanced will bring impacted infrastructure back online and reconnect services "as part of a phased return."
"With respect to the NHS, we are working with them and the NCSC to validate the additional steps we have taken, at which point the NHS will begin to bring its services back online.
[8]Major IT outage forces UK emergency call handlers to use 'pen and paper'
[9]Schneider Electric to sell Russian ops to local management
[10]Hospitals to use startup's AI tech to predict A&E traffic
[11]UK National Crime Agency finds 225 million previously unexposed passwords
"For NHS 111 and other urgent care customers using Adastra and NHS Trusts using eFinancials, we anticipate this phased process to begin within the next few days. For other NHS customers and Care organisations our current view is that it will be necessary to maintain existing contingency plans for at least three to four more weeks. We are working tirelessly to bring this timeline forward, and while we are hopeful to do so, we want our customers to be prepared. We will continue to provide updates as we make progress."
Advanced said it is the "early stages of our investigation into this incident" and has "not yet confirmed the root cause," which it admitted "may take time."
"With respect to potentially impacted data, our investigation is underway, and when we have more information about potential data access or exfiltration, we will update customers as appropriate. Additionally, we will comply with applicable notification obligations," it adds.
It thanked customers for their "continued patience", adding: "We fully understand the challenges this incident has caused for many of our stakeholders."
A security source close to the matter told us there are suggestions the criminals behind the ransomware could have been in Advanced's network for months, and that hundreds of millions of NHS records may have been captured.
We asked Advanced about this, and whether they are negotiating with the extortionists.
In response, the company sent us a statement from Simon Short, chief operating officer:
"We are continuing to make progress in our response to this incident. We are doing this by following a rigorous phased approach, in consultation with our customers and relevant authorities. We thank all our stakeholders for their patience and understanding as our team works around the clock to resume service as safely and securely as possible." ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YvZ5IwbTBDhx9Fn4djRLTQAAAI8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://www.theregister.com/2022/08/05/major_outage_at_it_service/
[3] https://www.oneadvanced.com/cyber-incident/#block451441
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YvZ5IwbTBDhx9Fn4djRLTQAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YvZ5IwbTBDhx9Fn4djRLTQAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YvZ5IwbTBDhx9Fn4djRLTQAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YvZ5IwbTBDhx9Fn4djRLTQAAAI8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/08/05/major_outage_at_it_service/
[9] https://www.theregister.com/2022/04/28/schneider_electric_to_sell_russian/
[10] https://www.theregister.com/2022/03/29/nhs_hospitals_ai/
[11] https://www.theregister.com/2021/12/21/nca_finds_255m_fresh_stolen_passwords/
[12] https://whitepapers.theregister.com/
National Hacking Survival?
If you are providing services online then you need to assume that they will be hacked like this, so maintain complete backups that are regularly maintained offline. That can help you restore services after you are hacked but it don't stop anything daily. When the Internet was created it was designed to be universally accessible ... these days we can see that this was a wonderful design feature originally, but nowadays it's a problem - restricted access would solve nothing but it could make the daily hacking attempts a bit harder and the defenses better.
Recovery plan?
Surely a service like this had a document recovery plan in place that would have as least some functionality back within a few hours and total recovery within 48 hours? Supported by regular disaster recovery exercises, of course.
No? Oh.
Makes you wonder who placed the order without checking stuff like this...
Re: Recovery plan?
Full recovery within 48 hours? I can bet that conversation would have gone something like this:
NHS: We want full recovery from any cyber attack in 48hrs please.
Bidder: ok, that'll cost this much.
NHS: We don't have anything near that much money.
Bidder: For that much we can do this.
NHS: ok. That'll have to do.
Government Minister: "We're putting not money than ever into public services, creating an NHS fit for the modern age blah blah blah tax cuts for all!"
Recovery redundant?
Yet another case in my experience where recovery is weeks/months.
Ransomware attacks are expected. No-one can be sure of thwarting every attack. Recovery from a complete network compromise must surely be part of any professional planning nowadays. The plan will have timeframes. Is anyone actually signing off any that don't have something like 48 hour to core re-functioning? A day to flush or replace existing systems - and another day to bring back core data?
Yet so many times it isn't happening. Some may be explainable because something outside of the expected happens. But not all. I suspect that having redundant hardware/people/licences and practising live recovery is a price many bean counters may pay lip service but when it comes to shove - today's emergency trumps next week's risk when it comes to budget.
And it's going to be expensive iif you need to retain existing kit for postmortem examination which implies to you need to bring up a parallel system. Redundancy big time,
Re: Recovery redundant?
Ransomware attacks are awkward. you have to be pretty certain that the recovery systems that you build are not coming from infected backups.
I'm not saying that they do this, but if I was someone wanting to place a ransomware bomb in a system, I'd probably want to install and spread it but leave the encryption dormant for several weeks, so that it would be copied onto the backups.
By doing this, you could probably immediately re-infect the environment that is being rebuilt, especially if it is just a timed trigger rather than an instruction from a command and control system external to the environment.
What I really struggle with is the fact that so many environments appear to be easy to infect. I know that the malware probably involves privilege escalation as well as the ransom encryption, but in a properly segmented environment, you should be able to contain an infection before it spreads. But I suppose the rush to consolidate systems into easy to manage large groups probably works against you there.
Re: Recovery redundant?
I think the answer is, don't rebuild the software from backups, build that from the original source. Only restore the data from backups.
Of course the software, even if it isn't infected, will still have the same vulnerability that allowed the original attack to happen, so you need to identify and fix it.
Recovery redundant?
When it comes to redundant, that's not how government understands that word.
They were originally blaming the heatwave for this, which I thought was a bit strange.
I get that heatwaves can cause servers to overheat and shutdown, but I couldn't understand how they weren't able to just switch it back on once it cooled down.
Now I understand.
Is it coincidental that I have today received my first fake/scam Covid proximity alert? The same message has already been reported multiple times in the past few hours.