Attack on direct debit provider London & Zurich leaves customers with 6-figure backlogs
- Reference: 1700740071
- News link: https://www.theregister.co.uk/2023/11/23/ransomware_attack_at_london_zurich/
- Source link:
London & Zurich's outage began on November 10 and was confirmed as a ransomware attack four days later on the company's website. The Register is the first to report on it, tipped off by sources impacted by the turn of events.
Since then, we're told customers have been unable to process the vast majority of their direct debit payments, with one managed service provider accruing a backlog of more than $124,000.
We know three other companies directly who all have the same issue as well and they've gone 'we've been left in the dark.' Nobody knows anything. One of those is in a very bad position – they've got no idea how they're going to make payroll
The UK-based MSP, which spoke to The Register , said on Wednesday it was able to process its first payment since the attack started, but said it was confused over when service would return to normal.
A particular pain point has been the communication from London & Zurich, which sources claim has communicated with customers infrequently and sometimes unclearly.
[1]
For example, the company's status page this week indicated that its direct debit portal should be up and running again by November 23, but emails seen by The Register have left customers confused after being told they won't be able to make collections until November 28.
[2]
[3]
According to [4]Google reviews left in the last week, customers were unable to reach any support services via the company's phone lines, with attempts often ringing out – an experience The Register also had when trying to make contact with London & Zurich.
The MSP said the duration of the outage is what has hit it the hardest. Usually a cash-rich operation, it recently made a large PAYE payment as well as putting a large deposit down on a new office building, making its inability to collect payments especially disruptive at this time.
[5]
"Thankfully, we should be alright," the MSP's director told The Register this week. "We've been looking at leveraging director money, we've been looking at leveraging money from the bank… so, we're in a lucky position that we can go to our lenders and [say] 'we need this as an interim measure.'
"Obviously, for a smaller company or for a company that's already debt-leveraged, this could have been hugely problematic, enough to put smaller companies under. The feedback from [London & Zurich] and the information on any estimated times for systems to go live is really horrendous.
"We've just been left entirely in the dark. We know three other companies directly who all have the same issue as well... Nobody knows anything. One of those is in a very bad position – they've got no idea how they're going to make payroll."
[6]
London & Zurich declined to answer The Register 's questions about whether any data was compromised during the [7]ransomware attack, how the attackers were able to breach its systems, what group was behind it, or when it started.
A spokesperson said: "L&Z recently suffered a ransomware incident. Upon learning about the incident, we immediately initiated an investigation with the assistance of third-party cybersecurity experts and took steps to contain the incident, including identifying and terminating access to the impacted servers.
"Only one environment was impacted by the incident and this has been rebuilt in a new, clean environment. This process is progressing at pace with our API service now fully functional and final testing taking place on two final service areas. We expect this restoration to finalize by the end of the week.
"We're grateful for the patience that our customers have shown us at this time and would ask that they continue to monitor our [8]status update page , which is updated daily, for further information."
The spokesperson also said in an email that the company's focus is recovery and supporting customers, and the investigation is still ongoing.
As of November 22, the MSP expressed frustration over the lack of firm commitments regarding the return to service. London & Zurich said it aims to be "back to normal by the end of this week" but couldn't commit to dates.
"If all systems are back online by Thursday 23rd, you will be able to create collections for the 28th November," the company said in an email to customers.
Based in Solihull, London & Zurich is one of the largest transactors of direct debits for businesses in the UK, serving small and large businesses including the Eden Project and ICPA.
It offers businesses an alternative to collecting direct debits through their banks, which require large indemnity payments. Companies like London & Zurich essentially act as middlemen, collecting direct debits from customers on behalf of businesses, rather than going directly from bank to bank.
How the incident unfolded
As of November 10 at 0919 local time, London & Zurich customers were experiencing "access issues" – terminology that was changed to "major service outage" less than two hours later. Customers at the time were warned that payment collections may be down until November 13.
The status page timeline was updated once a day over the next two days to say remediation work was continuing. There was no new information communicated on November 13, the planned return-to-service date, but a more substantial update came on the 14th when the company posted links to two separate web pages: A [9]general information page and a [10]security incident disclosure .
It told customers that the incident was ransomware in nature, that regulators had been informed, and third-party [11]incident responders were working on the case.
The case was confirmed as "contained" and the company started to rebuild the affected servers in "a new, clean environment as quickly and securely as possible," although it was taking longer than previously expected.
The incident impacted payment schedules dating back to the evening of November 8. Any schedule – a process that leads to a payment collection – made after 18:30 on that day was not backed up and would need resubmitting, the company said.
[12]Ransomware attacks register record speeds thanks to success of infosec industry
[13]Between ransomware and month-long engagements, IR teams need a hug – and a nap
[14]UK mulls making MSPs subject to mandatory security standards where they provide critical infrastructure
[15]Your password hygiene remains atrocious, says NordPass
On November 19 London & Zurich said direct debits for the period November 9-12 had been sent. Taking a few days to clear, this aligns with the reporting from the [16]MSP only receiving their first payment on November 22.
Scheduled payments for the period between November 14-22 are scheduled for same-day payment this Friday.
The company rotated customer [17]passwords which were sent out on November 21 in preparation for the customer portal to go back online on November 23. The online signup portal for new customers is also due to return by the end of the week. Any registrations made after 1830 UK time on November 8 will need to be resubmitted. ®
Get our [18]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZV@FNW3ldsSgNJIt-DDAJwAAAMY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZV@FNW3ldsSgNJIt-DDAJwAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZV@FNW3ldsSgNJIt-DDAJwAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.google.com/maps/place/London+%26+Zurich/@52.3881828,-1.9100804,17z/data=!3m1!5s0x4870bf1d8c09935f:0xe3366f1ef8650e38!4m8!3m7!1s0x4870bcf1131cbf97:0x64ababe5ffb664a7!8m2!3d52.3881828!4d-1.9075055!9m1!1b1!16s%2Fg%2F1hc3403m4?entry=ttu
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZV@FNW3ldsSgNJIt-DDAJwAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZV@FNW3ldsSgNJIt-DDAJwAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/10/10/ransomware_attacks_register_record_speeds/
[8] https://status.landz.co.uk/
[9] https://go.londonandzurich.co.uk/actions-to-take
[10] https://go.londonandzurich.co.uk/services-faqs
[11] https://www.theregister.com/2022/10/03/ibm_incident_reponder_survey/
[12] https://www.theregister.com/2023/10/10/ransomware_attacks_register_record_speeds/
[13] https://www.theregister.com/2022/10/03/ibm_incident_reponder_survey/
[14] https://www.theregister.com/2022/01/20/uk_nis_regulations_msp_plans/
[15] https://www.theregister.com/2023/11/20/your_password_hygiene_is_still/
[16] https://www.theregister.com/2022/01/20/uk_nis_regulations_msp_plans/
[17] https://www.theregister.com/2023/11/20/your_password_hygiene_is_still/
[18] https://whitepapers.theregister.com/
1) Don't operate your business so that a couple of week's delay in payment means you can't make payroll. It's the perfect way to tank a business overnight when all your staff just leave (and perfectly justified in doing so no matter what their contracts say by that point).
2) Have a backup provider wherever possible.
3) If you were banking through a bank and you can't pay direct debits because the provider is down, it would be entirely on the bank to sort that out - or arrange funds for you in the meantime.
4) If your financial provider was compromised - change provider at the first opportunity anyway.
Personally, at this point I'd be on the phone to alternative suppliers and looking at how to change those direct debits to come from another account, and I would only be dealing with L&Z to claw back whatever I'd paid them so far.
Ideally is super hard
Ideally you’d want a process where you could nuke the lot and rebuild it from scratch in a new clean environment and then ingest your backup data and carry on from near where you left off and then focus on the period of time you where down.
Reality is that it’s super hard to do something like that.
Even the big players would struggle.
Question is why is no one doing things in ways that disruption is minutes not months?
Availability, resiliency and backups was always what we strove for on prem but the cloud providers seem to have ditched that and gone for profit instead at the cost of their customers reputation.
Sales pitches always bamboozle the customers and promise uptime & resilience they ultimately never deliver on.
A daily snapshot (actually should only need one every upgrade) of their servers should be enough to get them back.
A proper backup policy of their data should be enough to roll back the clock enough to get going.
All that assumes a structured workflow and not a cobbled together barely working solution that they likely run as their super agile.
Feel for their customers and their customers customers
Dear God, just make it stop!
looking at leveraging director money, we've been looking at leveraging money from the bank…
.......a company that's already debt-leveraged,
If they talk like that, they think like that, which bodes far from well.
If only it were possible to prepare some sort of plan for a quick recovery in the event of an unforseen disaster...
DR or PR. You don't just _get_ to choose, you _have_ to!
Fully tested backups and affordability
So often backups are not tested. Some get lucky and have a good backup, then realize the time to restore is far more than anticipated. Proper backups should be tailored to the workloads and that includes the restore process and include a consideration for tolerable downtime.
The smaller customers who may be at risk of going under are with L&Z because of affordability, but maybe they cannot afford direct debit at all. But it is all the rage.
Judging from FAQs linked
They don't know if any the customer's customers data has been accessed