News: 1692775571

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Criminals go full Viking on CloudNordic, wipe all servers and customer data

(2023/08/23)


CloudNordic has told customers to consider all of their data lost following a ransomware infection that encrypted the large Danish cloud provider's servers and "paralyzed CloudNordic completely," according to the IT outfit's online confession.

The intrusion happened in the early-morning hours of August 18 during which miscreants shut down all of CloudNordic's systems, wiping both company and customers' websites and email systems. Since then, the IT team and third-party responders have been working to restore punters' data — but as of Tuesday, it's not looking great.

"We cannot and do not want to meet the financial demands of the criminal hackers for ransom," CloudNordic said in an [1]online notice , translated from Danish.

[2]

"Unfortunately, it has proved impossible to recreate more data, and the majority of our customers have thus lost all data with us," the alert continued. "This applies to everyone we have not contacted at this time."

[3]

[4]

The self-proclaimed "Nordic cloud experts" said they reported the intrusion to the police.

And while none of this is good news to organizations that have now lost all of their website and email data, CloudNordic does offer a slight silver lining: the biz doesn't believe that the criminals exfiltrated any information before encrypting the systems.

[5]

"We have seen no evidence of a data breach," the cloud provider said, adding:

We have not seen the attackers have had access to the data content of the machines themselves, but to administration systems from which they could encrypt entire disks. Very large amounts of data were encrypted, and we have seen no signs that large amounts of data have been attempted to be copied out.

CloudNordic says its "best estimate" is that the infection happened as servers were being moved from one datacenter to another.

Some of the machines were apparently infected before the move, and during the transfer servers that had been on separate networks were all connected to CloudNordic's internal network. This gave the intruders access to both the central administrative systems, storage, replication backup system and secondary backups, all of which they promptly encrypted for extortion.

[6]Ivanti Sentry exploited in the wild, patches emitted

[7]Leak of 75k employee records was insiders' fault, claims Tesla

[8]FYI: There's another BlackCat ransomware variant on the prowl

[9]Don't just patch your Citrix gear, check for intrusion: Two bugs exploited in wild

As of today, the CloudNordic said it's ready to get customers' web and email servers — without data — back online, albeit without DNS at present. To restore these services, the firm says to email: support@azero.dk with the word RESTORE in the subject line.

In the body of the email, include your email address, phone number, and domain, and CloudNordic will send you login details for a new website and email service.

However, the provider notes that it will take a "massive amount of time" to restore all of these services, even without data, and as such it encourages "critically affected" customers to find new providers "to minimize your downtime."

Or, there's the DIY option, which is the "fastest method to get DNS working again for your domain," CloudNordic said. Customers can find [10]detailed instructions for both options in the ransomware notification. ®

Get our [11]Tech Resources



[1] https://www-cloudnordic-com.translate.goog/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZOXYw9XOQsGAXAW7ZGXPVgAAAQs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZOXYw9XOQsGAXAW7ZGXPVgAAAQs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZOXYw9XOQsGAXAW7ZGXPVgAAAQs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZOXYw9XOQsGAXAW7ZGXPVgAAAQs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2023/08/22/critical_ivanti_mobileiron_sentry/

[7] https://www.theregister.com/2023/08/21/breach_of_75k_employee_records/

[8] https://www.theregister.com/2023/08/18/microsoft_spots_new_blackcat_ransomware/

[9] https://www.theregister.com/2023/08/17/citrix_mft_exploit/

[10] https://www-cloudnordic-com.translate.goog/?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp

[11] https://whitepapers.theregister.com/



Where are the backups?

b0llchit

...told customers to consider all of their data lost...

And the backups? Or,... we forgot to make any backups because our customers did not want to pay or we never tested a restore procedure or do not believe in backups?

Oh yes, it is "cloud" and all will be well and peachy. Guess it rained, then froze and the sun evaporated both ice, water, cloud and the business(es) in one go.

Re: Where are the backups?

wolfetone

Why is it the company's responsibility to make backups of the customer's data? It's the customer's responsibility, and any customer who doesn't believe that to be the case deserves everything they get.

Why is it the company's responsibility to make backups of the customer's data?

JimmyPage

Depends what they thought they were buying, really.

It's not unfair to want to offload the work and expertise required to to backups to a 3rd party, in exchange for a fee.

Indeed, almost all cloudy storage outfits make this a selling point.

Re: Why is it the company's responsibility to make backups of the customer's data?

wolfetone

If CloudNordic provided a back up service then yes, it's on them. But you are not meant to have just one back up for data resilience. In a physical setting you are meant to have an onsite back up, and an off site backup. If you're operating solely in the cloud you should have a back up away from the provider because it's not enough to just rely on the provider's backup.

Re: Why is it the company's responsibility to make backups of the customer's data?

SVD_NL

These days there's a plethora of companies offering cloud to cloud backup services. They are fully automated and tend to be rather cheap (think €2-3 per user per month for email and cloud file storage).

It's still the cloud, but what are the odds of two major cloud providers being affected at the same time?

I guess you could extend it with backups to an on-site server on a monthly or weekly basis.

People just don't realize how vulnerable the cloud can be.

Re: Where are the backups?

Filippo

The company is definitely responsible for providing the service they have been contracted for, though. They are clearly not doing it - I don't know what they're supposed to serve, but I doubt wiped servers can serve it. I suspect they are also responsible for storing the customer's data, which they are also not doing.

And the reason they are not providing the service is no (or vulnerable) backups.

The customers' customers , of course, will probably complain about CloudNordic's customers (their own service providers) not having their own backups, and they would also be right, for much the same reasons.

Re: Where are the backups?

Lil Endian

Why is it the company's [CloudNordic's] responsibility to make backups of the customer's data?

If a CloudNordic customer borks their own data, that's on the customer. If CloudNordic borks customer data, that's on the CloudNordic.

This case is the latter, and the onus is on CloudNordic. They fucked up.

[Edit: It's CloudNordic's own data that they failed to adequately protect and by extension, their customers' data.]

Re: Where are the backups?

Filippo

According to the article, the criminals managed to also get all the backups. If I understand correctly, there was an insecure server trasfer procedure that resulted in a window during which everything was connected at the same time, and the attackers exploited that. Apparently, there were no offline backups.

The cloud: someone else's computer.

Re: Where are the backups?

b0llchit

Then they have misunderstood the principle of backup , which is supposed to be immutable !

Re: Where are the backups?

Bebu

I am guessing there were before the migration there were carefully separated backup and/or archive systems but unfortunately the front door was left open to Mr Cock-Up...

Technically I imagine you might hold that the data and backups were still there (just add decryption key) although about as useful as dehydrated water.

Some good will have come from this cock-up if those, who ought to have known better, now understand the backup you don't directly control and can test, and do test, is no backup at all.

Re: Where are the backups?

Lars

I don't think this article included all information, and the backup part is certainly important.

While I have no experience of clouds I must admit i have assumed a cloud provider always have backups.

Re: Where are the backups?

Graham Cobb

It''s not just about backups because of hardware faults, fat-fingers, etc. In the case where you are using any cloud services, that provider could stop working at any second, without warning, for no apparent reason!

This case is one way that could happen but there are many others. The most likely, in my opinion, is a commercial issue: the company collapses, without warning to any customers, and ceases trading instantly. All data instantly inacessible - including any backups they hold.

For that reason, it is essential that if you use a cloud service, you have a disaster recovery plan which handles the cloud provider effectively disappearing into a wisp without warning.

Most importantly, if you contract with company A to run a service for you, make sure that you contract with a different company for the DR backups.

Re: Where are the backups?

teknopaul

Read the article.

It states clearly that backups were encrypted with ransomware.

Offline backups??

jmch

"This gave the intruders access to both the central administrative systems, storage, replication backup system and secondary backups, all of which they promptly encrypted for extortion."

So they had multiple backup systems, but all of them were online???

Re: Offline backups??

Jon 37

This is sadly common. It makes backups easier and faster. It makes restoring from backup easier and faster.

It also means that your backups offer no protection against ransomware or a hacker.

Re: Offline backups??

hertz

It appears that yes, all of them were online or at least all networked with internal systems during the move because no one could apparently forsee that could cause problems.

So, another cloud company that screws it up for all of its customers

Pascal Monett

Hey, CloudNordic, if I was one of your customers I wouldn't be worrying about getting my site back online with your help.

I'd be getting it back online with the help of a different provider.

You screw up in that magnitude, I vote with my wallet.

And Then ?

fg_swe

Do you seriously believe other providers are immune, by means of magic ?

Just a few days ago one of the megaCloud providers had their "master key" stolen, which meant ALL servers could be read and changed.

A few years ago a smaller, but non-trivial cloud provider had their "management console" hacked and essentially ALL servers in the open.

The entire "cloud" idea looks questionable.

Re: And Then ?

Anonymous Coward

Cloud services are increasingly commoditized and compete on price. Offline backups involving LTO tape libraries or hard drives will be one of the first things the bean counters will cut.

Re: And Then ?

Lil Endian

The entire "cloud" idea looks questionable.

While I fully agree that the cloud is not the place for business critical systems and data, and sensitive data, this is not a case of "magic". CloudNordic failed in basic security principals by their own admission:

During the work of moving servers from one data center to the other, servers that were previously on separate networks were unfortunately wired to access our internal network that is used to manage all of our servers.

Unfortunately? Try negligently. They put all of their eggs in one chicken coop, and the fox was already in residence. Granted, it's early days so things might change, but it's unlikely that a company would make such a statement on a whim.

Segmentation, anyone?

Mike 137

" Some of the machines were apparently infected before the move, and during the transfer servers that had been on separate networks were all connected to CloudNordic's internal network. "

Quite apart from the $64k question -- why the infected machines were not detected before or during the move, did anyone consider network security at all or was the 'internal network' effectively a giant hub? And if the machines were originally segregated (presumably for good reason), why was this not maintained?

Anonymous Anti-ANC South African Coward

Ouch, surely this must count as a very expensive lesson for both customer and service provider...

We do use online backups (quicker backups and restores) but we do store our data on offline media as well, but financial data is also stored online, offline and in the cloud.

Gone are the days when a simple backup to tape was good enough.

God, I ask for patience -- and I want it right now!