On the 12th day of the Rackspace email disaster, it did not give to me …
- Reference: 1671062113
- News link: https://www.theregister.co.uk/2022/12/14/rackspace_email_outage/
- Source link:
In the service provider's most recent [1]update , posted at 0844 Eastern Time on Wednesday, Rackspace said it had hired CrowdStrike to investigate the fiasco, and noted it continues "to make all of our internal and external resources available to provide support to the remaining Hosted Exchange customers."
Rackspace did not, however, say if or when it expects to recover people's data that was lost or scrambled when ransomware hit its systems – an [2]attack that took down some of Rackspace's hosted Microsoft Exchange services on December 2. Since then, affected customers have been unable to get at their data held in the hosted service.
[3]
"We understand how important data recovery is to our customers," Rackspace wrote. "In ransomware attacks, data recovery efforts do necessarily take significant time, both due to the nature of the attack and need to follow additional security protocols. We will continue to keep you updated on these efforts."
[4]
[5]
Here's a flavor of the customer sentiment right now:
Not a single comment on the FAQ saying why you havent restored from backups? Do you not have any? If not why did you not have them conforming to best practice? — JimtheITGuy (@JimtheITguy) [6]December 14, 2022
The company also claimed to have transitioned more than two thirds of its customers to Microsoft 365, and, as it has in [7]previous updates , Rackspace urged customers to migrate their users and domains to this environment.
"As a reminder, if you have not yet transitioned to Microsoft 365 or have not fully completed the transition, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064 (INTL: +44 (0) 203 917 4743)," the update said. "Wait times continue to average less than 30 minutes."
Some users, however, say it's [8]much longer . One Reg reader, Erin Lutz, told us that she hung up after being on hold for two hours and 40 minutes.
[9]
"If you are a Rackspace user, you have likely migrated to something else now to restore email to your domain," she said in an email to El Reg , adding that the biz's support team hasn't been very supportive. She told us:
If you want your email history prior to Sunday 12/4 or whenever it was that you moved to something else, you have to lob in a support request to Rackspace – and they advise that this email history may or may not return at some point – or you have the option of PAYING Rackspace's partner Barracuda Networks for the privilege of getting your email history restored. So basically, along with the hack of your data that may be out there in some nefarious users' hands due to the Rackspace ransomware hack, Rackspace users are also being extorted by Rackspace to get their own email history back.
Rackspace did not respond to The Register 's inquiries about these claims, or anything else related to the ongoing outage.
Barracuda Networks also did not respond to The Register 's questions about whether Rackspace customers have to pay to get their email history restored.
[10]Rackspace confirms ransomware attack behind days-long email meltdown
[11]Rackspace customers rage as email outage continues and migrations create migraines
[12]Rackspace rocked by 'security incident' that has taken out hosted Exchange services
[13]This ransomware gang is a right Royal pain in the AES for healthcare orgs
The hosting company still hasn't said how many customers were affected by the ransomware infection. But in today's update – and in a [14]December 9 filing with the US Securities and Exchange Commission – Rackspace said CrowdStrike confirmed the intrusion was limited to the hosted Microsoft Exchange environment.
"CrowdStrike has also confirmed that there have been no signs of attacker activity in the Hosted Exchange environment since the ransomware attack on December 2, 2022," Rackspace noted in today's update. "We are also continuing to support the FBI's investigation into the attack."
Previously, Rackspace has said that its hosted Exchange email business comprises about one percent of its total annual revenue. In a December 6 SEC filing, the cloud outfit said the attack "may result in a loss of revenue" to this part of the biz, which brings in about $30 million annually. "In addition, the company may have incremental costs associated with its response to the incident," it [15]noted in its Form 8-K. ®
Get our [16]Tech Resources
[1] https://status.apps.rackspace.com/index/viewincidents?group=2
[2] https://www.theregister.com/2022/12/03/rackspace_security_incident_hosted_exchange/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y5qp87umdSG-fk-fcMNjiQAAAEM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5qp87umdSG-fk-fcMNjiQAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y5qp87umdSG-fk-fcMNjiQAAAEM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://twitter.com/JimtheITguy/status/1603105215040786437?ref_src=twsrc%5Etfw
[7] https://www.theregister.com/2022/12/06/rackspace_confirms_ransomware/
[8] https://twitter.com/ravendreibelbis/status/1600201316013289473
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y5qp87umdSG-fk-fcMNjiQAAAEM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2022/12/06/rackspace_confirms_ransomware/
[11] https://www.theregister.com/2022/12/05/rackspace_hosted_exchange_security_update/
[12] https://www.theregister.com/2022/12/03/rackspace_security_incident_hosted_exchange/
[13] https://www.theregister.com/2022/12/09/royal_ransomware_hhs_warning/
[14] https://www.sec.gov/ix?doc=/Archives/edgar/data/1810019/000119312522301429/d429186d8k.htm
[15] https://www.sec.gov/ix?doc=/Archives/edgar/data/1810019/000119312522298940/d388117d8k.htm
[16] https://whitepapers.theregister.com/
Only insofar
as it relives me from having to administer it myself. I hope those that signed off on this have a signed piece of paper with someone elses name on it pointing out that they warned the company of the potential consequences of the "all-our-eggs-in-the-cloud" approach. The more cynical among us might have been happy to leave it at that.
Those that cared about continuity of business and the welfare of the company at all shouldn't have been using hosting that they couldn't back up to somewhere outside the providers cloud. An human enough mistake, but a teachable example for managers and IT drones alike. This stuff isn't academic, and if a third party loses your data, they can just leave you twisting in the wind, regardless of contracts or promises.
Even an ironclad SLA with sharp teeth won't make your data reappear if it has been destroyed. Neither will I told you so's or excueses. Only a working copy of your data can save you. So make sure the number and location of the copies matches the importance of the data, don't take other peoples word for it, and test the restore workflows before you need to use them for real if you aren't a twisted masochist.
We all should know this, most of us say it, and we all need to actually be doing it.
Right.
If you've ever been an e-mail admin or had to keep the wheels on even a medium sized exchange environment, you probably got all gushy inside when the various cloud vendors offered to allow you to wipe that booger on them.
E-mail and particularly exchange environments have been a huge vector and target forever. Hosting your own does not make it any safer than putting it in someone else data center. It does, however, let you point the finger at them instead of updating your resume.
"In ransomware attacks, data recovery efforts do necessarily take significant time, both due to the nature of the attack and need to follow additional security protocols."
Yes, it may take a couple years to infinity for the security engineers to reverse engineer the decryptor without paying the ransom.
Hey, but it's "the cloud" right? And a managed service provider?
That's gotta be better than hosting in-house, right? Right?