Rackspace runs short of Cloud Files storage in LON region
- Reference: 1701347413
- News link: https://www.theregister.co.uk/2023/11/30/rackspace_runs_short_of_cloud/
- Source link:
[1]According to the company , problems began on November 17 at 1545 UTC, as some customers experienced 503 errors when attempting to access their files.
On November 24 – yes, a week later – the company said its engineers increased storage capacity in the region. However, problems have continued this week. A Register reader, working for a SaaS biz that uses Rackspace's Cloud Files product, told us: "Since Monday we've been seeing a big jump in errors as they tried to upload files, and have had to delay a release because we can't publish assets to their CDN.
[2]
"To be clear, some operations succeed, but we're seeing between 1 and 6 percent fail in any given hour, and it's not getting any better."
[3]
[4]
By November 28, Rackspace told customers - in a communication seen by The Reg - that its engineers continued working and introducing additional storage "at a deliberate pace" to avoid worsening the situation. It said "customers should be able to access already uploaded content but may experience issues processing new uploads or delete actions."
As of November 29, more storage was added – and will continue to be added – to, in the words of Rackspace, "restore the environment's health."
[5]
The company said: "Engineers continue to recommend that customers retry upload and delete actions as needed as this is the only workaround currently available."
[6]How much to clean up a ransomware infection? For Rackspace, about $11M
[7]Facing a 30% price rise to park servers in a colo? Blame AI
[8]Rackspace datacenter infrastructure took 12-hour nap in London, Sydney, Hong Kong
[9]Rackspace racks up job cuts amid market downturn and talk of offshoring
The embarrassing event is yet another reminder of the fallibility of cloud storage, although unlike the [10]problems reported with Google Drive earlier this week, Rackspace has at least admitted that, yes, it isn't your imagination. Something has gone wrong.
However, the question remains – how did this happen? Storage needs are relatively easy to predict when paying customers are involved. Cloud Files itself is powered by OpenStack. [11]Rackspace says of the product : "Whether your storage needs are modest - or monumental - you enjoy built-in redundancy."
Unless, it seems, the disk fills up.
The Register contacted Rackspace for comment on the incident and will update this piece should the company respond. ®
Get our [12]Tech Resources
[1] https://rackspace.service-now.com/system_status
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZWi-tUqbFW7CAMqLeDP8cQAAAJY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWi-tUqbFW7CAMqLeDP8cQAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZWi-tUqbFW7CAMqLeDP8cQAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWi-tUqbFW7CAMqLeDP8cQAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/11/16/rackspace_ransomware_expenses/
[7] https://www.theregister.com/2023/09/25/ai_datacenter_price_rise/
[8] https://www.theregister.com/2023/05/30/rackspace_outage/
[9] https://www.theregister.com/2023/03/28/rackspace_racks_up_the_redundancies/
[10] https://www.theregister.com/2023/11/27/google_drive_files_disappearing/
[11] https://docs.rackspace.com/docs/cloud-files
[12] https://whitepapers.theregister.com/
Re: The fallibility of cloud storage ?
Odd. Everything I've been told (well, at least the believable stuff) is that the cloud is simply using someone else's computers.
All the "cloudy" stuff is just a Simple Matter of Programming. And networking.
Re: The fallibility of cloud storage ?
Cloud means that what's going on at someone else's computer is clouded and you can't see whether they are adding new drives or not.
" However, the question remains – how did this happen? "
Well, there are two scenarios.
The first is that the needle on their storage gauge just stuck at a certain point, they tapped the glass then it suddenly spun round and went beyond red.
The second is that they have the same sort of monitoring tools as they had at Chernobyl, and the reader only went so high.
> The second is that they have the same sort of monitoring tools as they had at Chernobyl
I read somewhere that one such nuclear power station used a strategically placed mirror, so as the operator could see one of the dials behind the main console.
That's not what you pay a lot of money to an outside company for.
Should never have got within 90% of capacity, let alone to the point where it errors and active traffic can keep filling it back up.
Why have regions?
If your data is encrypted, you can store it anywhere safely. Properly encrypted, NATO could store their files on a server in Moscow or Beijing. That's what encryption is for.
Re: Why have regions?
Because GDPR effectively requires me to have control of my data and never be processed anywhere that's not subject to GDPR or similar regulations.
Yes, storing or moving encrypted data around is "processing".
Though I'm a mathematician and computer scientist and agree with you on this (you should be able to publish all your traffic and your public key on the front page of every newspaper and it not matter), legal requirements mean that you just can't do that for anything that is considered "personal data".
Guess what? Much of the data any company, business, charity, school or organisation stores is "personal data".
And the brief of "processing" is so broad that, no, I can't transmit my data to the US, even temporarily.
What about...
Have they tried clearing the contents of the TEMP folder? Or emptying the Recycle Bin?
Move to something else
If they are only provisioning storage when the disks are full, it means that they didn't place an order for a few million quid's worth of SSDs whilst the array was 80% full. Which means that; they could not afford it (get out now!); their budgetary control system prioritizes bonuses over reliability (get out now) or they actually had no idea that disk space was getting low, because it's too hard to compute or their monitoring is poor (get out now).
Re: Move to something else
I know someone who worked for Rackspace (and Google, and a lot of other places).
I think they'd tell you that it was likely all three.
I didn't realise Rackspace was still a thing ... Oh well maybe not for much longer.
I always thought the writing was on the wall for them when I saw them advertising their expertise to help you with your AWS cloud journey or some such similar marketing speak. Quite the statement from the once mighty Rackspace that they will help you build on someone else's cloud.
I guess this sort of event here is what happens when you mess up your overcommitted storage..... Eeek
The fallibility of cloud storage ?
This is news to me. Everything we've been told up to now was: move to the "The Cloud". "The Cloud" is more reliable and cheaper then the old fashioned solution.