UniSuper Google Cloud outage caused by an unfortunate series of events
- Reference: 1715252407
- News link: https://www.theregister.co.uk/2024/05/09/unisuper_google_cloud_outage_caused/
- Source link:
In a [1]joint statement with UniSuper CEO Peter Chun, Kurian admitted that an "inadvertent misconfiguration" during the provisioning of UniSuper's Private Cloud services resulted in the deletion of the subscription.
In a cascade of catastrophe familiar to anyone using duplication, the deletion of the account resulted in deletion across other regions.
[2]
"UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies."
[3]
Fortunately, UniSuper had backups at another cloud provider. Otherwise, a bad situation could have been oh so much worse. As it is, it has only been since today that the funds' services have shown signs of life, and members have been able to log into their accounts. The organization is also further ahead in the restoration than initially planned, meaning that balances should be up to date.
[4]Google Cloud blunder sinks Australian fund for a week
[5]Techie's enthusiasm for decluttering fails to spark joy
[6]Techie saved the day and was then criticized for the fix
[7]Microsoft slammed for lax security that led to China's cyber-raid on Exchange Online
The joint statement states, "This is an isolated, 'one-of-a-kind occurrence' that has never before occurred with any of Google Cloud's clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again."
The Register contacted Google to learn more about this "one-of-a-kind-occurrence," but we were simply directed to the joint statement.
In the meantime, UniSuper's woes remain a lesson for companies leaping cloudwards. Someone clicking the wrong button, a previously unknown bug, an unforeseen series of events, or a combination of all three could have dire consequences for a business. ®
Get our [8]Tech Resources
[1] https://www.unisuper.com.au/contact-us/outage-update
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZjzzIpQZv8RqBP@7wxpNfgAAAIs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/paasiaas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZjzzIpQZv8RqBP@7wxpNfgAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2024/05/08/google_cloud_misconfiguration_takes_australian/
[5] https://www.theregister.com/2024/05/06/who_me/
[6] https://www.theregister.com/2024/04/05/on_call/
[7] https://www.theregister.com/2024/04/03/cisa_microsoft_exchange_online_china_report/
[8] https://whitepapers.theregister.com/
Re: Someone else's - Abend's Observation
I really liked commenter Abend0c4's observation:
"Many cloud systems are actually just distributed single points of failure"
"UniSuper had duplication in two geographies as a protection against outages and loss"
Using the same account on the same provider -- a classic predictable single point of failure. Even if they'd used two separate accounts on Goo cloud, the problem would not have existed.
When will we finally learn what redundancy really means?
Re: "UniSuper had duplication in two geographies as a protection against outages and loss"
> When will we finally learn what redundancy really means?
From the article (which you seem to have not actually read)
"Fortunately, UniSuper had backups at another cloud provider"
Strange anti-cloud emphasis
"In the meantime, UniSuper's woes remain a lesson for companies leaping cloudwards. Someone clicking the wrong button, a previously unknown bug, an unforeseen series of events, or a combination of all three could have dire consequences for a business."
There is absolutely no difference in this, between being on own estate or in the cloud. Either way, someone can goof up, or a natural disaster can happen, or a systems failure, and bad stuff occurs.
Like any sensible and well-prepared company, UniSuper's IT had mitigated against as much of the risk as humanly possible. Not only had they replicated across two regions, they were replicating to a second supplier. If they were using own estate, doubtless they would have done the same, with replication across two geographically well-separated data centres and use of a separate backup domain, which is exactly analoguous, except it would have involved a heck of a lot more CapEx and having to manage multiple redundant network links through different providers, to avoid a single point of failure between the sites.
And yet this is an excuse to bash cloud use? Weird.
Someone else's
At the end of the day, cloud is just someone else's computer stored away somewhere you don't have access to.
Then someone you don't know can just go in and delete your data, because they had a bad day or they were tripping still after a wild weekend with mates.
You don't know if they just given in to their intrusive thoughts telling them to write 'rm -rf /home' or whatever.
You also don't know if someone had a fight with a hubby over the phone whilst cancelling someone's subscription and you just happened to have a similar name.