Google Cloud's watery Parisian outage enters third week, with no end in sight
(2023/05/10)
- Reference: 1683693013
- News link: https://www.theregister.co.uk/2023/05/10/google_cloud_paris_outage_persists/
- Source link:
Two weeks after the unwelcome "water intrusion" inside a Parisian Google Cloud datacenter, the stricken facility remains offline – with no indication when it might resume operations.
Google Cloud's europe-west9 region [1]took a shower on April 25. As datacenters and water don't mix, outages resulted.
It later emerged that the deluge did most damage at the europe-west9-a zone – one of three in the region.
[2]
Datacenter operator Global Switch admitted to an incident at a time that coincided with Google Cloud's woes.
[3]
[4]
"A fire incident has occurred in a room at one of the two datacenters in our Paris campus this morning," states an April 26 [5]announcement . Two days later, an [6]update revealed "the fire in one of the two datacenters in our Paris Campus was rapidly extinguished and was contained to one room only. The datacenter has remained live and operational with a very small proportion of its capacity affected."
[7]Irony alert: Major airport to be interrupted for two hours to replace UPS
[8]A right Royal pain in the Dallas: City IT systems crippled by ransomware
[9]European air traffic control confirms website 'under attack' by pro-Russia hackers
[10]Central UK govt awards £12M+ contract to leave Google Workspace for Microsoft 365
But Google Cloud's Europe-west9-a has been down ever since. La chocolaterie has not updated the advice it offers customers, which still reads as follows:
Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected datacenter. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources.
Why the delays? A French [11]forum thread about the incident suggests it was more than just a fire. A post claims that the incident started when a cooling pump malfunctioned, causing a leak. That leak reached the battery room, where it sparked a fire.
Even les sapeurs-pompiers [That's French for firefighters – Ed tread carefully around electrical fires, and this one generated a lot of heat. So much so that it was hours before attempts at extinguishment were possible and/or safe. That meant putting out the flames took hours.
The clean-up from that sort of incident would be substantial.
[12]
To make matters worse, last weekend Google Cloud's europe-west9-c zone had a significant wobble for nearly five hours. According to Google's [13]status report , the incident meant "Google Cloud products in europe-west9-c may not be available to customers."
Paris in the Spring is supposed to be a time of delights, not the gloom of cloudy outages that bring metaphorical gray skies to IT operations. But the latter is what Google's dishing up at present.
C'est la vie. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2023/04/26/google_cloud_outage/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.globalswitch.us/about-us/news/26-04-23-statement-in-relation-to-incident-in-our-paris-campus/
[6] https://www.globalswitch.us/about-us/news/28-04-23-update-to-the-statement-in-relation-to-incident-in-our-paris-campus/
[7] https://www.theregister.com/2023/05/08/ups_replacement_project_to_shut/
[8] https://www.theregister.com/2023/05/05/dallas_royal_ransomeare/
[9] https://www.theregister.com/2023/04/22/eurocontrol_russia_attack/
[10] https://www.theregister.com/2023/04/21/cabinet_office_m365_from_google_suite/
[11] https://lafibre.info/datacenter/incendie-maitrise-globalswitch-clichy/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://status.cloud.google.com/incidents/W8APSvPHcxDNyEKyTqJP
[14] https://whitepapers.theregister.com/
Google Cloud's europe-west9 region [1]took a shower on April 25. As datacenters and water don't mix, outages resulted.
It later emerged that the deluge did most damage at the europe-west9-a zone – one of three in the region.
[2]
Datacenter operator Global Switch admitted to an incident at a time that coincided with Google Cloud's woes.
[3]
[4]
"A fire incident has occurred in a room at one of the two datacenters in our Paris campus this morning," states an April 26 [5]announcement . Two days later, an [6]update revealed "the fire in one of the two datacenters in our Paris Campus was rapidly extinguished and was contained to one room only. The datacenter has remained live and operational with a very small proportion of its capacity affected."
[7]Irony alert: Major airport to be interrupted for two hours to replace UPS
[8]A right Royal pain in the Dallas: City IT systems crippled by ransomware
[9]European air traffic control confirms website 'under attack' by pro-Russia hackers
[10]Central UK govt awards £12M+ contract to leave Google Workspace for Microsoft 365
But Google Cloud's Europe-west9-a has been down ever since. La chocolaterie has not updated the advice it offers customers, which still reads as follows:
Our teams on the ground are assessing the physical infrastructure in the cluster. The current impacted product list has been updated. The impact for Cloud Bigtable continues in europe-west9-a. For the remaining products, impact is limited to instances located in the affected datacenter. Previously unaffected instances for these products will continue to work with no impact. There is no ETA for full recovery of affected instances in europe-west9-a at this time. We expect to see extended outages for these resources.
Why the delays? A French [11]forum thread about the incident suggests it was more than just a fire. A post claims that the incident started when a cooling pump malfunctioned, causing a leak. That leak reached the battery room, where it sparked a fire.
Even les sapeurs-pompiers [That's French for firefighters – Ed tread carefully around electrical fires, and this one generated a lot of heat. So much so that it was hours before attempts at extinguishment were possible and/or safe. That meant putting out the flames took hours.
The clean-up from that sort of incident would be substantial.
[12]
To make matters worse, last weekend Google Cloud's europe-west9-c zone had a significant wobble for nearly five hours. According to Google's [13]status report , the incident meant "Google Cloud products in europe-west9-c may not be available to customers."
Paris in the Spring is supposed to be a time of delights, not the gloom of cloudy outages that bring metaphorical gray skies to IT operations. But the latter is what Google's dishing up at present.
C'est la vie. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2023/04/26/google_cloud_outage/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.globalswitch.us/about-us/news/26-04-23-statement-in-relation-to-incident-in-our-paris-campus/
[6] https://www.globalswitch.us/about-us/news/28-04-23-update-to-the-statement-in-relation-to-incident-in-our-paris-campus/
[7] https://www.theregister.com/2023/05/08/ups_replacement_project_to_shut/
[8] https://www.theregister.com/2023/05/05/dallas_royal_ransomeare/
[9] https://www.theregister.com/2023/04/22/eurocontrol_russia_attack/
[10] https://www.theregister.com/2023/04/21/cabinet_office_m365_from_google_suite/
[11] https://lafibre.info/datacenter/incendie-maitrise-globalswitch-clichy/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFtrRZlU-vWG-BXUSVr-zAAAAM4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://status.cloud.google.com/incidents/W8APSvPHcxDNyEKyTqJP
[14] https://whitepapers.theregister.com/
Re: Cloud is it
emfiliane
Strip away all the hype, and the value of the cloud is betting that renting your equipment and the ability to scale up and down in five minutes is cheaper than buying it all outright plus provisioning and running costs. For greenfield projects, or a major upgrade, it's worth weighing the different paths.
But for some reason management always wants to replace the datacenter wholesale, despite all the sunk costs that have already gone into it.
A fire incident has occurred
Dr Who
No it hasn't. You're just trying to sound official or technical or something. What has occurred is a fire, not a "fire incident". Just like it's not a "flood event" it's a flood. And when did we move from having a storm to having a "severe weather event"? Anyway, gotta go, I had a curry last night and can feel a catastrophic evacuation event coming on.
Yumky
herman
Pardon me, but I’m not going to share a scone with a bird.
Anonymous Coward
And the benefits of relying on someone else's computer the cloud are what again?
Vader
The cloud should never go down right ?
Heat management
Potemkine!
How is water-cooling going?
"Google Cloud products in europe-west9-c may not be available to customers."
Missing Semicolon
An announcement that was not followed by "so we have redeployed all of the workloads to europe-west9-q (a bunch of containers in a car-park} so that service is maintained". Odd that. Almost as if they don't have to give a s**t.
Cloud is it
Yeah, as I've been saying, no its not.
"Google Cloud products in europe-west9-c may not be available to customers."
So why aren't they moved to a different region? Is it because you have to pay more for that?
No matter how much our MSP pressures us to go full cloud I don't think its a good idea. But no one listens to me.