Google and Oracle clouds still affected by UK heatwave
(2022/07/20)
- Reference: 1658303826
- News link: https://www.theregister.co.uk/2022/07/20/google_oracle_cloud_heat_update/
- Source link:
Cloud [1]outages at Google and Oracle caused by the UK's heatwave have ended, but users have been warned some problems persist.
Google's [2]incident report states that the "cooling related failure" was resolved at 20:43 Pacific Time on Tuesday (03:43 UTC).
But the ad giant's update also reports "A small number of HDD backed Persistent Disk volumes are still experiencing impact and will exhibit IO errors."
[3]
That sounds like some hard disks cooked during the "cooling related failure". Hard disks are seldom rated to run at more than 50 degrees Celsius, and as the mercury topped 40 degrees in London yesterday it’s not hard to imagine that temperatures became so hot that mechanical disks in a densely packed device faced an extreme environment and suffered more than solid state components.
[4]
[5]
Google understands the frailties of spinning rust so will presumably have replicated data on multiple devices and understand how to recover the devices. If data is lost, it will dent the G-Cloud's reputation.
[6]Good news: Twitter fell over. Bad news: It's working again
[7]Power outage at Hiroshima facility may hit DRAM supply
[8]Massive telecom outage in Japan kicks 40 million mobile users offline
Oracle's most recent update is time-stamped 03:57:44 UTC and opens with news that "data center cooling infrastructure has been restored and temperatures have returned to normal operating levels" and adds "All services and customer resources have been restored."
But the update also states "a subset of Oracle Integration Cloud resources continue to experience impact. Engineers are actively working to mitigate those remaining service resources."
So maybe not all services have been restored.
[9]
Weather forecasts for the UK predict that parts of the nation hosting Google and Oracle datacenters will on Wednesday top out at a pleasant 27 degrees, but may experience a thunderstorm late in the day.
The Register hopes the big clouds have remembered to close their windows after yesterday's meltdowns.
Datacenters in other parts of the world routinely cope with temperatures experienced in the UK on Tuesday. But yesterday was London's first recorded day above 40 degrees – like, ever – so datacenter designers have a viable excuse for their creations not coping. It's not a brilliant excuse, though, given cloud operators and the companies that host them pride themselves on over-engineering to ensure resilience. ®
Get our [10]Tech Resources
[1] https://www.theregister.com/2022/07/19/google_oracle_cloud/
[2] https://status.cloud.google.com/incidents/XVq5om2XEDSqLtJZUvcH
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2022/07/14/twitter_down/
[7] https://www.theregister.com/2022/07/13/power_disruption_at_micron_facility/
[8] https://www.theregister.com/2022/07/04/massive_telecom_outage_in_japan/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://whitepapers.theregister.com/
Google's [2]incident report states that the "cooling related failure" was resolved at 20:43 Pacific Time on Tuesday (03:43 UTC).
But the ad giant's update also reports "A small number of HDD backed Persistent Disk volumes are still experiencing impact and will exhibit IO errors."
[3]
That sounds like some hard disks cooked during the "cooling related failure". Hard disks are seldom rated to run at more than 50 degrees Celsius, and as the mercury topped 40 degrees in London yesterday it’s not hard to imagine that temperatures became so hot that mechanical disks in a densely packed device faced an extreme environment and suffered more than solid state components.
[4]
[5]
Google understands the frailties of spinning rust so will presumably have replicated data on multiple devices and understand how to recover the devices. If data is lost, it will dent the G-Cloud's reputation.
[6]Good news: Twitter fell over. Bad news: It's working again
[7]Power outage at Hiroshima facility may hit DRAM supply
[8]Massive telecom outage in Japan kicks 40 million mobile users offline
Oracle's most recent update is time-stamped 03:57:44 UTC and opens with news that "data center cooling infrastructure has been restored and temperatures have returned to normal operating levels" and adds "All services and customer resources have been restored."
But the update also states "a subset of Oracle Integration Cloud resources continue to experience impact. Engineers are actively working to mitigate those remaining service resources."
So maybe not all services have been restored.
[9]
Weather forecasts for the UK predict that parts of the nation hosting Google and Oracle datacenters will on Wednesday top out at a pleasant 27 degrees, but may experience a thunderstorm late in the day.
The Register hopes the big clouds have remembered to close their windows after yesterday's meltdowns.
Datacenters in other parts of the world routinely cope with temperatures experienced in the UK on Tuesday. But yesterday was London's first recorded day above 40 degrees – like, ever – so datacenter designers have a viable excuse for their creations not coping. It's not a brilliant excuse, though, given cloud operators and the companies that host them pride themselves on over-engineering to ensure resilience. ®
Get our [10]Tech Resources
[1] https://www.theregister.com/2022/07/19/google_oracle_cloud/
[2] https://status.cloud.google.com/incidents/XVq5om2XEDSqLtJZUvcH
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2022/07/14/twitter_down/
[7] https://www.theregister.com/2022/07/13/power_disruption_at_micron_facility/
[8] https://www.theregister.com/2022/07/04/massive_telecom_outage_in_japan/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YtfSQyghPibGjKDwjHb8GwAAAMQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://whitepapers.theregister.com/
The Cloud can't work without clouds?
LDS
Maybe it's time to re-launch the Sun server line?
Re: The Cloud can't work without clouds?
Neil Barnes
Set the controls for the heart of the sun! https://www.youtube.com/watch?v=8RbXIMZmVv8
Idling
elsergiovolador
All those servers running idle "just in case there is a surge", surely don't help the environment?
Re: Idling
Tom Chiverton 1
Their not drawing much power till switched on. Why do you think AWS is always banging on about how quick their VMs boot ?
Re: Idling
localzuk
That's not really how it works. "Spare" capacity will mostly be in a low power mode, aka switched off. As demand grows, batches of server swill be spun up in preparation for services to run on them.
So, only a small number, comparatively, will be online doing very little.
Compared to running it all in house, where you end up only using 10% of the resources 90% of the time.
"pride themselves on over-engineering to ensure resilience"
Pride themselves ?
More like talk about it.
Over-engineering costs money. The Cloud TM is about making money, not spending it.
And, as this episode has proven, they didn't over-engineer their cooling systems, they put in the bare minimum to ensure "normal" operations.
Well they're going to have to go back and over-engineer that part a bit, because I'll wager that next summer, they'll be seeing those temperatures again.