Twitter datacenter melted down in Labor Day heat
- Reference: 1663097076
- News link: https://www.theregister.co.uk/2022/09/13/twitter_datacenter_labor_heat/
- Source link:
According to [1]a memo obtained by CNN , Twitter lost access to its Sacramento (SMF) datacenter "due to extreme weather" on Monday, September 5, 2022. The memo, attributed to VP of engineering Carrie Fernandez, says, "The unprecedented event resulted in the total shutdown of physical equipment in SMF."
The temperature in downtown Sacramento, California, that day – Labor Day in the US – reached [2]a record high of [3]113°F or 45°C, and local temperature records were broken [4]again within days.
[5]
While Twitter has backup datacenters in Atlanta and Portland that can handle the load of traffic that the social network handles each second, the loss of the Sacramento hub put it in a potentially perilous position.
[6]
[7]
"If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter's users," Fernandez warned.
[8]Lessons to be learned from Google and Oracle's datacenter heatstroke
[9]Underwater datacenter will open for business this year
[10]Google: We had to shut down a datacenter to save it during London's heatwave
[11]Immersion cooling no longer reserved for the hyperscalers, HPC
A 2020 study from [12]The Uptime Institute , "The gathering storm: Climate change and datacenter resiliency," warns that extreme temperatures represent a growing threat to datacenters.
"Increasing summer temperatures have put a strain on the cooling of many datacenters, and this is likely to worsen in the years ahead," the report says.
Beyond hindering the ability to operate, high temperatures can make datacenters cost more than anticipated.
[13]
"An outdoor air temperature increase of as little as four degrees Fahrenheit (two degrees Celsius) could make free air and evaporative systems ineffective on their own, and uneconomical in some situations," the report says.
About 40 percent of the energy consumed by datacenters goes toward cooling IT equipment, according to [14]a 2015 study , "Data Center Energy and Cost Saving Evaluation."
The memo was sent on Friday, September 9th, and it disallows all production and app changes while service is restored except for what's required "to address service continuity or other urgent operational needs."
Not the worst of Twitter's problems
[15]A whistleblower complaint filed against Twitter last month by its former head of security Peiter "Mudge" Zatko casts doubt on Twitter's data center resilience, citing the company's "Cascading datacenter problems."
"In or around the spring of 2021, Twitter's primary datacenter began to experience problems from a runaway engineering process, requiring the company to move operations to other systems outside of this datacenter," Zatko's complaint says.
[16]
"But, the other systems could not handle these rapid changes and also began experiencing problems. Engineers flagged the catastrophic possibility that all the datacenters might go offline simultaneously."
Twitter, currently in the midst of [17]a legal battle to force Elon Musk to honor his commitment to buy the company, brushed off its data center problems.
"There have been no disruptions impacting the ability for people to access and use Twitter at this time," a Twitter spokesperson told The Register in an email. "Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience."
We asked Twitter whether having the tools to ship updates means its engineers are actually doing so, given that Fernandez's memo suspended non-essential product updates until operations have been normalized. We'll update this story if we hear back. ®
Get our [18]Tech Resources
[1] https://www.cnn.com/2022/09/12/tech/twitter-data-center-california-heat-wave/
[2] https://twitter.com/NWSSacramento/status/1566958764355633158
[3] https://www.sacbee.com/news/weather-news/article265368881.html
[4] https://twitter.com/NWSSacramento/status/1567268886936186880
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YyD9dBwovgk2@iAY8@CLKAAAAIQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YyD9dBwovgk2@iAY8@CLKAAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YyD9dBwovgk2@iAY8@CLKAAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/08/22/google_oracle_heatstroke/
[9] https://www.theregister.com/2022/09/01/subsea_cloud_underwater_datacenter/
[10] https://www.theregister.com/2022/08/01/google_london_outage_incident_report/
[11] https://www.theregister.com/2022/06/01/rnt_rausch_submer_immersion/
[12] https://uptimeinstitute.com/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YyD9dBwovgk2@iAY8@CLKAAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://www.sciencedirect.com/science/article/pii/S1876610215009467
[15] https://www.theregister.com/2022/08/23/twitter_security_whisterblower/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YyD9dBwovgk2@iAY8@CLKAAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.theregister.com/2022/09/13/twitter_says_775m_severance_payment/
[18] https://whitepapers.theregister.com/
To be said tomorrow...
" Ah but look, since the infrastructure is clearly wobbly, I can completely legitimately walk away from the promise I made to purchase the entire company. "
Sacramento gets hot at the best of times. So forty odd degrees doesn't seem all that improbable from time to time, even had we not got climate change to factor in. And the UK had 40+ temperatures a few weeks back ( hard as it is to think so now).
So not being able to function at 45 degrees in Sacramento seems sadly like very poor planning.
True, it does feel like very poor planning. When the average temperature for July/August is 35 degrees, with regular variances +/- 5 degrees, not being able to cope with twice the regular variance is quite poor.
"If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter's users,"
My heart bleeds for Twatter. Positively BLEEDS.
Not really it doesn't. It'll just stuff those companies whose customer service is so shite that they rely on (anti)social media rather than phone(not premium rate) and email (not web forms).
Standards...
The international standard for temperature is Celsius . "115F" is meaningless to most individuals on the planet and is becoming steadily more meaningless.
Please use international standards by preference at all times but if you feel it necessary to also provide the same temperature in out-dated Imperial measurements to aid readers in the three countries on the planet that still use Fahrenheit, then do so as an extra level of detail afterwards.
Or, at the worst, use the correct Register Units, here (FTFY):
Bitbarn suffered 'total shutdown' after 2.5 Hilton heatwave
"If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter's users," Fernandez warned.
Oh dear. How sad. Never mind.