UK government in talks with datacenter operators over blackouts
- Reference: 1666096213
- News link: https://www.theregister.co.uk/2022/10/18/uk_government_in_talks_with/
- Source link:
According to Bloomberg, the discussions focused on ensuring diesel fuel supplies for backup generators if the National Grid should be forced to cut power. It cited the usual anonymous sources "familiar with the matter" who said the talks also covered whether datacenters should be considered critical national infrastructure.
A large chunk of the UK's electricity is generated from gas – 39.8 percent in 2021, according to [1]figures [PDF] from the Department for Business, Energy and Industrial Strategy (BEIS). The Russian war in Ukraine has posed a threat to gas supplies across Europe, and while the UK does not get much gas from Russia, the country often imports electricity from the region when required.
[2]
The arm of National Grid which operates the UK's power supply, the Electricity System Operator (ESO) published a [3]report [PDF] earlier this month that outlined potential scenarios. It reckoned there will be adequate margins through the winter to ensure enough power.
[4]
[5]
However, in a worst-case scenario it may be necessary to initiate planned, load shedding schemes under which means some customers could be without power for pre-defined periods during a day, "generally assumed to be for 3 hour blocks".
Datacenter operators have backup power generators that can supply electrical power in the case of events such as this, but there is always a chance that it may be more than three hours, which raises the question of whether the datacenter will have enough fuel to keep the generator running until the grid supply is restored.
[6]
Gartner senior director analyst Tiny Haynes told The Reg that his company published a research note in 2013 when Ofgem was warning that there would be 99.8 percent availability of power, so there were chances of blackouts then, and the risk is much higher now.
"There's a potential issue with energy supplies or gas supply to power stations, which will then have a knock on impact on not just the datacenter providers, but will also take out power in mobile networks as well," Haynes said.
He added that while datacenter operators will have UPS and diesel generators on site, they need to be fully testing that everything will function in the case of a power failure.
[7]
"The best practice is to test generators once a week, but that's just a generator test. They're often not doing a full load test, a failover of all systems test," he said.
"Operators need to test all the switchgear, that all of UPS systems work, all the generators spin up, that they have adequate diesel supply," Haynes added.
Earlier this year, The Register reported how Interxion suffered a [8]power outage at its Shoreditch datacenter, but the electronic switchgear designed to change the power over to an on-site generator failed.
Haynes also warned that with clusters of datacenters in places, especially in some areas around London, it may be difficult to get enough diesel fuel supplies if the blackouts last for any time, which is hopefully one issue the government is aiming to address following its meetings with operators.
"My concern would be is if you want to resupply, let's say Docklands where you've got a large build-up of datacenters. I mean, you've got Sovereign House, you've got Global Switch, you've got Telehouse, you're gonna have a huge demand for fuel to be shipped into that area. Is there sufficient capacity?"
Operators could face reputational damage and the cost of service level agreement (SLA) breaches if they are unable to keep up and running, according to Haynes.
"It's usually service credits for outage, but if you're running a Tier 3 datacenter, you need to be providing at least a 99.982 percent availability, and that is minutes per month, so if you go down for a few hours, then it's gonna get very expensive very quickly," he said.
[9]More datacenters coming to Ireland, despite energy concerns
[10]Homes in London under threat as datacenters pull in all the power
[11]The US's biggest datacenter market is short on electricity
[12]Planning for power cuts? That's strictly for the birds
We asked several UK datacenter operators if they had taken any extra steps to ensure uptime this winter, but most were strangely unwilling to speak to us about this topic.
One company that did respond is Amazon Web Services, which told us that "AWS continuously monitors and manages our datacenter facilities worldwide. Each of our facilities has contingency plans and we ensure that each of our facilities performs continuous readiness checks. For example, we monitor and test for proper configuration of our electrical distribution systems, the health of our generators, fuel quantity on-hand, and readiness of additional fuel supply."
AWS insisted that: "This issue does not pose a risk to AWS services at this time. We work closely with utility providers in each region to identify and review risks to utility supply, and coordinate any AWS response to a utility risk or request should the need arise." ®
Get our [13]Tech Resources
[1] https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1094025/UK_Energy_in_Brief_2022.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y07NmCN-dkLi0hSNZ01u7wAAAFM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.nationalgrideso.com/document/268346/download
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y07NmCN-dkLi0hSNZ01u7wAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y07NmCN-dkLi0hSNZ01u7wAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y07NmCN-dkLi0hSNZ01u7wAAAFM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y07NmCN-dkLi0hSNZ01u7wAAAFM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/01/11/interxion_suffers_outage_at_central/
[9] https://www.theregister.com/2022/08/15/irish_datacenter_ban_reversed/
[10] https://www.theregister.com/2022/08/01/london_electricity_supply_barrier_to_housing/
[11] https://www.theregister.com/2022/07/29/the_us_biggest_datacenter_market/
[12] https://www.theregister.com/2022/01/17/who_me/
[13] https://whitepapers.theregister.com/
Re: contingency planning
If only we'd planned and started building more nuclear power stations in the 90s, as we should :(
Had we done anything useful in the last decade in terms of building numerous non-gas generation sites that would have cost less than £60B, we would be better off now.
Re: contingency planning
"If only we'd planned and started building more nuclear power stations in the 90s, as we should"
I remember seeing a documentary about this very issue twenty odd years ago, maybe on Panorama? How the UK was becoming increasingly dependent upon energy from overseas and how vulnerable it would leave our power supplies to problems in Europe, Russia and elsewhere. So the issues were well known but there has been no political will or foresight to prevent the unfolding mess.
Re: contingency planning
I wonder how much things will just fail as we run out of supply in certain areas at certain times or if someone's going to bite the bullet and choose. Will we stop cars from charging? Will we shut off data centres as these should have backup generators and DR at other sites? Will we shutdown the Houses of Parliament of the basis that nothing useful happens there?
Re: contingency planning
I think the Houses of Parliament should be tapped as a communal source of heating - there seems to be an unlimited supply of hot air there
True
But very little of it is green...
Re: contingency planning
@Andy Non
"So the issues were well known but there has been no political will or foresight to prevent the unfolding mess."
But the gov did plenty. Vast amounts spent on monuments to a sky god to save us. And we can always rely on cheep gas when the sky god is displeased with us.
The govs did plenty, but not much of use.
Re: contingency planning
We have no had an energy policy in the UK for over 30 years. Labour and the Tories (and Clegg - remember him? No, me neither) are all guilty of this. Leave it to the market, was the mantra. A bit like getting PPE and sanitising gel at the start of the covid fiasco. It works well until everybody wants the same thing.
-> How the UK was becoming increasingly dependent upon energy from overseas
When the Royal Navy converted from coal to gas this was a subject of crucial importance. We have at least 500 years of coal reserves in the UK, which means basically that supplies can be guaranteed. This is not the case with oil. This is not a new subject.
Many of the politicians and their flag wavers in the press don't have an ounce of sense between them. I laughed so much when the Daily Mail published its "At last! A true Tory budget" front page waving a few weeks ago. Yeah. They are oafs. We nearly had a financial meltdown to ape the 1970s as a result. Don't expect too much from these highly-educated politicos. They are insulated from everyday life. I imagine that most of them does not know the price of a loaf of bread. But I bet they do know how to fill in their expenses forms in the most creative manner.
Re: contingency planning
If only we'd planned and started building more nuclear power stations in the 90s, as we should
It's a collective failure, many countries face the same situation for the same reason.
Politicians see till the next election, not till the next decade. Add to this that voters are often seduced by demagogic claims, and it's easy to use FUD with nuclear power.
== Bring us Dabbsy back! ==
Re: contingency planning
-> It's a collective failure, many countries face the same situation for the same reason.
A pretty damning and justified indictment of the neoliberal politics which pervades the West in general.
Great having a DC working but there may not be anyone who can connect to it! It'd be interesting to know what the maximum outages are that individual components such as networking nodes/switching kit/ engineers kettles can run for in a scenario such as this.
Without having a crystal weather ball it's impossible to know if this will be an issue, whatever happens my priority will be my family.
From experience in planning back-up supplies, the power from the generators is usually only supplied to certain circuits. So for instance CCTV keeps recording, but the monitors are not powered.
This would mean kitchens would have no power.
That's not to say the staff won't have hidden a kettle in a room that does have power, just in case...
Day tank + bulk storage
Isn't that the way it's done? Day tank good for a few hours, very local to generator(s), re-filled from bulk storage in a more accessible location for the tanker. Bulk storage good for 7 days at full chat, anything else is not really a backup.
Perhaps I have only ever dealt with folks who do it properly . . . .
Re: Day tank + bulk storage
I was going to ask much the same thing. Any DC I've worked with had 1 day on site and 7+ days on call from nearby plus prioritised contract for indefinite supply from further storage. Odd if this isn't the case any more.
Re: Odd if this isn't the case any more
Not really.
Beancounters have been on the rise for the past twenty years.
Why do you think there's a component shortage ? Because Just-In-Time delivery, which removed stock (which costs money), and beancounters are notoriously adverse to spending money.
Well they're going to be real happy now.
Re: Day tank + bulk storage
Only problem with this is that the crap they supply as diesel nowadays doesn't keep, so you can't afford to store it for very long, unless you start adding additives, and even then the time period is still fairly limited.
Problem?
A 1MW data centre powered by a diesel generator will use about 300 litres/hour (assuming efficiency as a perhaps optimistic 33%), so 900 litres for a 3-hour outage. A fuel tanker holds around 25000 litres, so one round per day will keep 30 such DCs online.
Re: Problem?
I'm thinking one large UK telco might not be too happy at the moment.
Their big research centre and test facilities has had lots of work done to ensure they have the power needed...
2 x 1MW Gas Turbine plants for electricity, heat (cooling in summer) and hot water.
Large off-site solar farm connected to one of their Grid Feeds.
So if it is cloudy and we are short of gas then they are going to struggle as both their own gas turbines and the grid might not have the gas to supply the power they need. I seem to recall they only had limited diesel backup due to having their own generation from gas, and one of those is so badly placed the exhaust fumes got sucked straight into one of the main buildings!
contingency planning
It all sounds rather sensible to me. I'm doing some of that at home, although I've decided the ROI on a large battery for the house means it's not an I because there's no R.
If only we'd planned and started building more nuclear power stations in the 90s, as we should :(