Data centre outfit Interxion hit with outage at central London facility
(2022/01/11)
- Reference: 1641912365
- News link: https://www.theregister.co.uk/2022/01/11/interxion_suffers_outage_at_central/
- Source link:
European data centre operator Interxion suffered an outage at its central London campus last night, with no service from its LON1 data centre for several hours and users complaining the company was silent about what was happening.
[1]
In pre-COVID times, clearly: Ely's Yard, behind the Old Truman Brewery building that houses LON1
The incident appears to have started shortly after 1800 UTC on Monday evening, with a loss of power at the LON1 site in the city's East End, which is conveniently located near to the financial district and the "Silicon Roundabout" tech hub in Shoreditch, according to Interxion.
Unfortunately for Interxion, built-in redundancy did not save it in this instance as the failure struck multiple power feeds into the building. Worse, it seems that the electronic switchgear designed to change the power over to an on-site generator also failed, and this prevented the company from switching between the mains supply and the backup generator supply.
A source whose employer is a member of the London Internet Exchange (LINX) informed The Register that there was "a flurry of emails on the member's mailing list covering the full spectrum from concerned (about what's going on and the possible extent) to angry (about lack of comms and inability to get hold of anyone from Interxion)."
LINX systems at the site appear not to have been affected, with the outage apparently affecting the entire first floor of the data centre site. It is not clear what may have caused multiple power feeds into the LON1 site to fail at the same time.
[2]
One user that was affected by the outage at Interxion's site was Vultr, a provider of cloud-based server infrastructure. Vultr's online [3]server status notifications reported a partial outage in its London location starting at 1805 UTC, updating to show a complete outage by 1836 UTC.
[4]
[5]
By 2124 UTC, Vultr was reporting that "our data centre partner is continuing to work on restoring power to the site."
By this time, Interxion had apparently managed to restore some building systems and was attempting to restore power to the rest of the facility. It was 0136 UTC the following morning before Vultr was able to report that its entire London-based infrastructure was back online and operational.
[6]Coin of Vantage looks good to Etix Everywhere: US bit barn beast eats European rival in bid to crack the Continent
[7]Public cloud has gone from existential threat to friends with benefits for colocation providers
[8]Data centre outfit Interxion admits to contact detail security breach
[9]Equinix moves into Europe by slurping up Telecity
Sargasso Networks, another Interxion customer, [10]tweeted at 1938 UTC that it was "currently experiencing a major outage due to failure of multiple (allegedly resilient) power feeds at Interxion, over 90 minutes ago."
The company complained that "as Interxion have still failed to share any information and refuse to answer the telephone, we do not yet have an ETTR."
[11]
We contacted Interxion for a statement regarding the incident, and will update if we receive a response.
Interxion (pronounced "Interaction", apparently, a subject of some debate in the newsroom this morning) boasts of having facilities across 11 European countries and 13 metropolitan areas. It was [12]acquired in 2020 by US data centre outfit Digital Realty in a deal said to be worth $8.4bn at the time.
We have asked the data centre outfit for comment and will update when it responds. ®
Get our [13]Tech Resources
[1] https://regmedia.co.uk/2022/01/11/shutterstock_elys_yard.jpg
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://status.vultr.com/#LHR
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2020/02/11/vantage_data_centers_buys_etix_everywhere/
[7] https://www.theregister.com/2019/03/28/public_cloud_is_fast_becoming_colocation_providers_best_friend/
[8] https://www.theregister.com/2016/01/11/interxion_security_breach/
[9] https://www.theregister.com/2015/05/29/equinix_slurps_up_telecity_interxion/
[10] https://twitter.com/sargassosupport/status/1480625119185846275
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2019/10/30/digital_realty_interxion_merger/
[13] https://whitepapers.theregister.com/
[1]
In pre-COVID times, clearly: Ely's Yard, behind the Old Truman Brewery building that houses LON1
The incident appears to have started shortly after 1800 UTC on Monday evening, with a loss of power at the LON1 site in the city's East End, which is conveniently located near to the financial district and the "Silicon Roundabout" tech hub in Shoreditch, according to Interxion.
Unfortunately for Interxion, built-in redundancy did not save it in this instance as the failure struck multiple power feeds into the building. Worse, it seems that the electronic switchgear designed to change the power over to an on-site generator also failed, and this prevented the company from switching between the mains supply and the backup generator supply.
A source whose employer is a member of the London Internet Exchange (LINX) informed The Register that there was "a flurry of emails on the member's mailing list covering the full spectrum from concerned (about what's going on and the possible extent) to angry (about lack of comms and inability to get hold of anyone from Interxion)."
LINX systems at the site appear not to have been affected, with the outage apparently affecting the entire first floor of the data centre site. It is not clear what may have caused multiple power feeds into the LON1 site to fail at the same time.
[2]
One user that was affected by the outage at Interxion's site was Vultr, a provider of cloud-based server infrastructure. Vultr's online [3]server status notifications reported a partial outage in its London location starting at 1805 UTC, updating to show a complete outage by 1836 UTC.
[4]
[5]
By 2124 UTC, Vultr was reporting that "our data centre partner is continuing to work on restoring power to the site."
By this time, Interxion had apparently managed to restore some building systems and was attempting to restore power to the rest of the facility. It was 0136 UTC the following morning before Vultr was able to report that its entire London-based infrastructure was back online and operational.
[6]Coin of Vantage looks good to Etix Everywhere: US bit barn beast eats European rival in bid to crack the Continent
[7]Public cloud has gone from existential threat to friends with benefits for colocation providers
[8]Data centre outfit Interxion admits to contact detail security breach
[9]Equinix moves into Europe by slurping up Telecity
Sargasso Networks, another Interxion customer, [10]tweeted at 1938 UTC that it was "currently experiencing a major outage due to failure of multiple (allegedly resilient) power feeds at Interxion, over 90 minutes ago."
The company complained that "as Interxion have still failed to share any information and refuse to answer the telephone, we do not yet have an ETTR."
[11]
We contacted Interxion for a statement regarding the incident, and will update if we receive a response.
Interxion (pronounced "Interaction", apparently, a subject of some debate in the newsroom this morning) boasts of having facilities across 11 European countries and 13 metropolitan areas. It was [12]acquired in 2020 by US data centre outfit Digital Realty in a deal said to be worth $8.4bn at the time.
We have asked the data centre outfit for comment and will update when it responds. ®
Get our [13]Tech Resources
[1] https://regmedia.co.uk/2022/01/11/shutterstock_elys_yard.jpg
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://status.vultr.com/#LHR
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2020/02/11/vantage_data_centers_buys_etix_everywhere/
[7] https://www.theregister.com/2019/03/28/public_cloud_is_fast_becoming_colocation_providers_best_friend/
[8] https://www.theregister.com/2016/01/11/interxion_security_breach/
[9] https://www.theregister.com/2015/05/29/equinix_slurps_up_telecity_interxion/
[10] https://twitter.com/sargassosupport/status/1480625119185846275
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yd23tzz4FEgw49-izwJ@PQAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://www.theregister.com/2019/10/30/digital_realty_interxion_merger/
[13] https://whitepapers.theregister.com/
These clouds looks blacker than they should....
LDS
And they black out just like old datacenters - just creating more widespread outages.
Doctor Syntax
"the electronic switchgear designed to change the power over to an on-site generator also failed"
It's always DNS - except when it's UPS.
Silicon Roundabout
Equality 7-2521
I remember it as the Old Street Roundabout.
It wasn't built on sand in those days. It's doesn't feel like a roundabout now with the road layout changing.
There are still large open plots, bombed during WW2 and now surface car parks.
I'll report back in another 35 years time.
TTFN
"Interxion have still failed to share any information and refuse to answer the telephone"
Anonymous Coward
'cos their power has gone off?
Wasn't one of the Docklands sites affected a few years ago because all their redundant power feeds entered the building at the same point?... one errant JCB wiped out the lot!
My hopes for an early night was dashed by this outage. The DC and networks teams figured out all was not well sometime after 1800. As the article states not much info was forthcoming from Interxion, I think we found out via a tweet somewhere what was going on and so a lucky volunteer went off to site. And took them quite a while to be allowed in.
I got dragged into all this about 2100 and spent ~3 hours trying to get our servers back up and running. Some of our servers really didn't take the outage very well and required some manual intervention of various forms.
There was no staged power on, so everything came on at once. Including the chillers/aircon which didn't have much time to get themselves up to speed before everything else. The result being things got quite hot before levelling out. But for a while we considered switching off non-prod and things we think we can get away without. Fortunately it didn't come to that.
I think we're still waiting for the official blurb/reasons.