OVHcloud datacenter 'lacked' automatic fire extinguishers, electrical cutoff
- Reference: 1647976597
- News link: https://www.theregister.co.uk/2022/03/22/ovhcloud_fire_datacenter_report/
- Source link:
The incident report was obtained and published by Journal du Net ( [1]JDN ), a French-language IT site. It describes several issues that contributed to the [2]destructiveness of the blaze, including the presence of toxic fumes from lead batteries, a wooden ceiling rated to resist fire for only an hour, and two inner courtyards that acted as "fire chimneys."
The fire occurred at 0046 on March 10, 2021 and destroyed SBG2, a five-story datacenter occupying 500m 2 . Servers in adjacent buildings SBG1 were also damaged in the incident while SBG3 and SBG4 remained operational.
[3]
"We have a major incident on SBG2," said OVH founder and chairman Octave Klaba [4]via Twitter at the time. "The fire declared in the building. Firefighters were immediately on the scene but could not control the fire in SBG2. The whole site has been isolated which impacts all services in SGB1-4. We recommend to activate your Disaster Recovery Plan."
[5]
[6]
OVHcloud operates 15 datacenters in Europe, with four in Strasbourg and a fifth being built.
[7]
Source: JDN. Click to enlarge
The Bas-Rhin report describes an electrical inverter fire that began on the first floor of the five-floor SBG2 structure. The cause of the fire has not been officially declared by OVHcloud and the company did not immediately respond to a request for comment.
[8]Talk about a Blue Monday: OVH outlines recovery plan as French data centres smoulder
[9]OVH rises to Europe data sovereignty challenge (and AWS) with tape-as-a-service
[10]OVH drops IPO target against figure mooted a month ago
[11]OVHcloud to share its OpenStack automation for use in on-prem clouds
JDN describes how arriving firefighters were met with "electric arcs of more than one meter around the exterior door of the energy room" where the fire is believed to have originated. The report, as algorithmically translated says, "The technicians of ES (Electricité de Strasbourg) met difficulties in cutting off the electricity in the room." ES personnel arrived on-scene about 0120 and didn't manage to cut the power until about 0329.
The report describes the firefighter group leader stating that the temperature in the ground floor room measured 400 degrees Celsius using a thermal camera.
Over 140 customers have [12]filed a class action lawsuit against the company seeking damages for losses arising from the fire. According to the law firm handling the complaint, Ziegler & Associates, numerous French government websites were affected by the fire, including data.gouv.fr, the National Education website, the Center Pompidou website and Meteosky. The law firm [13]claims many customers lost their data as a result of the fire and did not understand the need to pay for additional backups. ®
Get our [14]Tech Resources
[1] https://www.journaldunet.com/web-tech/cloud/1509749-enquete-sur-l-incendie-d-ovh-des-conclusions-accablantes/
[2] https://www.theregister.com/2021/03/10/ovh/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YjpVIF34aP1LXDX9PMVbPAAAAAA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://twitter.com/olesovhcom/status/1369478732247932929?lang=en
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YjpVIF34aP1LXDX9PMVbPAAAAAA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YjpVIF34aP1LXDX9PMVbPAAAAAA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://regmedia.co.uk/2022/03/22/reportimage_ovh.jpg
[8] https://www.theregister.com/2021/03/10/ovh/
[9] https://www.theregister.com/2021/01/20/ovhcloud-tape-storage-as-a-service/
[10] https://www.theregister.com/2021/10/05/ovhcloud_ipo_target_trimmed/
[11] https://www.theregister.com/2021/11/17/ovh/
[12] https://www.ziegler-associes.com/recours-ovh
[13] https://www.ziegler-associes.com/publications/articles/ovh-en-proie-aux-flammes-quelle-reparation-en-cas-de-perte-de-donnees
[14] https://whitepapers.theregister.com/
Re: Who else wants popcorn?
400 deg C is a little hit for popcorn
Re: Who else wants popcorn?
Actually it's a pretty big hit
got what you didn't pay for
Too bad for the customers not smart enough to realize the system was designed in a way so that failures would be have to be handled by the customer rather than the provider. They just saw the low price and said hey let's use that, all data centers are the same right?
https://www.datacenterknowledge.com/archives/2010/05/04/terremark-extinguishes-fire-stays-online
"Early on April 30, a fire broke out in one of the data center electrical rooms at Terremark's NAP of the Capital Region in Culpeper, Va. The fire department was on site for hours, and the event was covered by local media. But the facility remained online throughout the entire event, according to Terremark, with no downtime for customers."
https://www.datacenterknowledge.com/archives/2010/11/03/damage-from-fisher-plaza-fire-6-8-million
"A 2009 fire at the Fisher Plaza data center hub in Seattle caused $6.8 million in damages[..]The July 2, 2009 incident knocked payment processor Authorize.net offline, disrupting e-commerce for thousands of web sites, while also causing lengthy downtime for Microsoft’s Bing Travel service, domain registrar Dotster, colocation company Internap and web hosting provider AdHost,"
I worked at a company that was hosted at the 2nd facility(hosted there before I started my job), I moved them to a different facility in mid 2007 if I recall right, after two full data center power outages (there were more before I started). One of the power outages was a curious customer wondering what the "Emergency Power Off" button did(after that incident all customers had to attend EPO training before gaining access). Though in THIS case, the fire was contained to the electrical room and as far as I know no customer equipment was damaged.
The building ran on generator trucks for several months while they replaced the electrical system. The fire caused a roughly 42 hour downtime of the facility I think(including knocking the HQ of a local TV channel offline). I do recall being told stories some customers were freaking out because the batteries in their storage systems (to maintain data in the cache) of course can only retain power for X number of hours and there was uncertainty when power would be restored.
Though some storage systems were designed to handle that better in that they ran on internal battery long enough to dump the contents of cache to an internal drive(one per controller so there's two copies of the cache data) before shutting the controller down in the event of a power outage.
Re: got what you didn't pay for
I am pretty sure that the large volume of hackers they seem to host (judging by even just the 404 logs) won't mind, they're used to being cut off and just move to others such as Azure or DigitalOciean.
As far as I'm concerned the fire improved things for a bit..
Re: got what you didn't pay for
Bing has a travel service?
For your viewing pleasure
"electric arcs of more than one meter around the exterior door of the energy room"
Made me think of Daft Punk's Derezzed performed with musical Tesla coils.
[1]Daft Punk Derezzed
[1] https://www.youtube.com/watch?v=psoLXEBmfRg"
Ah, yes. Put all your confidential, proprietary, business critical on the internet without any backups. It's perfectly safe. What could possibly happen?
Who else wants popcorn?
*Starts handing out paper shopping bags stuffed full of freshly popped goodness to one & all*
Grab a drink, find a chair, & let's enjoy watching this FlusterCluck unfold!