OVH services still not fully restored as boss rates ongoing recovery efforts a 'real nightmare'
- Reference: 1618387092
- News link: https://www.theregister.co.uk/2021/04/14/ovh_restoration_update/
- Source link:
The fire took place [1]on March 10 and destroyed the SBG2 hall of the Strasbourg data center, damaged SBG1 so badly it won't be revived, and led to a massive effort to [2]clean salvageable kit so it could be installed in the remaining three data centers at Strasbourg, or moved to other OVH facilities.
[3]
Work is going slowly, according to the most recent tweet from OVH co-founder and chair Octave Klaba.
Update April,14 9pm
SBG1/E > Croix > SBG3
new 200 servers were shipped to Floor 5 which is now full. Tomorrow, we start filling Floor 3. Step by step. Yes, it’s slow. Real nightmare. [4]pic.twitter.com/ylzMjzlEY3 — Octave Klaba (@olesovhcom) [5]April 13, 2021
[6]
In other tweets, Klaba said that all servers due to be moved from SBG1 to SBG4 have made the trip, with moves from SBG1 to SBG3 due this week.
"All servers are cleaned and tested, but it takes time to reassemble the racks... a huge Tetris," he [7]wrote .
OVH no longer has a link to its incident report on the front page of its website, but an April 12 [8]update revealed the following progress towards restoration:
92 per cent of servers for OVH Bare Metal Cloud have been delivered to customers
86 per cent of customer services running on Public Cloud instances have been replaced
99 per cent of Public Cloud Storage has been replaced
79 per cent of Hosted Private Clouds have been restored
84 per cent of Bare Metal Cloud Virtual Private Servers have been restored
OVH says burned data centre’s UPS, batteries, fuses in the hands of insurers and police [9]READ MORE
The new update said cleaning each rack prior to reinstallation in a new data center takes seven hours, and that OVH staff are getting faster every day. However, France's renewed COVID-19 lockdown is not helping.
"We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex," the biz said.
Pandemic-related challenges, and the sheer complexity of the restoration effort, mean OVH has not offered a resolution date for the incident.
[10]
The company is also working on vouchers customers can use to pay for services and will not bill for services affected by the fire or replacement services ordered by customers. ®
Get our [11]Tech Resources
[1] https://www.theregister.com/2021/03/10/ovh_strasbourg_fire/
[2] https://www.theregister.com/2021/03/29/ovh_restoration_update/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YHa9OwzTwUup29C0zqYM2QAAAA0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://t.co/ylzMjzlEY3
[5] https://twitter.com/olesovhcom/status/1382042489977180160?ref_src=twsrc%5Etfw
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YHa9OwzTwUup29C0zqYM2QAAAA0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://twitter.com/olesovhcom/status/1380780485945753601
[8] https://www.ovh.com/world/news/press/cpl1787.strasbourg-datacentre-latest-information
[9] https://www.theregister.com/2021/03/17/ovh_restoration_update/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YHa9OwzTwUup29C0zqYM2QAAAA0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
Re: DR plans?
On the other hand, they seem to be fairly frank in keeping people informed of their progress, saying it's a "real nightmare" rather than corporate positive messaging. Still, not as good as better risk reduction in the first place.
Re: DR plans?
If you don't have good DR plans in the Cloud then you have no business in IT.
The Cloud makes things so simple, there is no excuse. Want a load balancer to run your load across two availability zones, yes, click this box, done. Want to replicate those backups to a different region, tick this box, done. Want to replicate that VM to another region, tick this box, done. There are zero excuses when compared to on-prem where you have to make massive investments to run active-active data centres, along with specialist knowledge required to run them.
Of course, not all clouds are equal, not all clouds have Availability Zones, or some only have them in a few regions - I guess that's what you pay for and go with a lower-cost provider at your peril. I guess in the case of OVH, customers got burned here..
Re: customers got burned
Only the customers that didn't check the appropriate box.
OVH has a DR plan, there are a number of other datacenters over Europe. Anyone who had included a DR zone is not having any problem.
Of course, it's the many, many OVH customers who looked at the price of the option and decided it wasn't worth it who are now crying in the corner.
No sympathy from me.
Assessment of risk
DR plans are (or should be) based on prioritised risks. But, as normally conducted, assessment of risk is a still a highly subjective process. They probably thought the fire was very unlikely.
The currently common assumption that minimum likelihood times maximum consequence is equivalent to maximum likelihood times minimum consequence is fundamentally flawed. In reality an extreme of either parameter should be considered as overriding, regardless of the level of the other parameter. So, provided it's realistically possible, a total FUBAR consequence should be controlled independent of its likelihood because you don't want to be wiped out, and a massively frequent event with even trivial consequences should be controlled because otherwise you'll spend all your time responding to it. Contrary to the model suggested by the infamous risk matrix, likelihood and consequence do not necessarily interact linearly.
DR plans?
This is a fantastic lesson on how not to run a data centre company.
Still, I'm sure someone at the bottom will get the sack for not putting the lid on the pen correctly, or something like that.