Data Center Backup Generator Kicks In During Power Outage and Catches Fire (theregister.com)
- Reference: 0144855960
- News link: https://hardware.slashdot.org/story/21/04/09/2153248/data-center-backup-generator-kicks-in-during-power-outage-and-catches-fire
- Source link: https://www.theregister.com/2021/04/06/webnx_data_fire/
> A power outage kicked off a fire in web hosting biz WebNX's Ogden data center in Utah on Sunday, [2]knocking the facility offline temporarily and leaving several servers in need of a rebuild . Kevin Brown, Fire Marshal for the US city's Fire Department told The Register in a phone interview that firefighters responded to a call on Sunday evening. The fire, he said, "originated in a generator in the building and spread to several servers." Brown said the facility's fire suppression system contained the blaze and that fire department personnel assisted with the cleanup. He said power was cut to the building until an electrical engineer could inspect the facility to make sure current could be restored safely, which he added is standard procedure. He also confirmed that some of Ogden City's IT services were down on Sunday and Monday as a result of the data center fire.
"Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on," the company said in [3]a Facebook post . "However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."
"Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact." They added: "Customer's servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers."
[1] https://slashdot.org/~Joe_Dragon
[2] https://www.theregister.com/2021/04/06/webnx_data_fire/
[3] https://www.facebook.com/WebNX/posts/4495315520484290
Redundancy? (Score:5, Insightful)
"Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact."
Isn't the whole point of offsite data storage that the servers have redundant backups designed to maintain operation and data should one set get knocked out or fail?
They added: "Customer's servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers."
Thank god I only lost my data from water damage, the thought of it burning to death is horrifying.
Define redundancy... (Score:2)
I guess they defined "redundancy" as: Another server right on top of the other one, or something like that. ;)
Hint: Never rely on just one site either.
Triple software times triple hardware (times triple cables) times triple sites times triple backups. And then add versioning to the entire thing. (From transactions one can roll back over snapshots to keeping a history of backups.)
Oh, and for cash payment you can rent a 18-ton truck. ;)
Re: (Score:1)
> I guess they defined "redundancy" as: Another server right on top of the other one, or something like that. ;)
Yeah, better put them in two separate towers. Oh, wait...
Re: (Score:3)
Its probably that they organized their racks by customer. This customer gets these racks together this one get those etc. While it does make it easier to physically and logically manage things, it does reduce the redundancy in this case. You would need to spread the customers boxes out and completely mix them up to prevent this situational outage. Would there be an discernable difference in fiber latency going say a football field size distance, additional switches, etc? Maybe there are technical
Re: (Score:2)
> Its probably that they organized their racks by customer. This customer gets these racks together this one get those etc. While it does make it easier to physically and logically manage things, it does reduce the redundancy in this case. You would need to spread the customers boxes out and completely mix them up to prevent this situational outage
If it's like the datacenter where my servers are located, customers rent one or more cabinets (it's also possible to rent 1/2 cabinet).
Any sensible configuration wi
Water damage? (Score:1)
Who puts water-based fire supression in a data center???
Re: (Score:1)
You can't use halon and other non-water based fire suppression in newer data centers in a lot of geographies.
Re: (Score:2)
The fire department if I had to guess.
Water in a data center? (Score:3)
What shit show of a data center uses water for fire suppression?
What a joke.
Also was the generator so close to the actual servers that they had to get doused with water to quench a fire in the generator?
> No fire damage was inflicted on customer servers
Because the water damaged them first, hard to set fire to a sopping wet server.
If this whole debacle doesn't make you take your hosting elsewhere then you deserve to lose data.
Re:Water in a data center? (Score:4, Insightful)
> What shit show of a data center uses water for fire suppression?
One that is following fire code? In my experience data centres will have both pre-action sprinklers and a clean agent (Something like Inergen) Depending on the local fire code, tehy will will require sprinklers to protect the structure and any adjacent structures. The Inergen (or other clean agent) is there to protect the equipment, but the sprinklers will operate to protect the building if the clean agent fails to stop the fire.
Also, if the fire department found active fire after the power was cut, they'll use water to knock that fire out.
Re: (Score:2)
> What shit show of a data center uses water for fire suppression?
> What a joke.
> Also was the generator so close to the actual servers that they had to get doused with water to quench a fire in the generator?
A real joke is making a lot of assumptions based on a throwaway line in a media release from a PR department that a server suffered water damage.
Re: (Score:2)
This reminds of of rackspace Texas data center indecent over a decade ago. They had an explosion and everything worked on the backup power until the fire department showed up and orderered them t turn it all off so they could safely put out the fire. The fire department doesn't care about uptimes or data integrity, they only care about saving lives.
Also, a few years ago I worked in a building that housed the head office and servers for a multi billion dollar business. One day some construction crew brok
At which point... (Score:3)
The secondary backup generators came online, put out the fire from the primary backup generators, and proceeded to spray raw sewage from the secondary sprinklers. A senior datacenter engineer who was reached for comment replied: "It could have been worse".
One Day of credit per 15 minutes of downtime? Ouch (Score:3, Informative)
per the article: WebNX's SLA guarantees 100 per cent uptime and uninterrupted power every month, with account credits of one day per 15 minutes of downtime in each case.
If they're down for a week that's 672 days free for each customer...
Re: One Day of credit per 15 minutes of downtime? (Score:2)
I guess they bet on people thinking "Why would I want credit for using a data center that just drowned my servers??" and ending their contracts early. ;)
Odd-Looking Installation (Score:5, Interesting)
With a little bit of help from [1]this article [datacenterfrontier.com], I was able to locate the data center in question using Google Maps.
[2]Here [google.com] is a Google Maps view of the same building from a near-identical location as the photograph from the "Data Center Frontier".
If you explore the Google Maps view, one of the things that you will not see is an external generator associated with this WebNX facility. The closes I could get to spotting it can be seen in [3]this Google Streetview image" [google.com]. If you look at the roof line, near the centre of the image, you will see a rotating-vent exhaust that looks like it might be connected to say a central heating system or similar. Its the only visible sign I could see of anything that looks like it could handle hot exhaust gas from an internal combustion engine powering an electrical generator.
Obviously I'm just speculating here - and more importantly, I don't have any information about the internal layout of the structure. It's entirely possible that the operators have a purpose-built interior room that is comprehensively fireproof and robustly separated from the technology part of the data center facility.
On the other hand... Unless there is an external generator installation at the facility [I am entirely happy to concede this point if I've missed it!!!] then it looks like this host has built their diesel backup generator inside their hosting facility. Which would rather suggest that any sort of fire or similar problem with the generator could have the reported impact of affecting racked hosts.
It is all to easy and not remotely helpful to cast opinions based on lack of concrete data and on information sources as incomplete as Google Streetview. So I won't. But I will say that this location does look rather unusual, given the lack of apparent distinct backup generator location.
Interesting to see the Fire Department write-up if/when they publish it.
[1] https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/
[2] https://www.google.com/maps/@41.2616273,-111.9923689,3a,75y,300.8h,80.44t/data=!3m6!1e1!3m4!1s66EAh51d2cmVtFlBDxVX7Q!2e0!7i16384!8i8192
[3] https://www.google.com/maps/@41.2624393,-111.9924583,3a,75y,257.87h,92.3t/data=!3m6!1e1!3m4!1s5iVKKomlMCp-Ke7UvN7dwg!2e0!7i16384!8i8192
Re: (Score:3)
then it looks like this host has built their diesel backup generator inside their hosting facility.
I'm anything but an expert in this area, but this is the first question I had while reading TFA: How would a generator incident trigger a water fire suppression protocol inside technical rooms ? It must be very close, or even inside the data center rooms, right?
Is there a real interest, or a particular constraint, to place generators so close to the data center ?
Re: (Score:2)
We both had exactly the same thought then? I read TFA and thought, “No way should a generator and server racks be in close enough proximity for a fire in the former to have *any* impact on the latter. Just no way.” That send me off exploring and led me to conclude that the generator is physically inside the core building.
Which, sorry, is nuts.
Apart from all the fire hazard risks, the challenges of storing sufficient fuel for decent run-time, leaves you with the risks of things like mainten
Re: (Score:2)
Yeah this didn't make sense to me either. All the data centers I've been to (or worked at) have the generators outside and away from the building. Looking at the building in question, it seems pretty obvious that they just repurposed some generic existing structure instead of doing things properly.
I bet it was cheaper though.
Re: (Score:2)
A long time ago, I designed and commissioned the build of a relatively small (4800 sq ft) data center / comms room setup). We ended up with a pair of decent-sized 6-cylinder diesel generators and a UPS that gave us 20 minutes of standby time, just in case we had trouble starting either generator (each of which could handle 100% of the load on commissioning date and was intended to be able to handle at least 75% through the life of the generator.
I simply wasn’t allowed to put the generators close to
Re: (Score:2)
That whole area of the building (behind the green transformers and white shelter thing) appears to be a large electrical room with 2 distinct electric utility service entrances (3 door grey boxes each have revenue meters on them). I would agree with you that this is likely the location of the generators and related switchgear.
In multiple articles, they quote the operator as saying "generators" but typically exhaust systems would be independent of one another. As you pointed out, there is only one vent sta
You can see the genset in streetview (Score:1)
[1]https://www.google.com/maps/@4... [google.com]
The electric radiator louvers are above motercycle on the loading dock. When the generator starts they open.
The exhaust pipe is the the brown pipe you see running horizontal above the silver station wagon.
[1] https://www.google.com/maps/@41.2626201,-111.9924752,3a,15y,233.97h,92.52t/data=!3m6!1e1!3m4!1s3ZAmo9XMF-vV6Mz0E9uDwA!2e0!7i16384!8i8192
Re: (Score:2)
> It is all to easy and not remotely helpful to cast opinions based on lack of concrete data and on information sources as incomplete as Google Streetview. So I won't.
What are you talking about. This is Slashdot. You're supposed to just assume the generator was in the middle of the datacentre and that the entire building got insta-flooded with water including all the servers, and then call everyone involved an incompetent moron. That's what everyone else on Slashdot does. Don't go doing investigations for details, and on top of that then not make an assessment. That would be talking too much sense for this site.
I *love* "standard procedure". (Score:3)
Like "security reasons" it relieves one of all need to actually state any reasons. (In this case, there were valid reasons, in the contect of an incompetent data center design, but they were not stated.)
Notable mention: "It's the law!". (Invalid implication: It is right!)
See... (Score:1)
No C4 needed.
EEVBlog used this data center (Score:2)
and as luck would have, some of the water damaged servers were for EEVBlog.
Battery Backup (Score:2)
I wonder how long Tesla Megapacks or the competitive battery systems could keep the data center running?
Re: (Score:2)
> I wonder how long Tesla Megapacks or the competitive battery systems could keep the data center running?
I don't know what the capacity of a Megapack is, but data centers require a pretty substantial amount of energy. There are various UPS systems available just for that purpose, so I don't see any reason why a large Tesla battery pack couldn't be used, but you'd still want a generator for any extended outages.
I guess what I'm trying to say is, generators make electricity and batteries don't.
I hate it when that happens. (Score:2)
> "However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."
I remember my first corporate sysadmin job. Built a server farm, full UPS support, self-tested every week, rundown test every month, and yanking the power cord out of the wall once in a while, just for good measure.
First time we have an actual power outage, the U
Test test test (Score:2)
Some of our water customers test-run their backup power generators weekly. It also keeps the fuel from getting stale.
Halt And Catch Fire (Score:5, Funny)
Literally!
Re: (Score:2)
Give the show a watch if you haven't.