Datacenter fire suppression system wasn't tested for years, then BOOM
- Reference: 1683271810
- News link: https://www.theregister.co.uk/2023/05/05/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Jack" who once worked in the computer room of a school board's HQ.
Next to the computer room was a walled-off shared office that had a window allowing a view of the computer room.
[1]
The two spaces shared a raised floor, so the very large tank crammed full of high-pressure halon gas in place to suppress fires would flood both rooms in case of emergency.
[2]
[3]
Both rooms were equipped with sensors to detect smoke, which would cause a bell to sound warning of imminent gas release.
But in case of a false alarm, or minor incident, the facility also included a Big Yellow Button that would abort the gas release. And in case the sensors didn't do their thing, the computer room also housed a Big Red Button that overrode everything and released the gas to put out any fire.
[4]
Those buttons and their respective roles became important when one of Jack's colleagues decided to do a spot of soldering in the office. A little smoke duly wafted past a sensor, and the warning bell began to sound.
Clearly the computer room was in no danger whatsoever. So one of Jack's colleagues pressed the Big Yellow Button to stop the release of Halon!
BOOOOOOM!
[5]
"The halon system immediately dumped its tank," Jack told On Call. "It was so fast that it sounded like a shotgun blast. Dust and paper flew everywhere in the air and one staff member sitting at their desk, not far from the halon nozzle in that office, dove under the desk, thinking that someone had fired a gun."
[6]Shocks from a hairy jumper crashed a PC, but the boss wouldn't believe it
[7]Thanks for fixing the computer lab. Now tell us why we shouldn’t expel you?
[8]Techie called out to customer ASAP, then: Do nothing
[9]Automation is great. Until it breaks and nobody gets paid
As explained above, the Big Yellow Button was there to stop gas being released. It had only one job – why had it failed so spectacularly?
Jack's investigation revealed that the Big Yellow Button and the Big Red Button had been wired in reverse many, many years before the soldering iron set off the alarm.
"Nobody had ever tested all the buttons with the system's dump mechanism disabled in order to ensure that everything was wired correctly," Jack explained.
"Seventeen years later, when the facility installed a new fire suppression system, the company that installed it worked with the IT staff to test each and every button to make sure that the system worked correctly and that all dump and countdown reset buttons were wired correctly," Jack wrote.
The new system sounds foolproof. As Jack tells it, at the first whiff of trouble a 30-second countdown commences – during which three high-pitched, warbling horns with strobe lights blare.
"The abort buttons reset the counter to zero with each momentary press, allowing a single staff member to run around the room, checking for smoke or fire and to allow that staff member to go to the fire suppression system's alarm panel and disarm the system if no issue is found."
What could possibly go wrong? On Call is sure readers will have stories of the frailties such a system possesses.
[10]Click here to send your story to On Call and we may feature it here on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZFTTw9IUv-bpZUPTeR886AAAAFY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFTTw9IUv-bpZUPTeR886AAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFTTw9IUv-bpZUPTeR886AAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFTTw9IUv-bpZUPTeR886AAAAFY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFTTw9IUv-bpZUPTeR886AAAAFY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/04/28/on_call/
[7] https://www.theregister.com/2023/04/21/on_call/
[8] https://www.theregister.com/2023/04/07/on_call/
[9] https://www.theregister.com/2023/04/14/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Can Confirm.
Can confirm that when halon goes off it does sound like a small tactical bomb. Especially if you are not expecting it.
Under floor halon globe blew the dress completely off one of our female tape operators and another drove the senior operators chair across the consoles and broke his arm. Unions had hell to pay for this.
No fun was had by anyone that day.
Tempt fate
"The abort buttons reset the counter to zero with each momentary press,...
When you need this button, it has been worn out by the testing (NC gets stuck and NO doesn't close). These buttons tempt fate.
Murphy will always be around...
Re: Tempt fate
I'm pretty sure with a button that 'resets' a 30 second countdown to zero Murphy will be closer than some think...
I had to deal with that kind of system at a place I worked at many moons ago.
Every Friday we'd test it by triggering the alarm, with the added pleasure of having to answer a phone call. Fail to cancel the alarm meant a halon dump, failing to answer the phone meant a few fire engines turning up at the front door.
Never triggered either the entire time I was there (a minor miracle in itself).
Used to have a guy turn up once in a while with a cup on a stick thing to test the sensors too.
Err
From my military days there's 7 P's
Proper Planning and Preparation Prevents Pi** Poor Performance.
Re: Err
I've always preferred "Poor Planning on Your Part, Does NOT Constitute an Emergency on Mine".
Although maybe that's not quite so accurate in the above story, since it would definitely be an emergency for whoever was in that room!
Re: Err
My framed version states "Bad planning on your part does not necessarily constitute an automatic emergency on my part."
Re: Err
We have had that on our department wall for years!
Sometimes, someone will point it our to a visiting manager...
Re: Err
As quoted in "Carry On England" (1976)
death trap
The whole setup sounds like a death trap to me:
- office also had detectors and would be flooded by Halon ? WTF ? Not so long ago, people would routinely smoke in offices !
- the whole 30 s counter and reset button (inside the area concerned by Halon) also sounds lethal to me by keeping people in the fire/halon area
- the above counter thing doesn't sound like something people wouldn't forget, 2 decades after setup
Hopefully, no-one was hurt this time, but like someone in road security in France I know, is used to say: "There are actually very few accidents vs. the amount that are just waiting to happen (due to people carelessness)".
Re: death trap
Makes you understand why the BOFH and PFY like Halon! Speaking of those gents... perhaps today we get an episode?
Re: death trap
Halon is much less immediately deadly than the now (apparently) preferred method of just flooding the entire room with CO_2
Re: death trap
With apologies to BOFH fans: Halon is not lethal upon inhalation (really; go and look it up) and if you remained in a room flooded with in for a few hours then the worst you'd get is a headache at very high concentrations.
Halon does not work by displacing all of the oxygen in the air (that's a CO2 flood and those are genuinely dangerous) Halon just has a very useful chemical reaction; which is when it has more than a certain concentration in the air then it messes with the chemical reaction that is fire by essentially removing the heat from the flame, which then puts the fire out for as long as the gas concentration remains above whatever the required percentage is. (and at which point this concentration drops hours or days later you'd hope that somebody has removed the source of ignition...)
Alas, it's equally as good at eating through the ozone layer so it's been ditched in modern years. Off the top of my head, modern equivalents are FM200 and Novatec 1230.
There's a wonderful promotional video from DuPont titled "halon 1301 the instant firefighter" that features a demonstration of a lab tech in a plastic box trying to light a fire in a halon filled environment (that he's breathing in). Given DuPont's track record with worker safety, how long he survived afterwards is another question but it certainly doesn't instantly kill or impair.
The mechanism is ingenious: it reacts with burning compounds to render them non flammable. It's really annoying that it's an ozone killer because having a breathable gaseous fire suppression compound that causes no damage really was a boon. I'd have it in my house if it were possible.
Edit: sorry, this was meant to be a response to Peter2
Business as usual
The component you don't test will mandatory fail, generally in a spectacular manner and always at the worst moment.
Need more ....
.... detail about the burst ear drums and associated trauma suffered by the poor peons in the rooms at the time.
--------> Obv!