Lesson 1: Keep your mind on the ... why aren't the servers making any noise?
- Reference: 1692603312
- News link: https://www.theregister.co.uk/2023/08/21/who_me/
- Source link:
This week, for your amusement and delectation, we present a reader we'll Regomize as "Hans" who once worked for a small software development house that also dabbled in application hosting.
Now, as luck would have it, the firm for which Hans worked was something of a leader in the field of WAP applications – back when WAP stood for Wireless Application Protocol, rather than something more lyrical and NSFW. Look it up if you must - but not if HR monitors your machine.
[1]
Anyway, in those pre-iPhone days WAP was quite the thing for mobile devices, so the small development house had landed a contract with a state-owned telco to build and host WAP, plus some SMS services.
[2]
[3]
We should mention that the building in which the business was housed had previously been used by a rather larger concern, and so was quite well equipped to host hosting infrastructure. It boasted a cooling system, server racks, and what Hans described as "a rather scary halon fire suppression system".
The fire suppression system does not feature in this story. Don't get your hopes up.
[4]
When the previous tenant moved out, it had left something of a mess of power and network cables snaking around under the floor. For whatever reason, Hans's firm had not seen fit to clear that away upon moving in, but rather had compounded the snarl it by installing its own equipment. As time passed and more servers were added, the tangle had reached a point at which Something Had To Be Done.
Hans was thusly handed the task of bringing order to chaos. The first task was to figure out which cables under the floor were not, in fact, doing anything. This was done using the time-honored method of jiggling each one until he found both ends. If one end was plugged in and the other was not, that cable could go.
[5]The price of freedom turned out to be an afternoon of tech panic
[6]Techie's quick cure for a curious conflict caused a huge headache
[7]Nobody would ever work on the live server, right? Not intentionally, anyway
[8]Linux lover consumed a quarter of the network
And so Hans proceeded about his noble task – jiggling, tracing, unplugging – gradually reducing the tangle to a mere knot.
As he was doing this, a colleague tapped on the window. Still holding the cable he was tracing, Hans opened up and chatted with said colleague for a moment. Not a long moment, but enough of a moment to be distracted.
Not very distracted, but just distracted enough to forget whether the cable he was holding was redundant, or essential.
[9]
Guessing, he unplugged it.
"Things got quite a lot quieter as a pile of 10Krpm SCSI drives stopped spinning and a whole lot of fans went silent," Hans recalled. He had disconnected power to the server rack belonging to that telco. This was not good.
He tried plugging it back in, but to no avail. Upon investigation, he found that attempting to plug the whole rack in at once was popping a circuit breaker. He'd push the breaker back in, but each time he reconnected the power it popped right back out.
You and I might suggest powering down the machines, reconnecting the rack, and powering the machines back on one at a time. Hans thought of that later. After he stopped panicking.
Before then he ran from the room looking for help and found the resident electrician. Hans explained the problem, and the bright spark said he could help.
The electrician pressed on the circuit breaker switch and told Hans to plug the rack back in. This is the bit where you might be expecting that scary Halon fire suppression system to play its part. Hans was concerned he may be tasked with cleaning up pieces of exploded finger from the server room.
Thankfully you worried unnecessarily about both fates. The machines powered on, the drives spun up to speed, and a pleasing hum once again filled the air.
Told you not to get your hopes up.
There were no serious repercussions for Hans from the incident, but he did learn an important lesson about chatting with colleagues in the middle of an important job.
Ever allowed your mind to wander for a moment and not liked where it ended up? Tell us all about it in [10]an email to Who, Me? and we'll … oh hi, how are you doing? A cup of tea sounds lovely. No, I'm not doing anything important …
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZOM1wlIkuHbgIaGndRMQoAAAAMw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZOM1wlIkuHbgIaGndRMQoAAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZOM1wlIkuHbgIaGndRMQoAAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZOM1wlIkuHbgIaGndRMQoAAAAMw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/08/14/who_me/
[6] https://www.theregister.com/2023/08/07/who_me/
[7] https://www.theregister.com/2023/07/31/who_me/
[8] https://www.theregister.com/2023/07/24/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZOM1wlIkuHbgIaGndRMQoAAAAMw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Been there done that
Sounds shocking...
Re: Been there done that
I hope you didn't feel to under the weather after the shock...
All fun and games until...
"This was done using the time-honored method of jiggling each one until he found both ends. If one end was plugged in and the other was not, that cable could go."
...some idiot decides to re-use a cable with a broken clip and you jiggle it right out.
Dodgy cables are the bane of my existence.
Re: All fun and games until...
That is why I always cut off the RJ45 connector when the clip comes off. Colleagues have been shouting and screaming at me for doing so, but it is better to have a hard failure than a heisenbug. Change the cable or replace the plug I say.
Interestingly, circuit breakers were redesigned some years ago to prevent exactly this scenario and in the event of overcurrent cannot be held closed.
Circuit Breakers
We had to test a machine driven by a large motor supplied via diesel-generator. The motor-absorbed current when running was a bit less than the 600-A rating of the breaker but the starting current/time were too high/long to prevent a (correct) trip despite judicious adjustment of the 'settings'. These breakers are only designed for a small number of trips on high current so it wasn't something we could 'experiment' with for long.
My older, wiser, bolder, colleague suggested we reduce the set-speed of the engine, set the AVR (automatic voltage-regulator) low and on manual control, connect and close the circuit-breaker (operating it from an external supply) and start the engine, alternator and motor all together when the loads, voltages and currents would be much reduced. We could then bring the whole set up to full speed within the trip parameters.... It needed good coordination between us and was a bit scary initially, but it worked..... To be fair, we did ask the genset supplier for his thoughts which were initially unprintable but their engineer was curious, pliable, interested and eventually persuaded. He came to see the procedure but stood well to one side with his hands firmly in his pockets.
We used this procedure a few times until inverter-drives became more available making the process routine by comparison. We would not be able to do this with modern gensets.
Reward --->
Sequential switches are your friend
In the dim distant past I worked for a firm that built complete bespoke Audio/Video systems in 6ft racks for people with more money than sense. If the whole lot came on at once it would take out quite hefty contactors, so each rack was fitted with these switchers. For one rather large auditorium we even have to have built a 'slow' sequential switch for the lighting system.
One of the more valuable things that learning to fly (unpowered aircraft in my case) has taught me is: Either you finish what you are doing before you let yourself get interrupted or start over from the start after an interruption. There is no in-between solution and getting interrupted is ALWAYS a choice. It's not a choice some people understand but that's their problem.
Upstream breaker
"He'd push the breaker back in, but each time he reconnected the power it popped right back out."
Many circuit breakers have two trip mechanisms, a fast magnetic one based on overcurrent (which Hans tripped - protects against shorts and gross overloads) and a slower thermal mechanism to protect against limited overloads.
If you trip a downstream breaker a couple of times on overcurrent then the upstream breaker might warm up and eventually trip. And often you won't have access to it, and it will disconnect more than just your own equipment :(
I hate it when people interrupt me in the middle of something.
This means that I have to start from the beginning, or sit for a long while staring into space wondering where I was before the interruption.
Some days I just get so disillusioned with the interruptions that I go for a walkabout, just to clear my head.
What is the worst sound in a server room?
No sound at all. Bah boom!
(How often have I posted that here?)
The idea of holding in a breaker to stop it doing its job reminds me of the old southern US steamboat ballad "Jim Bludso" which mentions "A (no longer used racial epithet) squat on her safety valve".
Did the WAP phones support Hans-free?
Been there done that
Fortunately it was only me that suffered...
Was doing unit test work on a small weather radar display system and had just checked the HT side of the display was running correctly as 15kV. Got the urge for a coffee so made everything safe and wandered off to get one and once back I picked up where I had left off in the test script. Sadly as we all know following a script does not require thinking to be engaged and I was wondering what that tingling was when I realised I was leaning on the HT lead. I know there was hardly any current but even so I was very very lucky