Network engineer chose humiliation over a night on the datacenter floor
- Reference: 1732519746
- News link: https://www.theregister.co.uk/2024/11/25/who_me/
- Source link:
This week our hero, if that is the appropriate term, is a reader we'll Regomize as "Erik" who works as a network engineer and was once asked to install a set of new switches for a client that was rebuilding its network topology.
The client had several rooms in which vital networking equipment was located and had installed appropriate security for the purpose – including both physical and electronic keys. In some of the rooms, Erik found a collection of shiny new uninterruptible power supplies into which the new gear was to be plugged. In one room he found a sole, lonely, router whose purpose was unclear except that it was not part of this project particular upgrade.
[1]
To install the new stuff Erik was obviously going to have to unplug things, so an after-hours maintenance window was locked in. Erik was provided with the appropriate keys that would let him into all the right rooms to get the job.
[2]
[3]
The job went well and by around 1:00AM Erik had done all the necessary unplugging, reconnecting, and software updates.
All seemed to be working brilliantly, so he returned to the main lab, which had been his base of operations, to begin packing up.
[4]
Just then the phone rang. The client employed monitoring services to keep an eye on its network remotely from another building across town (for security reasons). And according to that remote monitor, nothing was working. At all.
Erik immediately understood what that lonely router sitting by itself in a room must be for. He reasoned that if he rebooted it, all would be well.
He therefore sprang up, left the main lab, and made his way through the empty corridors to the room where the router could be found.
[5]
And as the door closed behind him, he made a horrible realization: he had left the keys behind in the main networking lab with the rest of his gear.
He was therefore locked in the client's premises, and could not escape.
[6]Undergrad thought he had mastered Unix in weeks. Then he discovered rm -rf
[7]The sad tale of the Alpha massacre
[8]Relocation is a complete success – right up until the last minute
[9]I made this network so resilient nothing could possibly go wro...
He figured he had two choices:
Total humiliation, in the form of waiting until morning when the client would arrive and free him;
Partial humiliation, in the form of calling his boss in the middle of the night to hopefully ask if the client had supplied an extra set of keys.
Erik chose partial humiliation, and luckily his hunch was right. Extra keys had been provided, the boss rode to the rescue, and the client was none the wiser.
Incidentally, Erik's other hunch – that restarting the monitoring company's router would fix everything – also proved well founded. The job was done, and Erik was liberated by 2:00AM and even managed a half-decent night's sleep.
Have you ever come up with a quick-thinking solution and then realized, like Erik, that perhaps thinking a bit more slowly might have been better? That's just the kind of story we love to read. [10]Tell us all about it in an email to Who, Me? and we may feature your exploits on some future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z0RY1tJudNbAEDmQc2xcCQAAAA4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z0RY1tJudNbAEDmQc2xcCQAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z0RY1tJudNbAEDmQc2xcCQAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z0RY1tJudNbAEDmQc2xcCQAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z0RY1tJudNbAEDmQc2xcCQAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/11/18/who_me/
[7] https://www.theregister.com/2024/11/11/who_me/
[8] https://www.theregister.com/2024/11/04/who_me/
[9] https://www.theregister.com/2024/10/28/who_me/
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Murphy's Law also *loves* demos
Our university ICT team once proudly wanted to demo the shiny new system they had for keeping track of ALL publications of the university, along with a shiny new search facility. Loads of dignitaries were invited for the event, and a seasoned professor volunteered to be the guinea pig for the demo of the search facility. Search result turned up with about two publications, for a career spanning decades. Cue red faces all round.
As a rule of thumb, the more people watch, the more likely a demo will fail (dignitaries count double).
Re: Murphy's Law also *loves* demos
Well, the search worked . . .
Re: Murphy's Law also *loves* demos
... for a given definition of "worked"
Re: Murphy's Law also *loves* demos
This is also as known as Cohen's law:
"If it can go wrong, it will at the demonstration"
N.B. Aways remember
Murphy was an optimist
Firewall configuration
A long time ago, back when I was a techie instead of an architect, and so doing more interesting work, I had to fix a firewall problem. After investigating, I found that two commands would fix it - the first to remove the problem and the second to configure it correctly. The first also happened to remove remote access.
Fortunately, I happened to be in the same building at the time.
Re: Firewall configuration
Many employers ago, we were having issues with a serial link between two offices (the same link that eventually had a hotel built in the line-of-sight that I've mentioned in a comment here before). The issues usually cleared up after a quick shut/no shut on the router ports, something we had to do every other week or so.
We'd recently moved the technical staff to new office on the other side of that link. As the link started to act up again, one of our senior hotshots did what he usually did when this happened: He logged on to the router, typed the shutdown command, hit Enter ...and then his face turned from healthy pink to pale to puce in a few seconds. He of course didn't factor into the equation that we'd moved, so he'd shut down the INSIDE of the link, the way he usually did. Though, of course, since we were now on the OUTSIDE of said link, he'd cut himself (and all the rest of the office) off from the network. It wasn't too far (it was line-of-sight, after all) but long enough that he'd have to drive between the sites. That's when I flipped open my Nokia 9000 Communicator, logged onto the Cisco AS5300 via the Nokia's built-in modem, hopped to the router terminating the link and opened the interface back up for him.
About a month later, I was promoted to senior myself. Not sure that that situation was the triggering factor, but I'm sure it helped.
Re: Firewall configuration
You need to earn at least 10 "oops" badges without needing to explain what happened to become fully qualified to tinker unsupervised.
I have a warehouse to store my badge collection...
Re: Firewall configuration
A very long time ago - back before I had graduated Uni and my IT career had even begun, I was earning beer money doing evening & night-shifts at a Mobil petrol station across town.
During my day-time induction no-one had thought to show me how me how to turn the outside lights on when it got dark (I mean, obvious, right?) In hindsight, moving a big lever switch to the "Off" position was never going to be the correct choice. In my defence, I was young, stressed, feeling a little bit out of my depth, and the big switch was very compelling...
I had to wake my boss up at home (No idea why he was asleep - it can't have been later than 22:00), and he did a sterling job of stepping me through the site power-up process from memory. Apparently if I'd left it 5 more minutes before ringing him, the pumps would have lost their internal config and needed re-programming.
Somehow I kept my job - I even got a reputation for being one of the more reliable members of staff, which might go some way to explaining why you don't see Mobil around any more...
As I get older, this sort of thing is happening more often. I have noticed that the "onosecond" [1] : [wordsense] is now more like an onominute.
[1] https://www.wordsense.eu/onosecond/
keys
Years of working in downtown Belfast in the 1980s taught me to always keep my keys on me. There's nothing worse than popping out of the office for a piss, only to hear the evacuation message over the tannoy, and realising that you won't be able to get your car or house keys until whatever it was is resolved.
Once I was over in Japan witnessing factory acceptance testing on behalf of a client; the testing was for equipment that would provide an inter-site connection across a world-wide set of ground stations (never actually deployed, but that's another story). The testing was scheduled to be conducted over about 1.5 days, with another half day as contingency (I had insisted on that - in my experience, Murphy's Law *loves* acceptance tests).
Everything went well until about halfway through the afternoon when one of the critical test had failed, badly. Repeated attempts resulted in the same wrong behaviour. As it stood I would have to fail the entire test campaign which would have left the project team with a major loss of face (this is Japan, so loss of face is treated very seriously). I discussed things with the test manager and suggested that I just settle myself into a chair in the corner of the room out of the way while his team tried to sort the problem out. This offer was gratefully received since it would allow them to at least avoid the official black mark that was heading their way.
By the end of the day the problem had not been resolved and I headed back to the hotel. The next day I came back to site and was greeted with a beaming and still very grateful) test manager; they had found and fixed the problem at about 2:00 am in the morning (it turned out that they had fumbled reconfiguring routers from one mode for testing to another mode, resulting in the router having some form of weird hybrid configuration that could not do anything sensibly). The downside is that the factory doors had been locked at 19:00 so the staff fixing the problem had been locked in and had to sleep on the floor.
Testing was successfully finished that day using some of the contingency time I had insisted on.