Yes, I am being intolerably smug – because I ignored you and saved the project
- Reference: 1722583808
- News link: https://www.theregister.co.uk/2024/08/02/on_call/
- Source link:
This week, meet a reader we'll Regmomize as "Nina" who told us about the time she and her boss convinced one of their top clients – a local government authority – to create a maintenance window in its datacenter.
"The hot aisle looked like an absolute rat's nest," Nina told On Call. She got the job of untangling the mess and implementing proper structured cabling, while her boss took advantage of the downtime to maintain software on the authority's server fleet.
[1]
Fixing the servers required them to be online, so Nina and her boss constantly checked in to ensure she wasn't unplugging a critical cable.
[2]
[3]
The job progressed well and reached the critical moment at which the council's WAN would be disconnected.
Such was the importance of that WAN that Nina's boss came into the hot aisle and did the deed.
[4]
Which was what the job required – but also a problem. Because neither Nina nor the boss noted which port the WAN cable had been connected to.
This was OK … for the duration of the planned outage. But the council very much needed to be online once the job was done.
Nina and her boss weren't entirely certain how to do that!
[5]Customer bricked a phone – and threatened to brick techie's face with it
[6]Dangerous sandwiches delayed hardware installation
[7]Stop installing that software – you may have just died
[8]Innocent techie jailed for taking hours to fix storage
Nina found a clue: a small piece of tape above one port had the word "WAN" written on it! Her memory of the cable didn't include that sticker, but she and her boss assumed the presence of the sticker meant it was the right port.
Bad assumption. Connecting the WAN cable to the port did not bring the org back online.
[9]
And the maintenance window was closing.
Nina searched her memory and felt sure the appropriate port was third from the left.
Her boss demurred. He spent a seeming eternity probing the router's interface searching for configuration data, and concluded that the port marked WAN was indeed not used for a wide area connection.
He then decided that port marked WAN should be used for the WAN and set about reprogramming the router to make it so.
Nina pointed out that was a very odd thing to do under pressure – shouldn't they just plug the cable into each port in turn until something worked? That approach might be little more than guesswork, but results could be swift.
The boss forbade any such plugfest, and spent yet more time trying to remember telnet syntax in search of a solution.
Nina decided to just go for it. She plugged the cable into the port she remembered – and by doing so restored the WAN connection within seconds.
"I immediately, and smugly, informed him that I had fixed the network by doing exactly what he told me not to."
The boss was "not happy about my smugness, but extremely grateful that I'd saved the relationship with that client."
"Ever since that experience, the boss developed a newfound respect for my attention to detail," Nina smugly told The Register . She signed off with advice to "Always check the little things you take for granted first, because it will either save a lot of time or be a bigger thing!"
Has a tech support job left you feeling justifiably smug? Share your smirk by [10]clicking here to send On Call an email so you can one day bask in the feeling of seeing your story appear on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZqyuSOd2hNwme6BLhQRfdQAAAMA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqyuSOd2hNwme6BLhQRfdQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqyuSOd2hNwme6BLhQRfdQAAAMA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZqyuSOd2hNwme6BLhQRfdQAAAMA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/07/26/on_call/
[6] https://www.theregister.com/2024/07/19/on_call/
[7] https://www.theregister.com/2024/07/12/on_call/
[8] https://www.theregister.com/2024/07/05/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZqyuSOd2hNwme6BLhQRfdQAAAMA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Simple rule
Photos don't help.
YOU ENTER THE ROOM AND DISCOVER A MAZE OF LITTLE TWISTING PATCH CABLES, ALL IDENTICAL.
This is why the gawd/ess(es) invented multicolo(u)r fine-point Sharpies.
Re: Simple rule
Photos can help - the cables may all look the same and be untraceable, but if you can identify which ports were in use and compare to which are currently in use hopefully there will be a difference of 1. Failing that, it at least helps you narrow down the possibilities.
Re: Simple rule
And you always spent 5 min with some tape and pen, and label those random patch leads. That 5 mins can save an hour of panic later
Re: Simple rule
Never get out of the boat. Absolutely goddamn right, unless you were going all the way...
Re: Simple rule
This is an excellent rule, but one that would have been inapplicable for the part of my tech career that involved pulling cables around--this was the years BCE (Before Cellphone Era). To be fair, a Polaroid camera would have worked, but I never saw one used.
I do remember a long-distance conversation with a customer who had unplugged, then re-plugged wrong. Well, I drew a picture, the customer said. I guess that she could have faxed it to me, but I don't see how that would have helped.
Re: Simple rule
At the (long ago) point in my career when I was plugging boxes in, there were two rules we had to follow:
1) The ops team would not accept a box into service without photos of the front and back, showing the location of all cables.
2) No cameras in the datacentres.
As a nod to my advancing years.....
I now use the camera on my phone to take photos of everything before and during any disassembly/disconnection of anything (even the "simple" things). Thankfully I don't have these automatically uploading to a social media account as it would probably only reinforce my family's opinion of me.......
Re: As a nod to my advancing years.....
I do the same ... Google automatically copies the photos to the cloud and every so often on my phone I get a "remember the day" alert from photos reminding me of what I was doing the previous year, "on a random Thursday", etc ... their AI hasn't yet managed to collate them in to a "plugs and cables" slideshow yet but I wouldn't be surprised if it eventually did!
Every office has one.
The guy (it's always a guy...) who thinks everything can be fixed via a CLI.
But when they just physically broke something? That's a pretty extreme example!
Re: Every office has one.
Um, I think that, these days, there are less and less offices who have anyone who even knows what CLI means.
And I suspect that, the further in time we advance, the more that will be true, even in IT offices.
Re: Every office has one.
fewer and fewer
/pedant
Re: Every office has one.
fewer doing less CLI
fewer CLI doing less
Re: Every office has one.
Those who know the difference between fewer and lesser are fewer, but those who don't are lesser.
Re: Every office has one.
Brain exploding ===>
Re: Every office has one.
Fewer - not as many
Less - not as much
Easy
Re: Every office has one.
Sometimes I have to ask a non-technical client to run a debug command on the command line and redirect the output to a file. I fully understand that they might not grok that greater-than output.log writes to output.log. I was, however, surprised yesterday when I had to explain this to our first-line support guy.
everything can be fixed via a CLI
If you're really "lucky", that guy insists on "automating" those CLI commands in a .bat that is totally unreadable.
Re: everything can be fixed via a CLI
just need a text editor that colours the code.
That instantly makes all languages readable.
I would not entertain looking at a bat in notepad.exe unless lives depended on it
Re: Every office has one.
"But when they just physically broke something?"
An unplugged cable is not b0Rken.
Re: Every office has one.
The connection is broken, though.
Ah, knowing where it all goes!
Back in the days I was working on VME bus hardware, I hated my boss for making me go through the entire chassis, noting down the position of every single DIL switch, jumper & board position before we shipped the kit out.
When we had to swap out a dual port SCSI interface at nearly midnight on the client site, under time pressure and whilst not entirely sure it was a card fault anyway, I was extremely grateful for being able to do a like-for-like, confident that I wasn't introducing an unknown.
Documentation: the hallmark of civilised society ;)
Re: Ah, knowing where it all goes!
Documentation: the hallmark of civilised society ;)
and promptly ignored by the (un) civilized.
Re: Ah, knowing where it all goes!
Documentation: the hallmark of civilised society ;)
In my experience, removed from scope by management because they see it as a burdensome overhead that they don't want to use their budget towards
Documentation and a cheap label maker
Will save the day more often than not. I have always found it useful to label the cable and backplane as you go to prevent hours and hours of “fun” (and by fun I mean really putting the FU back into it).
Re: Documentation and a cheap label maker
If only everyone had your common sense. When you are back to trying combinations of things to bring back someone else's systems (especially if that someone else was a younger version of yourself) always remember that in real life stressful situations there is always at least one more combination of things you can plug in then is mathematically possible - all the theoretical combinations plus the final one that works (think USB-A superposition).
Re: Documentation and a cheap label maker
> USB-A superposition
That's when the USB fits snugly into the ethernet port, right?
Re: Documentation and a cheap label maker
Not quite.
It's also sometimes called [1]Schrodinger's USB
[1] https://global.discourse-cdn.com/boingboing/original/4X/9/6/3/963247bf0e92b25e4b484f8be589297f502f7d42.png
Alarm bells
> what to do at 5:00PM Friday
In my experience working for a *nix systems supplier, 4:55 on a Friday was the time the phone would ring. On the other end would be panicking techy from one of our OEMs who had spent most of the week trying to set up a system for a client demonstration first thing on Monday morning.
They would alway expect us to bail them out, after a week of failure and demands (from their sales rep - sorry: account manager, as they continually reminded us) that we do everything possible to get them working. Including spending the weekend fixing the mess they had got into. Whether we had plans of our own, or not.
Needless to say, none of the management, nor the sales droid (who stood to gain a substantial commission) felt the need for their presence while we toiled. Nor the need to even acknowledge the personal sacrifices made.
As a consequence, all our team would often find themselves engaged in a conference call from about 4p.m. onwards. Discussing the weather, what we were doing on Saturday, where to go for Friday evening drinks and all the other burning issues of the day, that are necessary for a well balanced working experience.
Afterwards did you LABEL the cables?
If not, shame on you ;)
My other half finds it annoying that I use a sharpie to write on the back of all the mains plugs to identify what's connected to the other end.
> My other half finds it annoying that I use a sharpie to write on the back of all the mains plugs to identify what's connected to the other end.
See, another reason why the UK 3-pin plug is so much better - plenty of space for a label!
[1]https://www.fastcompany.com/3032807/why-england-has-the-best-wall-sockets-on-earth
[1] https://www.fastcompany.com/3032807/why-england-has-the-best-wall-sockets-on-earth
Amateur!
I use a label maker
"why-england-has-the-best-wall-sockets-on-earth"
There seems to to be many youtube videos on this subject . Its one of those things the British seem to be irrationally proud of for no good reason.
I usually use a small sticky label to do the same thing. Makes life so much easier!
I use a small zip-tie flag and Sharpie. Even easier, and doesn't get ruined when young Biff or Buffy manages to spill their sippycup onto it.
The form over function battle with the Sharpie..........
Many years ago, a group of us were overnighting at a friend's house. One was looking for a socket to plug in his cassette player (yes, it was that long ago).
Finding a seemingly "useless" plug, he removed it, and left it out overnight.
It was the greenhouse heater (other side of wall) , supposedly protecting a crop of that year's geraniums from a very chilly night.
A very glum look was on the face of the friend's mum later the next day.
but ... wha ??
Such was the importance of that WAN that Nina's boss came into the hot aisle and did the deed.
Which was what the job required – but also a problem. Because neither Nina nor the boss noted which port the WAN cable had been connected to.
Very important job! takes two people to ensure done correctly - like a code commit needing approval.
Not important enough to note basic shit like where you unplugged the wire from though?
Simple rule
Always take a photograph before you start.