Techie climbed a mountain only be told not to touch the kit on top
- Reference: 1706862668
- News link: https://www.theregister.co.uk/2024/02/02/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Edmund" who once worked for a very large internet service provider and was responsible for the operation and maintenance of several routers that were essential for the provision of broadband services in towns near his home.
These were the sort of routers that could easily support hundreds or even thousands of customers. Edmund's job was complicated by the fact his routers were mostly located in cold and mountainous locations.
[1]
So when the call to fix one came, it was of course winter, as the sun was going down, on a Friday, and the router was two hours away by car.
[2]
[3]
"The network operations center (NOC) requested I head to a site and power cycle one of the two redundant routers that carried all the traffic for the town," Edmund told On Call. "I was about two hours away, but it wasn't customer impacting so it wasn't a big deal other than my Friday night activity was canceled."
So off Edmund went. When he arrived, he joined a conference call in which he was asked to connect his laptop to the console port and share his screen before he cycled the router off and on again.
[4]
Edmund did as he was asked. As soon as he connected he saw a terminal filled with gibberish, and was swiftly told that NOC staff knew what was going on and that the router must be replaced.
Our hero reminded his NOC masters that he hadn't yet power cycled the router and was told that didn't matter.
His new instruction: wait for the vendor to deliver a replacement – which should take four hours to arrive.
[5]
After buying himself dinner, then checking the weather forecast and learning of imminent snowfall, Edmund rated his chances of receiving the replacement as very low. Around midnight the NOC agreed and sent him home because the router was not going to make it up the hill that night.
[6]Standards-obsessed boss ignored one, and suffered all night for his sin
[7]Junior techie had leverage, but didn’t appreciate the gravity of the situation
[8]While we fire the boss, can you lock him out of the network?
[9]Tech support done bad sure makes it hard to do tech support good
Come Saturday, the news was better. A courier would arrive at 10:00 AM, but as the router weighed over 100 pounds (40kg) Edmund would need a colleague to help move it into place atop a six-foot rack.
Edmund duly found a colleague, hit the road, schlepped the router into the facility, and dialed in to a conference call with his NOC colleagues.
Edmund didn't fancy a full router replacement, so reminded his remote collaborators that nobody had power cycled the machine. "Oh, you didn't do that?" was the reply
Edmund reminded them "You guys specifically said not to!"
"You can guess what happens next," Edmund told On Call. "I pulled the power, and it came back up like nothing happened. Twelve hours of overtime and a wasted trip because of an 'expert'."
This story had a strange sequel: three months later, Edmund was asked to locate the defective kit, because he had sent the unused replacement back and the serial numbers didn't match.
Have you ever been prevented from power cycling? Or sent somewhere cold and nasty for a futile job? We're sure some of you have and we want you to [10]click here to send On Call an email that shares your story.
Many of you very kindly say you enjoy this column. Step up to support it with an email! We never breach a confidence or write in a way that will land you in trouble. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZbzLXUQwggdJBRC2hUB8gwAAAE0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbzLXUQwggdJBRC2hUB8gwAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbzLXUQwggdJBRC2hUB8gwAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbzLXUQwggdJBRC2hUB8gwAAAE0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbzLXUQwggdJBRC2hUB8gwAAAE0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/01/26/on_call/
[7] https://www.theregister.com/2024/01/19/on_call/
[8] https://www.theregister.com/2024/01/12/on_call/
[9] https://www.theregister.com/2024/01/05/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: A wasted trip
What's not to like? Except he should have brought his skis as well ;) then it would have been a good trip. Ski down the mountain, see how many runs you can do before the equipment arrives. And Saturday is a good day for skiing as well, fresh snow etc.
(I never managed to land that job at WSL, pity, though I'm more for "going on ski" rather than "standing on ski" as the Norwegians say...)
Re: A wasted trip
I'd have challenged the decision as to why a power cycle wasn't an option and said, "I'm stood here in front of the bloody thing, you lot aren't. Standard procedure for any hardware issue like this is to reboot. Even the vendor will tell you to do that."
If there are enough technical people on the call, they'll probably support you as you're being completely sensible. If the router comes back up and there are still concerns about it's functionality, they can schedule a replacement at a later date.
Re: A wasted trip
"I'm stood here in front of the bloody thing, you lot aren't.
But they are looking at the same thing on a shared screen , so they are in effect there.
If they're the router experts and a screen of gibbesrish is telling them that , who are you , as a subordinate , to question that?
Could be many factors you dont know about .
The gibberish could be telling them "Dangerous heat levels in the PSU" which they have previously seen cause a fire.
That said a polite enquiry just for enlightenment might be nice ,
and also because the "turn it off an on" is a rarely not worth trying
Re: A wasted trip
"and also because the "turn it off an on" is a rarely not worth trying"
Reminds me of an early '80s AI koan ...
A novice was trying to fix a broken Lisp machine by turning the power off and on.
Knight, seeing what the student was doing, spoke sternly: “You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.”
Knight turned the machine off and on.
The machine worked.
Re: A wasted trip
>The gibberish could be telling them "Dangerous heat levels in the PSU" which they have previously seen cause a fire.<
In which case the only sensible instruction would be to do the first half of the power cycle - power it off and do it now.
I think I'd have said: "Ok, to be electrically safe I'll power it down - could be a short circuit". Then "accidently" powered it back up "for testing purposes".
Indeed, or the classic "dust in the power connection", where the only cure is to power down, pull the cable out, blow into the end of it and then reconnect it and test...
I once had ....
.... a badly designed 10base2 network that I had the misfortune to support.
Some bright spark had extended the segment that all the servers (in the server room) were connected to into the office outside. The sockets used were a weird (IIRC German) make before break connector. Anyway, one afternoon the whole network started glitching and everyone went into headless chicken mode. I managed to trace the fault to the office portion of the network segment, and informed my boss that it would be necessary to unplug it to find the cause of the issue.
I was told that I was not allowed to do that during office hours, and then my boss disappeared into a meeting. So I sat around doing nothing for 2 hours while the network ran at about 10% speed.
Returning to the area after a comfort break I spotted the rear end of my boss exiting the office as he finished his day, without even bothering to check with my on the current emergency situation.
Finally after everyone else had finished I was allowed to do the 5 minutes of work to identify which of the make before break sockets had failed, replace it, and everything came back up.
.
Within a week I had isolated the server room with a router (or bridge) that meant that office network issues only affected that segment rather than the whole network.
That was one of the few jobs I've ever walked away from because I needed to save my sanity.
Re: I once had ....
Back in the 80s we had to install a bridge to isolate the consultancy section of the office. They all used the then new Amstrad PC compatibles, which would randomly stop forwarding the tokens on the Token Ring Network and bring down the network for the entire building. We separated them with the bridge and let them get on with it.
Re: I once had ....
Those 10base2 sockets were a boon and a PITA!
Due to multiple buildings and segment lengths I was stuck with having offices and the computer room on one segment, with one bridge half way down the main building, another at the far end from the computer room and the third in the next building.
The separate building was occupied by a sister company who had a server of their own as well as using the main AS/400. That was all very well until their support people decided to use two network cards in the server and configure it as another bridge to add a few more PCs onto the system, without actually consulting anybody.
I see what you did there!!!
a reader we'll Regomize as "Edmund"
>>>>>>>>
Re: I see what you did there!!!
I don't? But I'm not firing on all cylinders today.
Could someone spell it out for me.
Re: I see what you did there!!!
It's such a Hillary-ous pun, this one.
Re: I see what you did there!!!
It's not entirely clear to me either. But possibly a reference to Edmund Hillary, who was famous for climbing a snow-covered hill?
The only other famous Edmund I can think of is the one in the Lion, The Witch and the Wardrobe, and I can't make that fit :)
Re: I see what you did there!!!
Also - no Blackadder references.
Re: I see what you did there!!!
Oh, but surely we all reach for the words of Edmund Blackadder from time to time?
"Whatever it was, I’m sure it was better than my plan to get out of this by pretending to be mad. I mean, who would have noticed another madman round here?".
Had a similar thing happen
Not going to mention names/companies. But I worked for a software developer about 18-19yrs ago. I was sent to Newbury (about a 100 mile trip) to fix an issue with a venue that couldn't be done remotely.
Upon arriving I was met with a system that kept crashing. As I was told to speak to the higher ups in tech support (who were too busy to go and sort it) before doing anything... I did so.
I was instructed to do about a dozen different things, none of which worked... and none of the suggestions involved actually opening up the system.... I'd suggested several times that I pull the system and open it up as it could be a hardware issue... This was dismissed and scoffed at, it was absolutely a software issue and they'd figure out what it was eventually. (every crash, they'd spend ages pouring over logs before making the next suggested change)
By the end of the day, it wasn't fixed... and I was told to find a cheap hotel for the night and be back there at 8am in the morning... because the support team were going home.
So I took matters into my own hands... pulled the system and stripped it down.
I took photos on my crappy little phone as evidence of what I found.
This thing was clogged up with dust, and the fan on the CPU cooler had failed... it was crashing because every time they tried to run tasks on it... it overheated and shut down, or after it was left on for a little while... heat soak caused the same issue.
I cleaned it out, went through their spares and old systems looking for a compatible heatsink... found one, cleaned it up and installed it. problem solved... support team very pissed at me the following morning.
But the venue, gave me a glowing report and gave the company a load of crap over the wasted day... I was lucky that one of the venue managers was present for a lot of the calls.
Sadly the company never learned the lesson... about 6 months later, started laying people off to cut costs. That same support team still held a grudge and kept sticking the knife in at every opportunity because I embarrassed them and I was one of those earmarked to be laid off.
The day after I was told... I never bothered going in and went straight into job hunt mode.
That's when I decided to become a contractor, I grabbed a temp job at a place I'd worked before to bring in money and was in a new contract with a 30% pay increase within 6 weeks... I've never looked back.
Re: Had a similar thing happen
Yup, as I often say when I'm teaching some of our new recruits on troubleshooting, there's never been an investigative tool or procedure produced which is better than the mk 1 eyeball.
It's amazing how often we get called in by customers to downed machines that have been that way for days (or even weeks) and they've done all their meetings, model based problem solving, devising of tests to run and analysis of data etc in nice cosy meeting rooms and have never actually ventured into the cleanroom (for background, I work for a semiconductor equipment maker) and looked at the damn thing.
Then you walk in there, take one look and have a "there's your problem" moment (to quote Adam Savage) when you see something on the floor or hanging off that shouldn't be...
Not Putting the Cart Before the Horse
I've seen situations where a cable hanging off the back or side of a piece of machinery or on-the-floor tower computer was cut or yanked out by a bunch of Facilities people moving heavy equipment on one of those very-low-to-the-floor carts. The floors of those carts stick out past the wheels, making it easy to catch things, and the weight and inertia of the cart+thing_on_the_cart ensures the people pushing it usually don't notice the brief impact.
This week, meet a reader we'll Regomize as "Edmund".... Hillary-ous.
Hillary-ous
Don't you Ever rest?
Remote people might be right
Isn't there a chance that the remote people looked at the "gibberish on the screen", and understood that the error message meant that if rebooted the router had a 90% chance of not coming back up again, and as everything was currently working a reboot would likely leave 100's of people unable to work until it was replaced, so it was best not to do anything until a hot swap was available.
The Regomiser …
… played a blinder there - “Edmund” indeed!
Did he find himself Tenzing up a bit as he talked to his remote colleagues?
The puns are reaching new heights today
Not quite the same, but the end of the article asks if you've ever been sent somewhere unnecessarily.
First job out of college was working for a small contracting place. I happened to be on the biggest contract, which could cover the entire payroll on its own. Anyway, my manager and the on-site person, absolutely hated each other. No idea why, but the on-site person was shall we say... difficult to get along with. Generally all I'd ever have to do is mention that I work in the department she managed and I'd get a knowing look and nod of sympathy.
Anyway, one day said on-site person comes along and is quite insistent that I go and look at a specific system. It's out in the guard house for the campus entrance. Like every five minutes it was, "When are you going to go look at that system?" So, I go out there, do my usual bit to see if I can troubleshoot it a bit to determine if there was a hardware failure. Think I spent maybe an hour on it running hardware diagnostics to see if anything popped up. Unbeknownst to me, this was a "time and materials" ticket, which meant basically all I was supposed to do was send it over to the "home office" and they'd send someone else out who could then bill separately for it. I have absolutely no doubt that I was deliberately sent out there just so the on-site person would have an excuse to yell at my manager. It wasn't the last time it would happen, and after I started learning enough about things to not get caught up in as much, I was removed from the contract.
Book suggestions?
Speaking of networking: does anyone have suggestions for a good, basic book (or website) about networking for small businesses?
TIA
For the weekend -->
A wasted trip
Yeah, but 12 hours of overtime.
This is what would happen in the days before smartphones. Then again, in this case it wouldn't have helped since the numpties mangling this job were idiots.
I suspect that the one competent guy was the one who sent him out to power cycle the router. Then he went home and the night shift came in without instructions, so they bungled the rest.
I call that a lack of internal communication.