I made this network so resilient nothing could possibly go wro...
- Reference: 1730100675
- News link: https://www.theregister.co.uk/2024/10/28/who_me/
- Source link:
As it happens, redundancy is at the core of today's horrifying tale – which our hero, whom we shall Regomize as "Clint", referred to as his "brown trouser day." We very much hope he had a spare pair with him – redundancy, you know.
Back in his halcyon youth, Clint worked as a network admin for a credit card company. The biz worked out of two offices, right across the road from each other. They were joined by a dedicated fiber, connecting a pair of Cisco 6509 routers – very modern for the time.
[1]
Of course that meant that there was only one "path" between the two buildings. Given both offices housed huge call centers and hundreds of employees, this was seen as far too risky. A second fiber link was installed, and a second pair of 6509 routers. So if one of the paths between the two buildings failed, the other could easily handle the traffic.
[2]
[3]
Building this second path had been Clint's special project, and he was rightly proud.
One day, after it had all been set up and tested, he found himself bored – "having finished reading The Register ," Clint wrote, because he knows we are suckers for flattery – and was nosing around the ports via Telnet.
[4]
He noticed that one of the ports was returning an error condition. After checking that no-one seemed to be having any issues with the network, he assumed this meant that the redundancy was working as intended. All he had to do was toggle that port, and all would be well. No-one else would even notice!
So he typed the appropriate command, hit enter, and …
[5]Linux admin asked savvy scientist for IT help and the boffin blew it
[6]Compression? What's that? And why is the network congested and the PCs frozen?
[7]After we fix that, how about we also accidentally break something important?
[8]Personalized pop-up was funny for about a second, until it felt like stalking
His session immediately ended. And he could not reconnect.
People around him started wondering what in the world was going on with the network. The VoIP-based call center fell deathly quiet.
Clint instantly realized what he had done. He was logged into the wrong router – the link to the other building was obviously down, so he'd been shuffled over to the router in the building he was in, without realizing it. He could see the error on one port, but in attempting to enable it he had in fact flipped the wrong switch and disabled the path that had been working perfectly well. Now there was no way to get back in without physically connecting.
[9]
He grabbed his laptop and a serial cable, then sprinted across the road like a man with a mission – for indeed he was a man with a mission.
He reached the router, connected to it, typed the appropriate magics, and everything came to life. For good measure he cleared the logs – because no-one needed to know, right?
Just then the head of the IT team came in looking for someone to blame. He found Clint moments after he made the save. Naturally our hero won praise for having acted so quickly to fix this mysterious fault, for which no cause could be found. Clint suggested additional monitoring on the ports – in case of future strange dropouts.
As Clint told us: "Fail fast, learn fast couldn't come fast enough!"
Have you ever had a "brown trouser day" like Clint? How did you manage to talk your way out of it – or even end up looking like the hero? [10]Tell us all about it in an email to Who, Me? and we may share your exploits on a future Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zx9u20x1tDYrMVKhYc4YTAAAARY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zx9u20x1tDYrMVKhYc4YTAAAARY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zx9u20x1tDYrMVKhYc4YTAAAARY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zx9u20x1tDYrMVKhYc4YTAAAARY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/10/21/who_me/
[6] https://www.theregister.com/2024/10/14/who_me/
[7] https://www.theregister.com/2024/10/07/who_me/
[8] https://www.theregister.com/2024/09/30/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zx9u20x1tDYrMVKhYc4YTAAAARY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Luckily for Clint this wasn't terminal for his career...
Even so, I bet his trousers were a bit shiTTY...
He pulled the trigger too fast
on that command to toggle the port.
Must be something in his regomised name.
I'll get me coat
Re: He pulled the trigger too fast
I guess you're running to the Eastwood...
Re: He pulled the trigger too fast
He doesn't want his coat; he wants his poncho.
Re: He pulled the trigger too fast
But then he'd have been regomised as "Null" or "None"?
Re: He pulled the trigger too fast
Guess the "punk" didn't feel too lucky after that !
I hope the network outage didn't cost his employer a packet...
Ah, remote network administration...
I spend a whole lot of time managing network equipment remotely, and i've had my fair share of "well, better grab my car keys" moments.
Recently i was managing a network containing 5 netgear switches. The customer didn't want to pony up for stackable switches, so i ended up having to go through each switch manually to make some changes to the trunk ports.
They had about 15 non-consecutive VLANs, so i essentially had to enter a comma-separated list of them, twice. Once the list of "allowed" VLANs, and once the list of tagged VLANs. Our management network was untagged over the trunk ports. Guess who forgot to remove VLAN ID 1 from the list of tagged VLANs on one switch?
I'm not completely sure why this messed up the network as bad as it did, the other VLANs should've kept working without problems, but a short car drive was required either way.
Usually when i lock myself out it's a bit more boring, just small errors when configuring WAN settings.
redundancy and diversity?
Instead of a second pair of Cisco boxen, Clint* might been wiser to go for another vendor with an obviously different CLI. Juniper or... Huawei ;). I recall the latter's new enterprise switches around the turn of the century were a fraction of the price of the US vendors. Presumably why so much fun has been had pulling them out later.
On hosts I *always* include the hostname (via $(hostname --short)) in the $prompt/$PS1 especially for superuser accounts. Some who ought to know better when replacing a running machine have the new machine with the same hostname *and* on the network but with different IPs. Plenty of scope there for "les culottes chocolate" (brown breeches.)
* the cinematic references abound " do you feel lucky punk ", or " a fist full of dollars " as well as " for a few dollars more " and not forgetting " the good, the bad and the ugly " each of which could apply quite generally to any of the large IT and network kit vendors but perhaps without the "good."
6509 was a chassis switch not a router
cisco 6509 was a chassis switch, yes it was L3 & also you could install a firewall module, redundant supervisors etc etc.
proper beast with its woeful blocking backplane as its achilles heel.
https://www.cisco.com/c/en/us/support/switches/catalyst-6509-e-switch/model.html
firewall module
https://www.networkstraining.com/cisco-firewall-service-module-fwsm/
cisco 7600's where the routers,
along with the old school HP laser printers, the 6500 series would be still operational post the apocalypse.
Re: 6509 was a chassis switch not a router
Pretty sure the 6509 supervisor engine featured routing, so it could be used as a router
IIRC it was the replacement for the CAT5xxx and RSM (router switch module), to perform inter-vlan routing and switching.
reload in, or its vendor equivalent
When the stuff you are managing is remote youdont get the luxury of nipping across the road to fix it.
before doing any thing, save the config.
next do a reload in 5.
when the inevitable happens you just wait for the box to reload into its saved config.
juniper has its commit confirmed equivalent which just reverts the config instead of reloading the box.
Clint (in all caps) can be used to get past basic word filters. Just sayin'
Ah, the days when you had 40 terminal windows stacked around two 17" CRTs... and being an old-school *nix guy it was focus-follows-mouse, not click-to-focus.
The paranoia over accidentally sending the command to the wrong system was legendary. I spent several hours writing scripts and terminal config files so that dev systems were green themed, QA/test orange and production red. Used various kludges and hacks to get them to pick up when I had a root shell open and invert their colors too.
One of my colleagues though I was being overly silly, and I admitted that I mostly did it because I wanted to find out if I could and if it made a difference (and if I could make it look cool into the bargain, compared to the "vanilla microsoft" look of the desktops belonging to people NOT admitted to the inner circles of systems administration and therefore not provided with a "real workstation" at their desks)...
Until the day that same colleague sent a shutdown command into prod rather than dev and asked me for copies of how I did it.
Every network admin has done this
And the ones who say they haven’t are either lying of have never worked on a production system.
It is a standard interview question round here, we all know you will have done something stupid but how you behave after doing said stupid thing is what is important.
Personally honesty is the best approach, own up before someone finds out it was you and you tried to hide it.
Re: Every network admin has done this
"It is a standard interview question round here, we all know you will have done something stupid but how you behave after doing said stupid thing is what is important."
And just as important is the reply a candidate gives. Same league as "whats the biggest mistake you ever made".
If a candidate replies that they have never inadvertently shutdown the wrong server / device interface Im instantly suspicious. Liekwise if they claim to have never made a mistake, they are either fibbimg, or I wonder how they would respond when they do make the inevitable mistake.
Reminds me of my first FDDI testing
when I learned the acronym really means Fall Down Drug Inducing when I tested between 2 buildings and didn't realise the fibre vendor had mis-built the connectors on one end. the server was live, but the disk network was dead...epicfail on our part!
Oh I think all of us at some time have entered a command in the wrong terminal. But rarely do they have such a dramatic and difficult to recover result.
At least Clint's other office was not 2000 miles away...