Beware the big bang in the network room
- Reference: 1644827287
- News link: https://www.theregister.co.uk/2022/02/14/who_me/
- Source link:
Today's hero is a Register reader Regomised as "Colin." Colin had accepted his first proper role as a network engineer for a small UK consultancy. Up to that point, his networking experience had consisted of swapping out the hub of a previous employer for a switch and getting the company's self-hosted web site and email server up and running.
He was confident. Gosh, he was confident.
[2]
Our story takes place in the early part of the last decade. The client had a London data centre and he was tasked with swapping out some of the switches within for modern units, replete with 10 gigabit uplinks.
[3]
[4]
"Regarding myself as 'not a total cowboy'," he told us, "I had surveyed the racks in advance and recoiled at the sprawling mass of Cat 5 cables that had 'grown organically' around this 48-port switch and confirmed that yes, of course there wasn't an empty slot in the rack..."
"I was quite new to the whole networking lark," he went on, "and had the bullishness to believe that with no assistance, and in spite of the Gordian-knot-like cable nest, since this was a Layer 2 Switch with identical VLAN config to its replacement, I should be able to replace it along with a sister switch in another rack within the same maintenance window."
[5]
"Mostly because I didn't want to run the hassle of agreeing multiple windows with the client (and their clients)."
Colin was careful. He labelled all the cables. Undid all the bolts. And ever so gently, he began sliding the old switch out of the rack.
"I felt some resistance and pulled gently since the RJ45 jacks on Cat 5 cables can take some strain without damage, and all the cables were out anyway..."
[6]
Out of the switch, perhaps. Alas, by the time Colin realised what was causing the resistance, the second power lead had popped out of the server that happened to be running BGP on an ancient version of Linux for the customer's entire network. And, of course, there was no backup server waiting to helpfully step in.
Maintenance windows can be used to cover many sins, and Colin was burning through his at a terrifying rate. Yes, the Penguin Gods smiled and the Linux box restarted without issue, but there was work to do and time was marching on.
Tired, Colin came to the final switch. It was at the top of a full height rack. Of course it was. The Data Centre helpdesk team had not shared their step ladder due to "health and safety", necessitating some precarious balancing but at least the cables were a bit more accessible. They needed to be - Colin hadn't been given the key to get into the rear of the rack.
"I swapped the switch and attached the power, fibre uplink, and customer Cat 5 cables one-by-one," said Colin, "Since I had been meticulously checking for traffic on each customer port as I connected them, I noticed that one cable was missing."
He hunted and hunted. But cable there was none. "I think it was somewhere around the side but falling towards the back under its own weight," he told us.
Midnight came and went and Colin was exhausted. There was a real chance that he might make things oh-so-much worse with tired hands. The maintenance window was also closing, "so I figured that the customer would understand."
He messaged a senior colleague to flag up the port missing its cable. Someone in the morning would have to deal with it. It was, after all, just one cable. How bad could it be?
[7]To err is human. To really tmux things up requires an engineer
[8]It's the day before the grand opening but we need a firmware update. It'll be fine
[9]How to destroy expensive test kit: What does that button do?
[10]IT god exposed as false idol by quirks of Java – until he laid his hands on the server
"My colleague impressed upon me the following day that the only missing cable belonged to the client's biggest customer, and he did not see my message until after a while had gone by since the customer's business hours began and they were fully affected by the outage."
Colin was allowed to forget his mistake. It was used, he said, "as a prime example of why you should never go 'big bang' with a swap-out like this, not to work onsite alone, and to always declare a much longer window than you think you need."
Lessons learned after event tend to be the toughest ones. Ever found yourself fighting through a forest of cables and finding a customer less than understanding and forgiving of an unforced error? Confess all with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/Who,%20Me?/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Ygo2WPzVICk-Y7M9RjibAwAAAJY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ygo2WPzVICk-Y7M9RjibAwAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ygo2WPzVICk-Y7M9RjibAwAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ygo2WPzVICk-Y7M9RjibAwAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ygo2WPzVICk-Y7M9RjibAwAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/02/07/who_me/
[8] https://www.theregister.com/2021/12/27/who_me/
[9] https://www.theregister.com/2021/12/06/who_me/
[10] https://www.theregister.com/2021/11/01/who_me/
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Re: "Lessons learned after event tend to be the toughest ones"
Traditionally expressed as:-
"Experience is a hard teacher because she gives the test first, the lesson afterwards."
When you do have an expensive mistake like this, you ensure that the maximum amount of learning is done from it by other people, as learning from somebody else's mistakes is a lot cheaper than letting you commit them yourself. An entire new generation of people may be helped to avoid certain mistakes via this article.
Alas, the young engineer here didn't hear "Always multiply your repair estimates by a factor of four." else he'd have had a maintenance window plenty large enough. ;)
Maintenence window, gosh how quaint
Everything's got to be 24/7 now. Even on ancient systems which were designed with a maintenance window in mind and to be offline while overnight batch processes are run, you find sales have sold it as some shiny new always-available thing, so now you've got to persuade it do something to do it was never designed to do.
Re: Maintenence window, gosh how quaint
Hmmmm. "Sales sold it as ... ", and I think that at certain times, the server should mysteriously go offline.
Not to punish the customer, but to alert management to the frailty of the setup. Network errors are normally the least detectable, ahem, traceable to an individual, as they can be done with a single physical cable. YMMV, some setups are more secure than others. but we all know the weak spots.
You might find then that cash will flow for proper redundancy. If not of hardware, then at least people providing TLC.
Re: Maintenence window, gosh how quaint
By proper redundancy do you mean the systems or the staff?
I was expecting more sparks from the title :(
I thought the loosened power lead was going to light up the networking room for sure, looks like Colin got let off easy.
Honestly for getting as much done on his own without proper equipment as he managed, surely conflagrations are in order.
I was expecting our hero running across The Boss in flagrante delicto in the dark corner behind the servers ... It happens. It's soul scarring. Mind bleach doesn't exist. Whisky helps.
"Whisky helps."
It usually does. Can you remember what the problem was?
Problem?
Horizontal appraisal
I heard from a customer who went into the machine room to find the (female) manager giving a "horizontal appraisal" to a member of staff as part of the year end assessment.
BTW The engineers went in to the machine room in pairs.... for safety reasons.
There are a lot of people who should never be let near a RJ45 patch panel.
I have seen some bad ones the worse was a cabinet at a company my employee at the time had bought. The cabinet sat in a corridor and the patch leads extended in a pile of spaghetti which extended almost a metre in front of the cabinet. I upset the network person on site with my reaction.
I said get a maintenance window and unpick each cable label them and it back beater.
Mind you I have done the same pulling a cable and finding someone has wound something else round it or even worse used spiral wrap on a few cables.
But did he learn the biggest lesson of all?
In my opinion, learning to pick locks it's a "must learn" skill for any well-rounded sysadmin, network engineer, consultant, or whathaveyou.
Re: But did he learn the biggest lesson of all?
You mean it's a key skill?
Had he never seen Star Trek?
"...and to always declare a much longer window than you think you need."
A factor of four, according to the [1]Scotty Principle
[1] https://tinyhydra.com/the-scotty-principle/
Re: Had he never seen Star Trek?
Hofstadter's law. It always takes longer than you think even when you allow for Hofstadter's law.
Re: Had he never seen Star Trek?
I often use "multiply by two and increase the units by one":
5 minutes -> 10 hours
1 hour -> 2 days
1 day -> 2 weeks
1 week -> 2 months
etc.
Re: Had he never seen Star Trek?
Aaah, Laddie you didn't say how long it would really take?!!
Re: "as a prime example of why you should never go 'big bang' with a swap-out like this, not to work onsite alone, and to always declare a much longer window than you think you need."
Whenever you're calculating a time estimate you always - ALWAYS - put a healthy extra for unforeseen events. We call it the fear factor (that you use to multiply your real estimate by)
It's not fear. It's pragmatic.
I'm actually quite open about this these days. I go with "it might be done in x, but allowing for unforeseen issues, we should allow y"
It leaves the customer with some hope that things will be back up somewhere between the two but leaves plenty of wiggle room when things go wrong.
red alerts
Good preparation, but a number of red alerts were ignored, here.
"I felt some resistance and pulled gently since the RJ45 jacks on Cat 5 cables can take some strain without damage, and all the cables were out anyway..."
Red alert one: if there is resistance and you can't explain it right now, stop, and only restart until you're confident it has been dealt with.
"The Data Centre helpdesk team had not shared their step ladder due to "health and safety""
Red alert 2: if you don't have every tool you'd potentially need, refuse the operation.
"Colin hadn't been given the key to get into the rear of the rack."
Red alert 3: like #2
Transformation is about sorting out years of mess, so you'll always step onto some crazy temporary "fix" from a long time gone permie or contractor.
It's really like walking on a mine field. Every bizarre thing counts.
Talking about cables
Off topic...ish
A shipyard in Scotland is trying to build two ferries. They are over budget and late. The latest problem is the data cables are not long enough.
The cables were long enough when they were ordered a couple of years ago, but someone decided to move the displays and instruments, so the cables are no longer long enough. Swapping the cables around is not an option, so even more will have to be ordered.
The obvious answer would be to move the displays back again - but that's another can of worms.
"Lessons learned after event tend to be the toughest ones"
Indeed. You can't go back and sort things out anymore so, more often than not, you're looking at one of your failures. Or, if you're really, really lucky, one of your closest brushes with disaster.