BOFH: The evil guide to upgrading switches
- Reference: 1649930405
- News link: https://www.theregister.co.uk/2022/04/14/bofh_2022_episode_7/
- Source link:
His final task will be a firmware update for the core router – which will, if we're lucky, stop a six-monthly out-of-memory issue.
He'll have already updated the firmware of our security camera system (and the resultant downstream updates of the attached cameras' firmware) and then moved on to updating the access control system – which is a little trickier, as the update causes every door in the building to unlock for around 15 seconds before re-securing.
[1]
In the old days – well, a few years back – we'd have change-controlled this activity. But luckily change-control was one of those management fads easily stifled with a relentless campaign of punctiliously observing the change control procedures for even the tiniest change.
[2]
[3]
Even the most stalwart advocate of change control would crumble under the weight of 48 separate "Network Switch Update" change control notifications when permitting an additional VLAN for every user port on a switch.
Multiply that by 28 switches and you've got an inbox full of rubbish obscuring the needle-in-the-haystack firmware update message.
[4]
But those days are long gone. Now we only have to change control the important changes – and "important" is a very subjective term. I mean, upgrading the firmware on the espresso machine? IMPORTANT. Flashing the BIOS on my gaming laptop? IMPORTANT. Core Switch firmware? Not even on our radar.
And speaking of not on our radar, I get a text message that the management VLAN and console port of the core switch are no longer with us …
Obviously there's no need to panic – the pre-boot self-checking of this particular vendor is renowned for being glacially slow and nitpickingly verbose. I don't expect that to come back up for another ten minutes.
[5]
Ten minutes later the PFY reports no activity on the console, even after checking the serial port settings in case the new firmware's changed the port characteristics – which vendors like to slip into firmware updates without telling anyone.
Ten more minutes later the PFY tells me he's in the server room, looking at a non-standard set of status LEDs. He sends me a photo of the LEDs – which don't represent any of the fault codes I've seen.
[6]BOFH : Putting the gross in gross insubordination
[7]BOFH : The Hypochondriac Boss and the non-random sample
[8]BOFH : Thermo-electric funeral
[9]BOFH : The Idiot-ware Project and the Meaningless Acronym
After another couple of minutes he sends me an email with a WARNING.TXT file in included in the firmware zip, pointing out that leap-frogging of major updates is no longer supported.
In situations like this there's always a choice.
CHOOSE YOUR OWN ADVENTURE!
A: Go into the ROM monitor, do some low-level deleting, then perform an excruciatingly slow serial upload of the old firmware (which will fail four times out of five, with buffering issues).
OR …
B: Remove any evidence of your work and come in at 8am to "find the switch has failed!"
"It'll be a hardware fault," I tell the Boss, choosing option B like a true professional. "Happens all the time."
"The switch is almost three years old, after all."
I don't tell him that some of our switches are so old they have dust from the Big Bang in them, choosing instead to imply that three years is pretty old for a switch.
"Do we have a spare?" the Boss asks.
"Oh no! I mean, that switch was pretty expensive," I respond.
"How expensive?"
CHOOSE YOUR OWN ADVENTURE!
A: Tell him the truth: that the switch concerned only cost a couple of grand and we can probably fix it with a firmware flash in a couple of hours.
OR …
B: There's this app-connected espresso machine with multiple bean hoppers which can grind a blend specific to your personal tastes …
"I think it was about seven grand," I lie, "Option-B"-ing again.
"SEVEN THOUSAND POUNDS!" the Boss gasps.
"It's the 10-gig connectivity," I add. "And we'll have to pay a premium to jump the queue to get it delivered today."
One of the immutable rules of IT budgeting is that a critical failure opens a fatter chequebook than a beautifully presented business case …
"Well, I suppose we'll have to do it," the Boss sighs, heading off up the food chain to break the bad news while I, meanwhile, get onto my coffee supplier with the good news.
Three hours and one large "switch" delivery later, I'm enjoying my first Brazilian/Ethiopian blend of the day as the PFY reloads the switch and we get back online again.
"Is this new?" the Boss asks, spotting our new coffee machine the moment he walks through the door.
"This?" I reply. "No, had it for ages. We just never got around to plumbing it in …"
The words are barely out of my mouth when the Boss's brow wrinkles at the information on his phone in response to his scan of the QR code on the side of the machine.
"This says it was commissioned today."
" Commissioned , yes, but we've had it for ages."
"It says it's the 2022 model."
CHOOSE YOUR OWN ADVENTURE!
A: Admit you have misappropriated a vast stack of cash for a coffee machine.
OR …
B: Admit nothing and cover your tracks.
"2022 model? No, that's the coffee beans we ordered for the machine this morning. They're over here, on the shelf by the window …"
[10]BOFH: The whole shebang
[11]The Compleat BOFH Archives 95-99
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YlhFMypB6KLuF4Zhmpr1YAAAANg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlhFMypB6KLuF4Zhmpr1YAAAANg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlhFMypB6KLuF4Zhmpr1YAAAANg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlhFMypB6KLuF4Zhmpr1YAAAANg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/bofh&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlhFMypB6KLuF4Zhmpr1YAAAANg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2022/03/25/bofh_2022_episode_6/
[7] https://www.theregister.com/2016/11/25/bofh_2016_episode_14/
[8] https://www.theregister.com/2016/04/22/bofh_2016_episode_6/
[9] https://www.theregister.com/2016/10/07/bofh_2016_episode_13/
[10] https://www.theregister.com/data_centre/bofh/
[11] http://www.bofharchive.com/
[12] https://whitepapers.theregister.com/
Re: rule 1
Always save those old asset tags from deprecated equipment. They are recycled, responsibly.
Re: rule 1
>Always save those old asset tags from deprecated equipment.
I have a box full of old laptop bottom cases with asset tags - we've never found the procedure for disposing of a corporate PC, so officially I have all these laptops
Re: rule 1
Even better, use an older casing, replace the QR code label with one of those old"Intel Inside" stickers and if the thing has evident "Gaming kit" lights, disable them.
Virtual Friday BOFH
Damn needed that.....
Re: Virtual Friday BOFH
I thought my calendar was off! I was about to reach for my phone when I finished reading ;-}. Will we get a bonus episode tomorrow??!!
"we can probably fix it with a firmware flash in a couple of hours."
This was the switch which was just borked by a firmware flash so are we looking at a previous Choose your own adventure?
A. Go without a new coffee machine.
B. Break into the server room at dead of night by fixing the security cameras and access control to administer a firmware update that will bork the switch recoverably.
Maybe they have a spare in storage?
Choose Your Own Adventure
I think I am older than that Big Bang dust on the routers.
I loved the Choose Your Own Adventure books by Steve Jackson and Ian Livingstone back in the dark mists of time.
Thanks for that Simon. 40-odd years gone just like that.
Re: Choose Your Own Adventure
Lone Wolf by Joe Dever... liked it when it came out. Got the first 3 or 4 books of the series.
And, yes, there's a whole subreddit devoted to the Lone Wolf adventure books.
Project Aon on Android is where it's at if you like to have a stab at defeating dark and evil forces.
Re: Choose Your Own Adventure
The Steve Jackson and Ian Livingstone ones were "fighting fantasy", "Choose Your Own Adventure" was actually its own brand of gamebook - without any stats to track, die-rollijg or combat mechanics. I didn't like them because they were "for little kids", unlike my own very grown-up copy of "DEATHTRAP DUNGEON".
Year and a half old switch without trouble. Recently firmware updated and a month later all PoE power drops off, though "switch" is still working for traffic that is not dependant on PoE, unlike the security cameras for example...
Soft reboot recovers PoE. New firmware bug, or random glitch? Makes me wonder how often such upgrades are worth the risk of new exciting bugs versus any genuine fixes they include.
Time for some of this =>
A layer-2 office switch sitting inside a campus LAN with RFC1918 management address, port security (if not 802.1x) and unused ports disabled? If it ain't broke, don't fix it.
(Routers, however, are a different beast. Stay on top of your vendor's CVE publications and upgrade regularly. And not to the bleeding edge version of you value sleep over the overtime compensation.)
Never trust point 0 release and wait for point 1 release.
You should update your switches as well because nearly all of the old ones can be subverted with random malformed packet data (specific to each bug in their packet forwarding engines...).
On the other hand, port security, 802.1x and no unused ports enabled are DEFINITELY a good start.
This is familiar... Switch dropped PoE randomly. Yours wasn't a Net**** by any chance?
Serial port settings ...
... in case the new firmware's changed the port characteristics – which vendors like to slip into firmware updates without telling anyone.
Here's looking at you, pfSense !
"They're over here, on the shelf by the window …"
I can sense a Database Normalisation Warning, and a problem with a windows install coming up
Brilliant episode, once more
Shelf by the window ...
... which happens to be open, affording a bird's eye view into the skip below ...
Re: Shelf by the window ...
Or carefully timed refuse collection…
Of course, the boss should have used the "emergency" to fill his own boots for essential IT upgrade. How about a coffee machine that also brews beer?
Re: Shelf by the window ...
... which happens to be open, affording a bird's eye view into the skip below ...
Or the boss's new car.
The old switch and bait.
So tempting
a BOFH episode on a Thursday? But that screws up the flow of the weekend, and just makes the universe seem a little sideways for a couple of days. So must not read until tomorrow. Oh, so tempting. But no, tomorrow! Tradition! But didn't the BOFH used to come out on Thursdays? Shut it, that's a lie! Tomorrow! Yes, yes, tomorrow.
Is it Friday yet?
Re: So tempting
Here on the correct side of the pond it's Good Friday tomorrow and a bank holiday, hence the early BOFH :-)
Re: So tempting
It's a bank holiday here in Jesusland as well. Although possibly it's the killing of a middle-eastern man that they approve of
I loved total control
Over the asset management database... What record? No, you must have been looking at similar unit...
rule 1
Make sure to remove any incriminating codes of any description to cover youre tracks
(rule 2 is to put an 'old' asset tag on the device insinuating that it has been about for ages, cost peanuts, and was sent foc as an afterthought by a vendor with some other kit 'to get it out of their inventory as they had been sitting on it for ages')