To err is human. To really tmux things up requires an engineer
- Reference: 1644222731
- News link: https://www.theregister.co.uk/2022/02/07/who_me/
- Source link:
The latest confession comes from an exotic South American city and one of the country's many ISPs. Our reader, helpfully Regomised as "Paolo", was an engineer working for the company and, as is the case for so many of us in the IT world, a wearer of multiple hats.
Not to worry though, thanks to tmux (a terminal multiplexer) he had everything at his fingertips. Consoles for different servers, BGP and ERP settings, FTP and so on. Over his 10 or so tmux sessions he had pretty much every acronym nailed down and was god of all he surveyed.
[2]
The event in question took place on a Friday evening, which is when everyone does all their best work. The office was closing up – engineers were heading off into the night to do what engineers do with their time off.
[3]
[4]
In between his tmux juggling, Paolo kept an eye on the clock as the big hand neared the 12 and the little hand settled on the 5.
Finally. Home time.
[5]
Switching to the tmux pane of his local machine, Paolo issued the shutdown command. The notebook screen was slammed shut. Stuff was gathered. The weekend was on the way.
Except... except... there had been a LOT of tmux panes open. Had he gone to right one? The answer came all too soon, and all too loudly as the boss's despairing shriek echoed through the building: "WHO SHUT DOWN THE BGP SERVER?"
"It was not my own machine," recalled Paolo, "but the firewall and BGP server."
[6]
One facepalm later, and an admission of his mistake, and Paolo was in the network room, powering up the server he'd accidentally turned off.
"Except this wasn't a normal firewall and BGP server," he told us, "it was a custom one, integrated with the ERP. Upon booting, it would parse the ERP database, fetch and create the VLANs, firewall rules, and so on."
Seems reasonable enough. However, Paolo went on: "But a few engineers, to speed up folks working on the street in cabling and new clients, would issue commands direct into the firewall, later they would log into ERP and register the new client... thus, no persistence."
[7]When forgetting to set a password for root is the least of your woes
[8]Firewalls? Pfft – it's no match for my mighty spares-bin PC
[9]How to stop a content filter becoming a career-shortening network component
[10]Undebug my heart: Using Cisco's IOS to take down capitalism – accidentally
We'll pull the covers over these shoddy practices, and we're sure no Register would ever stoop to such bodgery but, for Paolo, the damage was done.
Rather than a carousing through the local public houses and sampling adult beverages, Paolo's (and his team's) Friday night was instead spent painstakingly working out which client hadn't been saved in the ERP. The task took hours .
"I never issued a poweroff in the wrong tmux pane anymore."
What lesson did you learn the hard way? Or were you at the sharp end of another's educational moment? Tell us with an email to [11]Who, Me? ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/Who,%20Me?/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YgD73E22qlPjbz9SLpvprwAAAIs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YgD73E22qlPjbz9SLpvprwAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YgD73E22qlPjbz9SLpvprwAAAIs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YgD73E22qlPjbz9SLpvprwAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YgD73E22qlPjbz9SLpvprwAAAIs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2022/01/31/who_me/
[8] https://www.theregister.com/2021/10/04/who_me/
[9] https://www.theregister.com/2021/08/31/who_me/
[10] https://www.theregister.com/2021/08/02/who_me/
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Re: A good thing
Yeah, the fun of "oh, I'll put that into the correct production system / the database / the version control / whatever LATER". It will bite you. Hard. (yes. yes it did bite me. hard).
I hope Paolo did spring for a couple of "adult beverages" after the team had stayed longer. In German glider clubs you have to buy a case of beer (and some non-alcoholic stuff) for several occasions, like birthdays, passing certain milestones (forst solo flight, licenses etc.), and there's also the acronym "WGU" = wegen groben Unfugs (due to fsking up - badly and possibly in a stupid way). This one would qualify.
(same thing in certain teams I worked in - you f' up, you stay, you buy beer for the rest who stays and cleans up your mess, I think it's a good policy).
Re: A good thing
For the love of god and sanity. Save in what you type. even better is logging the terminal session. Email it to yourself and one other.
Bin there, done that.
It's right up with "just take a backup of the running config before I touch something, and keep it in several safe plces". Murphy knows when there is only one backup and screws with that. And do the same when finished because I know someone is going to "improve" things a day or two later.
First day in my first job in a bank my new boss demonstrated how easy it was to remotely manage multiple systems from his desktop. He proceeded to shut down the main NIS server for the trade floor systems instead of the dev box he had intended, queue much screaming as hundreds of clients very slowly switched over to the secondary
A nice welcome to the job. Always good to have a boss on the defensive.
And welcome to el Reg.
You had people patiently queuing up to scream? Typical British. :)
yay, mollyguard
saved me a few times, always when I'd forgotten it was even there
"Paolo" really tmuxed up there...
Things were almost terminal...
Moar visibility
This is why I only use tmux to make a session survive disconnection, never to run several sessions in one terminal. Not enough visible cues about which session I'm actually looking at.
I use many terminal windows, so that I know that one there is connected to this service. Over there is doing something else. Each of them may be running tmux if I think the session is important enough to protect.
I might still do something that dozy, but it's more difficult.
Re: Moar visibility
I use multiple sessions but never to different servers. If I want a different server connection, I'll put that in its own terminal window (well, OSX Terminal tab ).
They say that the reason experienced people are paid so well is because they've already got most of their screwups out of the way on a previous employer.
Re: Moar visibility
Yes, lots of terminals, each with different colour schemes so as to get a visual cue as to which system I'm on. I do still use GNU Screen a lot though (mainly because I can't be arsed learning tmux after all these years).
Re: Moar visibility
I have coloured bash prompts for all of my systems (well... classes of systems - my laptop is normal linux-y, some servers are nasty garish cyan-like, some are more red). I have not yet f'd up (after that change).
I have this problem not only with tmux screens, but rather with the (sometimes stupid) number of open terminal windows...
I shut down which node of the cluster?
I could have been handed my coat that day! But my boss at the time must have peered into his crystal ball, and seen the good things I would do, and kept management at bay.
thats a tricky one
Hard to imagine how Paulo would actually recover and implement those undocumented / unsaved / on the fly thingies .
Re: thats a tricky one
Go around all the service centre desks and look for scraps of paper with connection info on them, then compare them to the ERP system. All those missing must be added to ERP and BGP.
Here's Johnny...
I was doing some security testing for a client, was back at the turn of the century. They wanted their eCommerce system checked. I did some looking at the source code and marked all the locations, where they hadn't sanitized input and handed in my report and started testing the weak points.
The dev team and management didn't want to know. Running a SQL injection to display a list of users and their credit card numbers wasn't convincing enough... So I went nuclear on them, my next test included a SQL injection of "DROP DATABASE;--" Bye, bye dev environment...
The handy part about dropping the database is that you get immediate feedback from all the devs and ops currently working on the system, about whether the test worked or not...
Re: Here's Johnny...
Ah... little Bobby Tables. :)
Immediate feedback
indeed.
Did the same thing during a penetration test with SNMP management of a mainframe network interface.
Definitely wiped the smug grin off the local mainframe God who had claimed his dinosaur was "not hackable".
If it is easy to colour a screen - just do it!
I sat in the tail end of a post mortem in India which was trying to find out why a production system had been shut down.
The site "expert" (arrogant, would not be told) was defending his corner. An external expert said "6 months ago we did a review of your systems and one of the top 3 recommendations was to enable the screen colouring; red text for production; Yellow for test. Why wasn't this done? It is a one line change"
The site expert tried to bluster his way out blaming someone else.
A humble (but skilled person) said "excuse me, we did create a change record, but it was never implemented....".
We had a comfort break, while management found out the status of the change record...... all the recommended improvements were waiting for the site expert to approve!.
We didn't see the expert again... but someone else, a team leader (rather than an expert) took over and within a week all of the recommendations were in progress, and calm and confidence spread throughout the team.
Re: If it is easy to colour a screen - just do it!
First thing I do on all production server VMs is change the text to red on the console or use a red desktop background for the GUI...
A good thing
It sounds like he did the company a favour by forcing everything to be documented/saved. Just a shame to happen at that time of the week really, but hopefully the clients were quiet at that time so disruption was minimised.