Undebug my heart: Using Cisco's IOS to take down capitalism – accidentally
- Reference: 1627893014
- News link: https://www.theregister.co.uk/2021/08/02/who_me/
- Source link:
Our story takes us back a few decades and concerns an adventurous time in network support.
"Mort" – for that is not his name – was working for a well-known stock exchange, the network of which was running on Cisco gear. The cards, he recalled, cost $50k each and the whole shebang could probably be replaced by a single unit nowadays. But back then this was top-end stuff – nothing but the best would do for the nation's traders.
[2]
There was, however, a problem. Packets were being lost, resulting in delayed or lost trades. "When I think of traits that traders possess in abundance, tolerance is not one of them," Mort told us. "Neither is patience, so as you can imagine the pressure on the network team to resolve the issue was intense."
[3]
[4]
Those Ferraris weren't going to service themselves, and one lost packet could mean a world of difference in the quality of Champagne purchased at bonus time.
The support team struggled in vain to recreate the problem in the lab, and with pressure mounting, Mort decided the only way to get to the bottom of matters was to plug into the network and diagnose the problem live.
[5]
"Anyone who's ever worked on Cisco IOS (Internetwork Operating System, they beat Apple to the acronym by well over a decade) knows that debugging is a minefield as some commands can cause serious impact to the system," he said.
It's true. A glance at [6]an example of Cisco's documentation shows it festooned with warnings designed to deter all but the most determined and (ideally) competent of users. Debug output spewed over the console could make typing a command difficult, and the wrong debug command could easily take down a router.
That said, it was also pretty straightforward. The syntax was debug followed by whatever was needed. The output could then be captured, and the command backed out by prefixing it with un . "More often," explained Morty, "you'd just use ' [7]undebug all ' to turn it all off."
[8]Somebody is destined for somewhere hot, and definitely not Coventry
[9]How to keep your enterprise up to date by deploying the very latest malware
[10]Ah, I see you found my PowerShell script called 'SiteReview' – that does not mean what you think it means
[11]One good deed leads to a storm in an Exchange Server
The opposite command was the most dangerous of all: "You never, ever ran debug all unless you wanted to demonstrate how quickly you could take out your network."
Cisco had yet to add any "Are you sure?"-type prompts, doubtless assuming the operators of its hardware knew what they were doing. Instead, "if you typed it in it would salute, shout 'sir yes sir!' and then obediently jump off a cliff, similar to 'rm -rf /' on a *nix box."
[12]
You can probably see where this is going.
Our hero was careful – very careful – and had taken great pains in his planning. However, after a very limited set of packet captures, he was having problems getting the data and grew concerned that he might adversely affect the network. So he issued undebug all .
All hell broke loose. Text whizzed across the terminal screen faster than he could read and he realised that something had gone terribly, terribly wrong. Panicked, he tried undebug all once again. However, since he had clearly accidentally fired off every possible debug command at once, the terminal process was no longer bothering with such fripperies as input from the keyboard.
The un Mort is positive he typed clearly had not made it to the Cisco hardware (perhaps some transient unresponsiveness). But the rest of the command… oh yes.
With the network rapidly overwhelmed with the results of his actions, and the terminal ignoring his pleas to stop, Mort took the only avenue left to him and pulled the plugs.
All around our hero could be heard the sound of blades being unsheathed as the representatives of several large investment banks demanded his head on a platter.
Dispirited, Mort started cleaning his desk, expecting an escort to the kerb and the brown cardboard box of career limitation.
However, it is here that our story takes a turn. His boss was that rarest of managers – a decent human being. Rather than fling Mort under the bus, he stood between him and the pitchforks and took responsibility for the cock-up.
The result? A multimillion-dollar investment in the network infrastructure.
"Now that's management!"
Ever done something very silly, only to have your boss take the bullets heading your way? Or were you that boss? Tell all with an email to [13]Who, Me? ®
Get our [14]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YQfCXDiGhmPLFCf@37QnFQAAAIU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQfCXDiGhmPLFCf@37QnFQAAAIU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YQfCXDiGhmPLFCf@37QnFQAAAIU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQfCXDiGhmPLFCf@37QnFQAAAIU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.cisco.com/c/en/us/support/docs/dial-access/integrated-services-digital-networks-isdn-channel-associated-signaling-cas/10374-debug.html#warn
[7] https://www.cisco.com/c/en/us/support/docs/dial-access/integrated-services-digital-networks-isdn-channel-associated-signaling-cas/10374-debug.html#stopdebug
[8] https://www.theregister.com/2021/07/26/who_me/
[9] https://www.theregister.com/2021/07/19/who_me/
[10] https://www.theregister.com/2021/07/12/who_me/
[11] https://www.theregister.com/2021/07/05/who_me/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YQfCXDiGhmPLFCf@37QnFQAAAIU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] mailto:whome@theregister.com
[14] https://whitepapers.theregister.com/
Re: "he had clearly accidentally fired off every possible debug command at once"
You've never had the fun of tinkering with IOS have you?
It wouldn't surprise me at all if it didn't bother listening to the commands lovingly mashed into either the SSH command or if they're more locked down via a RS-232 to ethernet adapter and minicom software. Janky does not come close.
It's not quite as rage inducing as handling HP's iLO cli but close.
Re: "he had clearly accidentally fired off every possible debug command at once"
The CLI command buffer is normally quite good at accepting input from 'copy'n'paste' but it does occasionally get side-tracked executing a command and character get dropped.
Generating a crypto key can take 30s on some of the older gear and that could be 30s of characters piling up in the buffer waiting to be processed, and Sod's Law says your mega important command is the one that gets mangled.
Re: "he had clearly accidentally fired off every possible debug command at once"
Cisco CLI is not immune to losing characters from the input if the system is busy and/or the buffer fills up.
What jumped at me in the story is that with such careful planning of what to do, why on earth not do undebug for the exact debug command you know you have executed earlier just to be safe.
Anyone with any experience in Cisco IOS, especially on a more heavily loaded device, would be acutely aware of its tendency to occasionally miss characters from input as it quite understandably prioritises shuffling packets to what is going on in the CLI.
So he issued undebug all.
All hell broke loose.
So what you're saying is that command was terminal...
Cisco
At a certain large UK ISP, the
And a joke from back then: What's the difference between Cisco and Huawei? Well, they both design routers, but Huawei actually make them as well.
Yes, I've had a boss like that
But first, a boss who definitely wasn't. I was working for a well-known company in one of their manufacturing research teams. My boss decided we needed to fix a process problem with a few simple changes to the equipment; I argued against it as I saw it would cause new problems - but he was the boss. We ran it and it wrecked the equipment. He called me into his office and tried to tear off a strip - I stood my ground and told him, in no uncertain terms, how my report on the trial would be worded (and copied to his boss - who had initially hired me for another project a couple years earlier, and asked me to move with him to the new one). I should add that our facility was in a security area and you needed clearance to enter; my wife also worked for the company and came by one afternoon to say she would be working late. My boss called her into his office to demand what right she had to be in a secure area - she (not so quietly) pointed out that she had a higher security clearance than he did!
Anyway, I soon moved from there to a new company and a boss who was quite the opposite. He knew his technical skills were limited but knew the company politics (and was on first-name terms with the bosses from head office); he hired people with the required skills and let us get on with our jobs. If anyone decided we'd crossed a line into their territory (as we frequently had to do in order to fulfil our responsibilities, he would step up to take any and all the flak (and usually fire a bigger salvo back). Afterwards, he might call us into his office for a private chat and ask if we were sure we had done the right thing. His overriding policy was that whatever his staff did was done in his name. If we had made a mistake, it was his and he would defend us. It generated loyalty and, on the occasion when one of us thought we might have got something wrong, we would tell him straight away. It also meant that, when he was tackled on something he hadn't been told about, he was confident he was right to defend us to the hilt. I was sorry to leave that job, but the offer to double my income from one of our big customers was too good to turn down - and if I had, my divorce would probably have come about quite a lot sooner!
I remember cisco debug commands being a minefield...
Best thing you could do before running them is issue the commands to ensure it doesn't dump to console (and to logs only), but even then some commands would cause spikes in CPU usage and make the CLI sluggish.
"debug spanning-tree all" is another one you don't want logging to console in a switch that's part of a live/prod environment. Every STP broadcast, event, topology change, uplink change or error thrown onto the screen, and getting it to turn off once it's running is almost as bad...
Also a minefield - making sure your colleagues are aware of what debug output looks like and what it means. A former junior associate of mine was running DHCP debug on a pair of campus distribution switches (troubleshooting an IP address allocation issue), another engineer saw it and assumed it was a problem - he responded by rebooting one of them. Thankfully quicker hands managed to stop him from rebooting the other at the same time (which would have taken out the entire site whilst the switches came back up...)
Love those managers
My manager has our back... And back long ago, during a maintenance window, upgrades and patches went so well, I reminded my then manager, also a lovable manager that there was another niggly bug that we wanted to fix, he fired up a utility to fix bug x, and due to a poorly worded dialog box, he proceeded to take the RAID system down, wouldn't come up again, and quickly found out the backups were not up to the task due to the best effort job done with the woefully lacking hardware. In a month we had shiny new Dell PEs. Nobody was shown the door, but I did get some OT out of the exercise.
Hmm
Two network engineers sitting in an office, one (not me) types in a debug command that was missing a parameter so it started overloading the console.
I glanced over at him as things started slowing down - his glanced towards the door a quick nod and we both got up with laptops and quietly walked to the comms room. Where we found a switch stuck in a debug loop
Rebooted switch and as it was spooling up management burst into the comms room
There was a panic when the network went slow followed by a bigger panic when both network engineers had vanished.
We both stayed working their for another 7 or so years - management put it down to human error…
"he had clearly accidentally fired off every possible debug command at once"
So he properly issued an "undebug all" command at the console, and said console decided that no, it was going to do a "debug all".
I don't get how that is possible. There must be some shoddy programming behind that thing.