You want a reboot? I'll give you a reboot! Happy now?
- Reference: 1618816511
- News link: https://www.theregister.co.uk/2021/04/19/who_me/
- Source link:
Our contributor, Regomised as "Sam", regaled us with a story from a mere five years ago when he was still a fresh-faced worker doing time as second-line support at the service desk of a large motorcycle broker.
[2]
He had been called into the office on a Sunday to deal with a problem with the systems. "My boss," he said, with a less than fresh-faced weariness, "was nagging me about something another colleague did (as she saw everything as my fault) and she wanted me to restart a training server of the main brokerage platform because the restore had failed."
In order for training to be performed, data had to be restored on a Friday night to get the rates loaded. The eager beavers could learn the ins and outs of the system during their tutorials.
[3]
Quality control, Soviet style: Here's another fine message you've gotten me into [4]READ MORE
The failed restore had, alas, rendered the training rig borked. Sam had a terminal window connected to it so he could pick away at its internals. He also had one open and connected to the live system.
"My boss had rung me for the eighth time telling me to restart the bloody server," he told us, "even though it was still broken and explaining it to her was pointless."
FINE! He thought. I'll reboot the training server, not that it will make the blindest bit of difference!
Doubtless harder than necessary, he tapped in the required command and struck the enter key.
Almost immediately the alarms began to go off. An alert appeared saying that the brokerage platform had gone down. Which was fine – it just had. The training version, that is.
It then occurred to Sam, as realisation dawned with creeping horror, that the training system wasn't actually monitored. He'd mixed up the windows and managed to reboot the live brokerage system. Worse, he'd managed to reboot it during the Motor Insurance Database (MID) upload process, "which uploads our entire book of live policies to the MID database after deleting every entry."
"It's the database the police use," he added, matter-of-factly.
It got worse. "The reboot meant every client of ours was uninsured for the eight minutes it took to reboot the server."
What had been a simple task to deal with a borked training server (broken by somebody else) had now escalated into a full-on fiasco (entirely of Sam's own doing).
Those eight minutes were likely the longest of his life.
As for consequences, Sam eventually received a telephone call from the MD of the company demanding an explanation. He trudged to the office, fully expecting to be hauled over the coals. A P45 might even have been in the offing. The MD, however, had other ideas.
"He just handed me a pair of glasses instead."
[5]
Ever tried to fix one thing, but ended up making something else much, much worse? We've all done the two-window testing/production fandango, but was your boss as understanding when the inevitable happened? Share your moment with an email to [6]Who, Me? . ®
Get our [7]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YH1UuKsH9YSBYwo9PXKgiQAAANc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YH1UuKsH9YSBYwo9PXKgiQAAANc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2021/04/12/who_me/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/servers&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YH1UuKsH9YSBYwo9PXKgiQAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] mailto:whome@theregister.com
[7] https://whitepapers.theregister.com/
Re: Background
I sent a corporal off to the server room to restart one of the boxes (it was taken out into the field on exercises but otherwise just sat their and maintained).
On his way there, a WO2 pulled him aside and bent his ear about some stuff that the WO2 felt was outstanding.
So, said corporal went into the server room and went onto the production system (which was supporting active operations), checked the tasks the WO2 had insisted on, and then issued the reboot command...
When shit hit the fan, my position was that it was an easy mistake to make, and that the jobs the WO2 had tasked him with
a) weren't actually outstanding
b) weren't that important
I also pointed out that, as a civvy, I'd have told the WO2 his tasks needed to wait, but a lowly corporal doesn't really get that luxury so the WO should perhaps have spoken to me instead.
Corporal got away with a bit of egg on his face, but no other real consequences.
Re: Background
It's a fairly sure-fire way to tell whether an organisation has suffered an episode of server-identity crises at some point.
If they have, they'll have the nasty colour scheme in place.
Yes, systems I control do have that.
Re: Background
It works in user territory as well. Several systems, all running similar backgrounds but against different databases. They were set up with different background images.
Longest wait
I PM'd our project to change our call recording system. 500+ users.
Was updating some permissions and accidentally deleted the source permissions file by uploading a completely blank table.
Stage 1: Panic.
Stage 2: Panic some more.
Stage 3: Try and find backup. No joy.
Stage 4: Ring supplier, who managed a remote restore.
Stage 5: Dent wallet badly at project end buying copious amounts of beer for tech who not only restored but forgot to document said restore.
I do like working with Business Systems.... Lovely company and oh so discreet!
Should have added, this was a parallel run to old system before switching over.
It could be worse
When it's your home setup you just trashed. Especially when you'd just been thinking it really was time you sorted out those stale backups... and the other half has just asked why she can't get onto facebook.
Re: It could be worse
Can’t get onto Facebook? Sounds to me like you’d done her a favour. A Facebook outage is a significant security upgrade!
Re: It could be worse
A Facebook outage can be grounds for divorce north of Watford.
Re: It could be worse
I trash my home system all the time. I fear nothing now. At least, not since I learned the true value of the word "backup".
It happened one day, way back when, and I was making a copy of my company mail file to my home system. It's just that I had forgotten that the file name was the same as my home mail. When faced with the "Do you want to replace" popup, I clicked on Yes, of course.
Then it dawned on me. I had just erased my home mail, with all my contacts and years of correspondence - including a response from Ian McKellen himself concerning a mail I had sent to him before LotR came out. I had emailed him to tell him how awesome I thought this was going to be, and he had nicely responded to me saying he was indeed thrilled about being in the film. Now that conversation is gone for all time.
That's the day the word "backup" became important to me.
A long, long, long, long time ago, when we were just beginning to get PC based Linux and Windows NT based servers, we still had some hefty old iron based on ancient DEC platforms. Sure, plenty of Vaxen (because everyone did), but also old PDPs. As a noob manager, I had responsibility for a rack of mixed kit in the Development Data Centre. The rack had a PDP-11 in it running - oh bloody hell. The mists of time obscure what it was running - I can’t remember. Suffice it to say, that I was one of the few developers there who understood the software, and who could boot the bloody thing up (although there was a better that 50/50 chance that any attempt at booting would result in failure - so it was a bit of a shenanigan and a bit of a pain in the arse).
One of the developers who reported to me was a real bright spark (genuinely), but also freshly minted from University. He was developing a genuinely cutting edge platform on a Linux foundation, and it was in its earliest earliest days with some gnarly bugs in it. One gnarly bug required the system to be powered off and rebooted. I mean. No problem there, surely? A PC and a PDP don’t even look alike.
I was upstairs, in the offices, happily beavering away on my VT when… loss of connection. The PDP had disappeared off the face of the earth, taking the code I was writing with it (not too much though - I saved regularly). I ran downstairs to discover noob-report looking sheepish. For some reason which I still can’t fathom, but might have something to do with it being very late and well past beer o’clock (his choice to stay and finish some work, not an instruction from me), he’d decided to power off the entire rack rather than just the server that we was working on. His server was up and running again, but he didn’t know how to boot the PDP.
He watched with increasing concern as I toggled in the bootstrap on the switch panel on the front, the PDP began to boot… and then failed. Again, and again, and again. Capricous bloody computer. With each failure, I got more and more angry to the point that, noob-report admitted to me over a beer in the pub after the PDP eventually started, he thought he was in danger of being subject to a walloping administered by myself. Which would have been a remarkable turn of events since I’ve never hit anyone in my life.
Hands up, intentional mixup
Periodically, as I complain about a lack of backup, monitoring, process etc, I've seen techies deliberately shutdown/reboot important servers just to get management to pay attention to the need for failover etc.
Techies shouldn't need to do it, but sadly even now we have to.....
There was always something *very* satisfying about deliberately seeing it done though..... it's ability to focus managerial types on the important things is magnificent!!
Re: Hands up, intentional mixup
Unleash the Chaos Monkey!
https://netflix.github.io/chaosmonkey/
Re: Hands up, intentional mixup
I've always thought a good response to bean counters wanting to put a figure on the value of some system would be "Shall we switch it off and find out?"
Re: Hands up, intentional mixup
>it's ability to focus managerial types on the important things is magnificent!!
Certain managerial types can become too focused when deadlines, bonuses etc are involved...
Around 15 years ago we were performing system upgrades in a financial organization as mandated by the internal audit. A part of this upgrade was a NetApp storage appliance with around 10-20TB of data fronted by a Windows server and used by literally every department of the said organization. It was not business critical on paper but the audit revealed that a lot of business critical processes were in fact using it.
The amount of files was so large and the NTFS structure so convoluted that in the attempts to back up the data standard copying was simply failing and even more low-level Robocopy reported it would take weeks if not months to complete. Needless to say, regular backups were not an option even before.
Of course, when the cold, sleeping data was disturbed, it did not take it well and one disk promptly failed leaving the system in a degraded state. Taking a full copy of data from the degraded storage system would then literally take months if ever. A sane approach would be to fix it first before moving any further but the IT infrastructure boss was pressing hard because he had his deadline and his boss had his deadline etc. Being technically literate, he even went as far as to ordering the team to override NetApp's foolproof safety protections via the serial console and proceed with the upgrade.
Somewhere around this moment it became a big pressing question as to whom the blame will be put on should the charge of the light brigade somehow fail. NetApp was finally contacted for the advice and their engineer came up with the following response, 'I am not sure that the customer fully understands what they are doing'. Needless to say, all engineering efforts switched from completing the upgrade successfully to not being around the fallout zone when the worst happens. Fortunately, the management sensed that their chairs might become shaky too and NetApp was paid to take risks (as in, if the vendor failed, no one could have done it), perform the fix and upgrade properly and with enough time, which they did.
A couple of decades ago
I wanted to check how vulnerable one of the company's public facing applications was to SQL injection and issued a drop table command embedded in user input on the "test system". I must have been distracted or otherwise brain not engaged as protests all around the office erupted as I trashed the live system. Oops. Thankfully I just got a bollocking, not a P45.
Investment bank production settlements power off
Not me fortunately and many years ago, late 90's
Tech workinng on some HP-UX UAT system, for whatever reason, he wanted to reboot or power-off/on some multi-cabinet HP-UX server.
He had major visual impairment, he had these huge monitors with very large font sizes and he read from a distance of about 4 inches.
He decided to go up a floor to the computer room and past the ops to the HP-UX server and flipped the power button, except..... he powered off the production settlements server during the business day.
Don't quite remember how long it was down for before it was got up running again and a lot of shouting happened.
Access rights for tech people were rescinded soon after, tickets had to be raised with th Ops people to gain access and then they let you in,
Saying that, it was a big room, easy to get lost in and every corner looked the same, always fun finding BT people and others who were lost in there and pleading for help on finding the exit, internal people too - there were 1000 full height racks in there.
The person who deserved the real bollocking was the mico-managing manager. Very likely, however, she didn't even recognise it as a learning opportunity, at least, not for her.
Background
And that's why some of our more important systems had a garish pink desktop background when logged in as admin.
And the Linux server had a specially crafted command prompt, nasty colours and all.
Yes. There is a reason for this. Yes same as in the article. Yes, the boss was not pleased but understanding (just don't do this again, were his words).