Dumpster diving to revive a crashing NetWare server? It was acceptable in the '90s
- Reference: 1587971709
- News link: https://www.theregister.co.uk/2020/04/27/who_me/
- Source link:
This time we return to the 1990s, and to a region of Australia that our protagonist, a person we'll call "Howard", described as "a three-horse town where everyone knew who's dating everyone else's sister".
Top marks for the use of the word "dating" there, Howard.
Howard was plugging through the end of his education and working part time at a local computer store. His job mainly entailed popping out to deal with customers bigger than those that might be merely toiling along with word processing on a Windows 3.11 box.
His confession starts at what he told us was "a decent-sized financial institution" having trouble with NetWare 3.11. The system in question had to log (for legal reasons) card transactions that had been sent to an AS/400 for batch processing. While IBM's metal kept chugging along, at some point between 2 and 6am the NetWare server would lock solid.
While not privy to what this logging app was doing, "given NetWare 3.11 was co-operative multitasking and the custom logging app was a NLM [NetWare Loadable Module] and ran in Ring0, it was most likely the culprit crashing the whole show... but it was out of scope."
NLMs turned up in NetWare 3.x and were the bane of many an administrator's life. The handy binaries could do all manner of things, such as implementing hardware drivers, but bad behaviour could take down a server or make it unresponsive. Things improved in version 4 and 5, but the smell of instability lingered.
Getting a pizza the action, AS/400 style [2]READ MORE
The NLM in question here was off the table so Howard had to come up with another solution.
"The usual procedure," he recalled, "was that one of the locals would arrive for work the next morning and they'd eyeball the server."
No flickering of the hard disk LED after a few minutes meant the reset button needed a prod and the night's transactions manually reconciled. Fortunately, the fact the crashes always occurred in the small hours meant that there weren't too many but...
"Management," he remembered,"would have an early morning meeting containing a minor blame-storm."
This gradually escalated, as the frequency of the incidents increased, into a full blown "rage-fest," according to Howard, which was "shoved through the phone line at us."
An immediate workaround was demanded for the reboots with a fix for the cause needed at some undefined point in the future.
"It already felt like we were circling the drain," sighed Howard.
However, there was this newfangled thing called "Linux" doing the rounds with some flaky NetWare Link Services Protocol (NLSP) support. It wasn't complete, but would allow a type of "Who's on the LAN" broadcast. Parse the response, and it would do for an "Are you alive?"...
"The penny dropped with a resounding clang."
At this point, Howard transformed from being merely a computer shop part-timer with a penguin fetish into a hero for all time.
A battered old 486 box was pulled from storage and had Linux installed. A bit of Perl was used to check if the NetWare server was still alive; if there was no response within a set period then a reset was needed.
And the reset?
"I found a broken CD-ROM mechanism along with its Mitsumi 8bit ISA card in a dustbin, and with a generous amount of stolen Blu Tack sourced from a few dozen or so cubicles, a broken pencil and some tape, this was all added to the front of the non-reading CD-ROM drive as a 'finger'."
You can guess what the next step was.
"When the server would not respond, the Perl script (running as root) would eject /dev/cdrom, send an email to double-check for dropped transaction log entries, and go to sleep for 20 minutes till the server came back up."
The final flourish was the positioning of the old 486 in front of the NetWare tower server. With a pair of Yellow Pages, and some copies of a long-defunct print publication for fine-tuning of height, the CD-ROM tray and its "finger" were perfectly positioned in front of the reset button.
Better still, Howard ensured there was not enough space for the CD-ROM tray to fully open, so having jabbed the reset button it would retract: "Ready to strike again."
"GENIUS!" he told us, modestly.
Despite a slap on the back and the promise of a free beer on Friday, Howard soon moved on to other things and thought no more about his hack until 1999 when a friend told him about "the most dodgy thing [he'd] *EVER* seen in this accounting and finance firm that had a crappy old 486 rebooting a server tower via a pencil on the front of the CD-ROM tray."
Howard's hack had spent half a decade toiling away.
"I laughed," he said, "and drove on quietly."
Can you top Blu Tack, a pencil and broken CD-ROM? Share your tales of blatant bodgery from days long gone (or all too recent). Discretion (and a dodgy pseudonym) assured via an email to [3]Who, Me? ®
Sponsored: [4]Legacy Modernization: Finding Your Way With Low-Code
[1] https://www.theregister.co.uk/Tag/who-me
[2] https://www.theregister.co.uk/2020/04/20/who_me/
[3] mailto:whome@theregister.co.uk
[4] https://go.theregister.co.uk/tl/1936/-8553/legacy-modernization-finding-your-way-with-low-code?td=wptl1936
Re: Not a Who Me?
Agreed. That's an On Call candidate! :-)
Genius ...
Excellent improvised engineering "solution".
They say there is a fine line between genius and insanity. I seem to spend most days either repositioning or trying to erase that line ...
Re: Genius ...
My trick to avoid losing the plot is not to start reading the book!
Re: Genius ...
I spend my days RiverDancing on the bloody thing while naked, twerking, & playing the bagpipes! =-D
*Inflates the bag, squeazes the blob betwixt my knees, & dances off twerking to a bagpipe rendition of "Waltzing Matilda"*
Re: Genius ...
Had you thought of starting a YouTube channel?
Re: Genius ...
It's already on My Skippy's List of Things I'm Not Allowed To Do.
*Comical pout*
*Wanders off dancing & twerking to "Tie me kangaroo down"*
Dear Reg...
...I think you will find that it is actually the 118th of January today.
Re: Dear Reg...
I am sure its April now, I think it may be 94th of April. I think its something like year 7 of the lock down, although it feel longer as my exwife was meant to move out 2 weeks after the lockdown, started and we are still rattling around the same flat.
A long time ago
I learned that if you bodge something with a bit of tape and string (or these days a Powershell script and task scheduler) you can expect it to still be there years later! These days I always put a plan together to 'tidy up' bodge work into something more long lasting once the crisis has past.
Re: A long time ago
Too true - if a bodge job seems to work, the attention of management moves on and a temporary hack held together with (literally in this case) blu tack soon becomes the new normal.
I'd never have been guilty of that .......
Re: A long time ago
The short version: There's nothing more permanent than a temporary fix.
Re: A long time ago
which is why I have taken to documenting even the most trivial of 'bodges' cos I know some poor sod, who isn't me, will have to sort it out in eleventy years time when things have gone all FUBAR as various unforseen "1 in a million" events congregate around said bodge..
no wait... that poor sod will probably be me... .
We have a dev server here which sometimes locks up on boot (locks while detecting drives), it seems to be a fault on the motherboard itself as the drives are fine.
We noted that when it failed the font panel was distinctive in that the hdd light was lit, network 1 lit and network 2 not lit.
so we added an arduino monitoring the lights so that if net1 & hdd were lit continuously for 30 seconds and net2 wasn't the arduino would hit the reset on the server.
It's been like that for 5 years so far.
why dont you just ping it?
The wacky hardware bodge ala the story is for the pressing reset , surely the monitoring csan be done more conventrionally
You still need to physically hit the reset switch.
There are many times where pinging isnt enough, a device might ping when in fact applications it host have stopped responding.
LOM
That is why you have proper servers with LOM, on a different network.. they are essentially the advanced pencil...
Famous last words
as long as nobody complains, I don't mind...
hot-wiring with office supplies
We had an critical email server running on a Dell box, unfortunately this model's power button was so poorly designed it wasn't up to the muscular fingers of the support team, or - eventually - the biro you needed to depress the damn thing.
After it completely gave up the ghost the only way to power down. the server was pull the power out, and the only way to power the thing back up again was to short out the connections to the power button using a specially shaped paper-clip.
Re: hot-wiring with office supplies
Then there was the Sun E150 (which was basically an Ultra-1 desktop in a tower case with 12 disks). If you powered it off, you either had to have a Sun keyboard to power it on, or open up the case to hit the power switch which was internal only. Not one of Sun's finest designs...
I'm pretty certain I've heard this solution before ...
... right here on ElReg, in fact. Probably several years ago.
A quick poke around in the forums turns up nothing.
Anybody?
Re: I'm pretty certain I've heard this solution before ...
Not an El Reg story, but from the Daily WTF. Even mentions using phone books to get the height right. I'm thinking this story may be in Urban Legend territory.
https://thedailywtf.com/articles/ITAPPMONROBOT
Very Minor...
Had a very minor bodge on an HP 486 SCO Unix server where the power switch was perilously close to the tape eject button. Fortunately we discovered this whilst still in the test phase and not when the server was live.
The fix was simple, cut up a DAT tape box to provide two pieces of plastic with a workable hinge between them and attach to the server with double-sided tape forming a cover over the power switch.
Re: Very Minor...
Heh. I've got reset & power buttons on the top front of my system's case handily placed between the headphone jack and a row of usb ports.
There is a coaster taped at an angle cunningly calculated to cover the buttons, but leave the rest available...
... and in many years of operation, I have so far managed to resist the temptation to use said coaster as a resting place for beverages.
"GENIUS!" he told us, modestly.
There's no need for modesty. Some instances of genius should be applauded & celebrated.
And this is one of them.
Overtime
This wasn't me, I remember reading it when I was a young whipper-snapper. I think it was in either Practical Computing or PCW in the UK.
The journo had a friend (apocryphal?) who had worked on a large system in the 70s. It was fairly advanced and used an early teletype terminal. He would have to wait for the users to finish for the day, then he could start the reconciliation jobs. He would have to let them run, before powering down the computer and going home...
Only the jobs took hours to complete, which meant missing Corrie or valuable drinking time.
Being a primitive teletype with a roller and moving carriage, the BOFH candidate became creative. A line feed would execute a command, while CR/LF would execute the command and return the carriage to the start of the line... Being sneaky, he batched up all of the commands in the input buffer, then attached one end of a piece of string to the carriage and the other to the power switch of the computer (a throw switch, not a push button).
Thus the jobs would all run sequentially, the carriage would gradually move to its full extent and once the final job had run, it did a shutdown and when that sent the session termination string, the teletype would throw a carriage return, yanking the power switch of the computer to the off position in the process. Obviously, there were a few flaws with this, a spelling mistake in the type-ahead buffer would leave jobs un-run and failed jobs would be ignored...
But, hey, extra beer time whilst being paid overtime for running the jobs - the log showed when "he " shut down the machine, so they "knew" when he had left the building...
Closed a knowledge gap
When studying compsci, I could never really answer: What would be a practical and achievable use case of "cyber-physical systems"?
Bodge repair
Nothing like this level of genius bodgery, but I once fixed a PC with a rawl plug.
I was onsite for server related stuff, when one of the users complained they couldn't switch their PC on. Despite not being there for desktop support I agreed to have a look. The PC had a plastic front panel with a spring loaded button. This had a peg on the back which pressed the actual power button on the chassis. For some reason this was no longer making a decent enough contact to press the power button in far enough to switch the thing on.
After a quick look through the crap at the bottom of my toolbox, I found a packet of old plugs that came with a router for wall mounting it that had never been required. I chopped one of these in half and jammed it onto the plastic peg on the button on the front panel. A quick test and it would now reach the power button and switch on.
One very happy user.
Nowhere near the sheer X factor in this
but I was asked once by a friend at this non-profit I was working for, to fix her home PC, which had stoped working.
I got there and discovered that the power switch had broken off. Supergluing it back together got it back up and running. I didn't really think I did anything ...
Photo
The saddest part of this story is that it happened before the ubiquity of camera phones.
Re: Photo
If you try hard enough, you may still be able to find it working somewhere.
Well, I had just started at a company 22 years ago, and there was a big ups in the corner. It had an emergency bypass button on the front. For some reason best known to the manufacturer the button sat 1mm proud of the surrounding panel, and was built with a hair trigger.
I had been there for a week and was doing some work in the server room and my knee caught this button powering down the computer room.
The solution was to stick a backup tape box over the switch with sellotape so it could be removed if needed buy not if kneeded....
I was still working there until 18 months ago....
Did something similar about 30 years ago with a bench drill, for some reason the power-on button was really easy to push, and could be set of by someone just leaning against the thing (waste high controls). Which could be quite dangerous!
We took some heavy cardboard tubing (think toilet roll but about 4mm thick), cut about a 2 inch length, and then gaffer taped it over the on button. Worked like a charm.
Software bodges are good, too
I remember reading about an old system (80's, maybe?) which would occasionally throw a wobbly. Debugging revealed that one particular byte in a program structure was being corrupted, but despite much digging they couldn't find out what was doing it.
As a temporary workaround a piece of dummy data was added at that address, moving the real data one byte down in memory, so that corruption didn't zap the important stuff. Everything worked fine from then on, and I don't think it was ever actually fixed.
If it is stupid, but it works...
...it is not stupid.
Better than driving in the wee hours of the morning to slap a RESET button in a server somewhere.
Not a Who Me?
That hardly qualifies as a Who Me? No blame or shame to hide there. It's a story worthy of On Call (and a medal for novelty) - Heath Robinson would have been proud!
PS I'm not up this early normally during lockdown - it's just that I drove a friend to hospital for a deferred op early this morning. Now trying to keep awake. Strange, getting up early didn't use to be any problem: 4am to catch the 6am flight to Schiphol was a regular occurrence. Nowadays, lockdown and retired, my coffee m/c (bean to cup espresso) doesn't always have time to rest between breakfast and elevenses!!
Beer icon - not every pub across the globe will be shut...