Oh, no one knows what goes on behind locked doors... so don't leave your UPS in there
- Reference: 1607674506
- News link: https://www.theregister.co.uk/2020/12/11/on_call/
- Source link:
Today's story, from a reader Regomised as "Paul", is a reminder that it isn't only [2]network cables that have two ends... and two plugs.
Paul's yarn takes us back to the early 1990s and the mighty [3]MicroVAX II , as the minicomputer business was undergoing its decline.
Introduced in 1985 as a follow-up to the unsurprisingly named MicroVAX I, the machine was part of a range of relatively low-cost minicomputers (or minis) from Digital Equipment Corporation (DEC). While it could run DEC's own variant of Unix, ULTRIX, VAX/VMS was the operating system running on the hardware Paul was supporting.
Paul's task was handling service calls for hardware and software for various clients. "One business," he told us, "was very diligent in following recommendations, even to the point of equipping their payroll machine (a MicroVAX II) with a 1.3KW UPS unit so it could gracefully shut down in a power failure."
A sensible precaution to our mind, although he noted: "Unlike PCs and Windows, VAXes and VMS were particularly robust on abrupt failures, but management declined to take the risk anyway."
However, even with all the precautions, Paul still received The Call. The payroll server was down and no amount of jabbing the power button would bring it back. The building had power, and that whizzy UPS should have kept things ticking regardless, but the MicroVAX was dead.
Fearing the worst, and expecting to find a smoking ruin of a circuit board waiting for him inside the server case, Paul made the trek to the site.
He hit the power switch and, sure enough, nothing happened. Just like the customer had said. "But then," he told us, "I notice the console, a venerable VT420 dumb CRT terminal, is also dark and cold."
A bit odd. Even if something had gone catastrophically wrong within the MicroVAX, the terminal should at least be showing some sort of baleful power light, but it too was dead.
Curious, Paul plugged it into a handy wall outlet. The VT420 lit up.
Using impressive deductive reasoning, he followed the power cord to which the MicroVAX and terminal were connected (at least until recently). It led to a locked door behind which was the company safe, telephone gear and, thanks to some dubious planning, the UPS.
It took a little while longer to track down a key in order to unlock the door before – hey presto! – the mystery was solved.
"It seems someone had unplugged the UPS from the wall and not bothered to plug it back in," he sighed. "After about an hour the UPS shut down, taking the server with it."
The phantom plug puller was never identified.
A swift reconnection and flick of the switch, and service was resumed. And Paul, of course, was hailed a deductive genius with near-Sherlockian powers.
We've encountered servers bricked up into secret rooms many times, but never a UPS hidden behind a locked door. Ever had to follow the power lead only to discover another person's stab at security? Let us know with an email to [4]On Call . ®
Get our [5]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://www.theregister.com/2020/12/04/on_call/
[3] http://www.computinghistory.org.uk/det/40924/DEC-Microvax-2/
[4] mailto:oncall@theregister.com
[5] https://whitepapers.theregister.com/
re "management declined to take the risk"
Oh, what a soul-soothing thing to read. Especially so when the root cause of so many of these Monday and Friday stories is penny pinching and inept bean counters at the helm.
MicroVAX II Mayflower
Was amazed when I saw one of those installed in my "datacenter" (basement office with a lock on the door in a dingy part of Westminster). I thought that they were intended for industrial use and kept being told, "statistics, now shut up". This one was a beast with 16Mb RAM added on, plus a backup module in case that failed. Never had to go near it apart from to replace the UPS several times.
Re: MicroVAX II Mayflower
I'm not sure about Industrial use as such, there was a lot of scientific use as well. I can well believe the Statistics claim as there were several stats packages that ran on VAX architecture back then.
I cut my teeth on the larger machines (11/780 and 11/750 in first role for scientific work then an 11/785 in education) but installed two smaller machines.
First one was a K9* MicroVax II at the local college for the engineering department, dedicated to the PAFEC finite element analysis suite (which at the time included such wonderfully named packages as SWANS - Surfaces With A Nice Shape!)
Later I installed a MicroVax 2000 to run a 20 user MRP system so sort-of industrial.
*K9 'cause they resembled the robot dog from Doctor Who.
Ahh Pafec
A UK software house that punched above its weight.
Other products of theirs had similar fun names:
DOGS - Drawing Office Graphics System - a 2D CAD system that was an AutoCAD challenger up to the late 80s.
BOXER - their 3D solid modeller.
DOGS was eventually offered for free (pay for the floppies) as a cut-down PC port. Required more than 5meg of HD space which put it out of reach of most engineering students :(
Re: MicroVAX II Mayflower
I used to work for PAFEC, coming up with new names for packages was always fun. SWANS was a a bit of a effort.
We had PIGS (Pafec Interactive Graphics System), DOGS (Drawing Office Graphics System) MOGGIES (Mapping Orientated Graphics ...something something)
I seem to remember that the lubrication module for the finite element software was SLUGS
It was the cleaners!
Was the bane of my existence... Now we do our own house cleaning in secured areas.
Why is that thumbnail with the lady trying to open the door so familiar?
Dunno. Do you happen to know my ex?
Because you've been reading El Reg for a while?
https://tineye.com/search/2964449cf69a8ac5bb45d9bf08077af48d989358?domain=theregister.co.uk&sort=size&order=desc&page=1
We've encountered servers bricked up into secret rooms many times
we have?
Well, at least once...
[1] Lost server
[1] https://www.theregister.com/2001/04/12/missing_novell_server_discovered_after/
If the general public start reading this column, they're going to think all you have to do in "I.T." is "turn it off and on again" and in extreme cases "plug it back in" !
"...If the general public start reading this column, they're going to think all you have to do in "I.T." is "turn it off and on again" and in extreme cases "plug it back in" !..."
What? You mean you don't??
And a good thing too. It would solve a lot of problems at source even if it dried up "on Call" fodder. I've even be called out by a niece to fix her "internet" - all it needed was power-cycling the router.
Staff reduction...
...can cause interesting issues as well.
I looked after a bank's back office in the North East of England.
It was staffed by lovely people with a small in-house IT team that knew their systems (and their stuff generally) so I only tended to get involved in project delivery or when things couldn't be fixed by themselves.
Alas the bank saw no need to have a support team so far away from their head office and made them all redundant. After all, it could all be done remotely, right?
Some months later I got a call asking me to head up there somewhat urgently as they had a server that wouldn't boot - it was giving a "Non system disk or disk error" type of message.
Of course, Compaq iLO (because it pre-dated HP buying them) wasn't much use for what was believed to be a disk / controller error.
So I drove to the office. Picked up a replacement RAID controller, cables and disk and then drove up to site.
Of course, it took the better part of an hour to find a security guard with access to a key to the room with the server in it.
And about .5 seconds to eject the floppy disk that had somehow been inserted, so the server could reboot.
All in all, a good 5 hours of faffing around to fix something their on site support would have done in seconds had they not got rid of them.
Re: Staff reduction...
As soon as I read "Non system disk or disk error" I thought, "Oh, bet it's a floppy left in the drive." Been there, done that...
Re: Staff reduction...
Vaguely similar. Many years ago in my last job on my very last night on call I had to go and sort out a Windows server (and I'm a UNIX sys admin). Fortunately this was a known problem in that the computer would somehow lock itself up and the fix was, of course... press the power button on the front to reboot.
I recall having an argument with my boss of why should I have to go into the office a 11pm to sort out a very well-known problem (with that server), especially when the fix was the press of a single button. I lost and had to go in. Oh for the sort of remote access we have now!
Not quite a power lead...
But I did work in a place where they had to get an axe to chop through a server room security door when the machine controlling the swipe card access failed.
Re: Not quite a power lead...
Which, presumably, was behind said security door...
This is when Dymo tape shows its worth..
One of the best I saw was "Do not unplug this on pain of death, yes this mean you! I don't care who you think you are. Unplug this and you're toast"
Re: This is when Dymo tape shows its worth..
Could you combine the Dymo tape with a fake laser gun on the ceiling, pointing to where the person doing the unplugging would stand? And change it to "Unplug this and you're toast. Literally. Look at the ceiling. "
I've worked in a ton of places where only one person has access to the building's fuse box.
I can see the point, but one dodgy kettle and you're offline for hours.
"...one dodgy kettle and you're offline for hours..."
Talking dodgy and kettles, I worked at one Atos site that had a sign in the kitchen area that said "Please do not user the kettle to boil milk. Use the microwave!"
Who the hell needed telling that? Clearly it must have happened to justify a sign!
I dated an american back in the late 2000's and we stayed at the Hilton in Croydon (don't ask).
She thought she'd be sweet one day and proceeded to make me a milky coffee.... But pouring the water, coffee and milk into the kettle.... Which then proceeded to trip the floor's electric.... Whilst I was in the shower.
Fun times....
Ah, never underestimate the depths of stupidity that can be exhibited by members of the human race.
In my student halls in Manchester I think our rooms had a breaker set for something measly like 10A, so wasn't impossible to trip it with the right equpiment :)
The fix was to trundle off to the Hall's reception to report it and then somebody would come maybe within a day to flip the breaker. Happened two or three times to me, and obviously very annoying having no electric for the evening. At some point however, I noticed/figured out the cupboard in the corridor could be opened with some key-like device aka screwdriver to reveal a few breakers - happy days!
We had a Prime minicomputer, running a stock control system at a warehouse in the Cotswolds. When it was installed, a UPS was also fitted.
One day contractors digging up the road outside hit the electrical supply and the power to the building went out. After finding a torch, I headed to the computer room to gracefully shut down the machine, as the power was likely to be out for several hours and the UPS was rated for about 20 minutes. When I got there, the machine was dead. Later investigation showed that between the UPS and the computer was an RCD circuit breaker wired in such a way that a loss of the mains power would trip the breaker, cutting all power to the machine.
Should I mention that the warehouse was part of a department full of electrical engineers, who installed and wired up the supply themselves? No - that would be embarrasing!
A few years ago we had a small customer whose office server went down one morning: "help! we can't get to our files. No, nothing's changed. Nothing". My colleague went to the site, which luckily was only 10 minutes away. When he got there, he admired their large Christmas tree festooned with twinkly lights. In the corner where the server was. Within about one minute he also admired how they'd been so creative with the mains plugs. He reported his findings to the office staff immediately: "Well the good news is that your Christmas tree lights will carry on working for hours, or even days, in the event of a power cut. The bad news is that you unplugged the server so you could power the Christmas tree lights. From the UPS".
They were up and running within minutes, and no harm done.
Till the next year, when EXACTLY the same thing happened.
This happened to me. I sarcastically suggested gluing the plug to the socket excepting the manager thought I was being serious so he did.
Several years later, the electrician was asking which dimwit glued the plug to the socket ...
1.3KW UPS
Going to need a bigger battery...
"It took a little while longer to track down a key in order to unlock the door ..." Presumably there were only a very few keys?
"The phantom plug puller was never identified." Publically...?
I wonder if they noticed the metal shavings around the safe from the electric drill... The sound of which would have been covered up by cabinet fans, printers, etc. outside the room. And no one would have ever known except the thieves forgot to re-plugin the UPS. Another security failure covered up by the board of a company.