Thanks boss. The accidental creation of a lights-out data centre — what a fun surprise
- Reference: 1623051067
- News link: https://www.theregister.co.uk/2021/06/07/who_me/
- Source link:
Naming no company names, our reader (regomised as "Jim") told us "I was a shift leader in a data centre that ran the European IT for a big American pharma way back when."
These were the days of big metal, and the data centre housing the mighty IBM 3081 mainframe was co-located on a manufacturing site. The site also had its own IBM System/34 to handle local site operations.
[2]
The System/34, which was introduced in the 1970s, was one of the last great hurrahs of the breed. By the time of its withdrawal from marketing in 1985 the world was moving on and its original concept as a "low-cost approach to distributive data processing for businesses of all sizes" was beginning to look a little dated.
[3]Bite me? It's 'byte', and that acronym is Binary Interface Transfer Code Handler
[4]Terminal trickery, or how to improve a novel immeasurably
[5]Forget GameStop: Keyboard warriors and electronic trading have never mixed well
[6]How do you save an ailing sales pitch? Just burn down the client's office with their own whiteboard
The IBM 3081 was launched in 1980 and [7]lingered on until 1987 .
Back then, however, the bosses of the manufacturing plant were proud of the kit entrusted to their location and never missed an opportunity to show visitors around and "boast about the big shiny boxes that were located on their site," as Jim put it.
[8]
[9]
"We often had to play host to gawking tourists," he sighed.
And so it was, on the fateful day in question, that a bigwig was once more dragging a gaggle of onlookers around Jim's domain. Puffed up like some giant corporate peacock, he waxed lyrical about investment, new technology (yes, the System/34 was once considered new) and the expensive infrastructure that was required to keep everything ticking over nicely.
[10]
Jim and his fellow shift workers kept busy as the corporate effluent flowed overhead. A shame, because one of them might have managed to stop what happened next.
Turning to the shift workers, the bigwig pointed to a big, red button on the wall and loudly asked: "What's that one do?"
Without waiting for a response (or pausing to read the notices above) the manager jabbed the button, keen to show off both his seniority and technical prowess to the visitors.
[11]
"The world," said Jim, "went silent." The bigwig's face turned the same shade as the emergency stop button he'd just pressed.
It took the rest of the day for the team to recover both the mainframe and minicomputer; computers reacting badly to a surprise shutdown has, after all, always been a thing in the IT world.
A precursor, perhaps, to the lights-out managed dark data centres of today. Just with those pesky humans to really mess things up.
Tell us about your slow-motion moment as a manager hit a switch that should have been left alone, or when some innocent showing off to tourists went catastrophically wrong. All it takes is an email to [12]Who, Me? ®
Get our [13]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YL3uaEq3xJAT1r6JKfReMgAAAMU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.theregister.com/2020/06/01/who_me/
[4] https://www.theregister.com/2021/05/03/who_me/
[5] https://www.theregister.com/2021/02/22/who_me/
[6] https://www.theregister.com/2021/02/01/who_me/
[7] http://www.chilton-computing.org.uk/ca/technology/ibm3081/p001.htm
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YL3uaEq3xJAT1r6JKfReMgAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YL3uaEq3xJAT1r6JKfReMgAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YL3uaEq3xJAT1r6JKfReMgAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YL3uaEq3xJAT1r6JKfReMgAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] mailto:whome@theregister.com
[13] https://whitepapers.theregister.com/
Re: Push me
I once had a poster with the Hand of God reaching down out of the clouds & poised to push a big red button. The caption read "I wonder what this button does?"
I bought a USB-attached big red button under the clear plastic "missile launch" cover, the words "Emergency Use Only" across the top. You flipped it open, it made the classic warning klaxxon siren, and the button lit up even brighter. Press the button & the light/siren turned off to be replaced with a cheery female voice "Thank you for pressing the Big Red Button. You have aproximately 5 seconds to kiss your ass goodbye. 4... 3... 2... 1... Too late!"
My posting of the poster above the physical button on my desk made my coworkers a bit amused/nervous. I don't know why...
*Wanders off whistling innocently*
Re: Push me
I'd managed to get this far into Monday without hearing of something I wanted to buy. If I ever see the inside of an office again I'm getting one of these.
Re: Push me
I don't know if the original is still floating out there on Ebay, but the Staples (office supply store) "Big Easy" button might be easier to find. Scrub the original letters off, replace them with well-crafted Tippex lettering, & give it an internal RPi to give it the ability to play prerecorded audio clips of your choosing.
*Hands you a pint*
Just remember NOT to wire it up to the orbital LART controls lest you turn someone into a smoking glassy crater "on accident". =-D
Access denied
The last data-center I was working at had a strict policy of non-interference. That meant, the "Boss" did not have the key to the rooms where both active and passive equipment were located. Only certified people were allowed entrance.
Re: Access denied
An entirely sensible policy. So sensible I'm genuinely surprised!
Re: Access denied
I had the same policy, plus all guests had to be authorized by me, in advance so I could ensure we had a couple of sysprogs to act as guides / wardens.
Imagine my surprise returning from a meeting to find a tour of school children in the data center, no I hadn't been informed, no there were not sufficient people to safeguard my DC and yes it had been organised weeks ago by a county Councillor who hadn't bothered to tell anyone.
It was to late to stop that one but I did have an interesting 'chat' with my manager involving resignation threats, and a press release in the case of an unplanned outage. I was completely serious about both. I'd put my body and soul into taking a team who were not meeting any SLA's to a 99.999% availability across 3 different mainframe technologies and was not having a return to the bad old days.
Re: Access denied
Only certified personnel allowed? That's fine, I'm certifiable so I should get in, right? =-D
Re: Access denied and Big Red Buttons
We had the same thing at BT data centres. Many's the time when I had to inform a prospective client that the requested contract clause that they could turn up and inspect the servers running their service without notice was not possible. I would carefully explain that the sites were secure, some ran Critical National Infrastructure (CNI) such as 'Blue Light' (999) services, larger supermarket chain logistics systems and suchlike, and that even the Chairman and CEO turning up would not get in without an appointment.
If they wanted to come in and test the power fail-over in the event of a mains failure, they could arrange to conduct the once a year test. One of the data centre managers told me that only half of the customers who had arranged to literally 'press the big red button' and turn off the mains power actually had the courage to press it during their visit.
I once, as a consultant visited Shell's North Sea modelling data centre, getting into the computer room (very secure actually). And yes there was a 'big red button' - I felt a strong urge to press it, but managed to keep my hands in my pockets.
If you display a big red button, expect it to be pressed. Stupid IT.
Obligatory quote...
I feel a Terry Pratchett quote coming on....
"If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH,' the paint wouldn't even have time to dry."
Re: Obligatory quote...
*Presses the button*
=-D
Hands in Pockets!
Anytime I visit a place like this as a visitor.... keep my hands in my pockets! No touchy nothing, nada!
My coat has pockets!
Re: Hands in Pockets!
Many moons ago I took my daughter to SLAC on take your kid to work day. At the ripe old age of 9, she had been there many times before and knew the ropes, but I figured she deserved a day out of school.
She told me as we were walking in that it'd cost me ten bucks for her to not push any buttons. I gave her the money.
On the way back out, I told her that it'd cost her ten bucks for me not to tell her mother she was running a protection racket. She made a face and paid up ... and promptly told her mother as soon as we got home. They both still laugh about it :-)
At Jake...
With skills like that, your daughter will rule the world someday. Please put in a good word for me to be fed to the sharks last, ok?
I, for one, approve of our new Evil Overlordess. =-D
Re: Hands in Pockets!
Jake: "Many moons ago I took my daughter to SLAC on take your kid to work day. "
Oooh, Oooh, can I come next year, please, pretty please?
I promise I'll be good, and not press any buttons, except in the lift or on the coffee machine.
Hmm well I remember a few years ago that as you left a comms room there were two buttons by the door.
The top button released the lock on the door, the bottom one was a big red button.
Yes someone (not me) pushed the wrong button to open the door.
After that the red button got a cover that had to be hit hard (punched) before the red button would operate.
A&E light switch
Was in A&E at the weekend for a few hours. We were all being kept amused by people trying to leave. Sign on door said "press button on wall to open". Which meant the waiting room kept being thrown into darkness by the wrong button pressed...
One button was small and unlabelled and looked like a light switch. The other was a large six inch square metal button with Door Open on it in large green letters. Easy mistake to make...
Re: A&E light switch
On the subject of poor automatic door design, I have frequented several shops / banks etc that have upgraded their doors to operate by push-button. I have no problem with this.
The problem I have is that a lot of the doors open at the speed of an arthritic snail, they can't be opened manually, and that it isn't immediately obvious that the doors are automatic at all. So you have a consistent flow of able-bodied people ploughing into the door at pushing force. I've seen doors that either open automatically and quickly, or can be both pushed open and opened via button. This makes the above design just baffling to me.
Re: A&E light switch
Button "noun - a small device on a piece of electrical or electronic equipment which is pressed to operate it".
Six inches square is not small, and not a button.
"Press pad on wall to open" may have been more helpful!
Re: A&E light switch
Trouble is that doors with electrical locks in that situation are an invitation for that sort of mistake.
For a start you're in a hospital which by definition is a place you wouldn't be at unless somethings bad has happened. You're either relieved to be fixed or stressed, if you have kids with you that's additional stress, and the last thing you want is a door that doesn't just open but also has an instruction just of out of eyeline and a choice of switches probably a bit further out of eyeline. And you've already made yourself look a bit of a numpty by walking into a door that didn't open so you're even more enthusiastic now about just getting out. You scan the environment for clues, see the words Switch, Exit and Press and hit the first one you see.
The people who fit these things aren't ergonomics experts or sociology graduates, they're electrical fitters who have a practical plan and approach to doing it. As everyone knows (on here at least ) if there's more than one option and one of them has an undesirable outcome, it's better to make the desirable outcome the default that would be hard to get wrong. Don't rely on people in unfamiliar situations having the time or the wherewithal to weigh up all the options and to carefully consider the various options and outcomes before executing the default or - to their eyes - the easiest action. Sod's Law still applies.
Sound of silence
At the beginning of each shift, there was a meeting to discuss what had happened and anything special for that shift, at the end of the meeting, everyone traipsed to the bridge in the computer room.
Access was via a card held near the wall at a certain point and an electromagnet held the door in place.
The door magnet was a bit weak so with a bit of a shove it was possible to open the door without the door being unlocked (there were manned desks in the next compartment so anybody coming in was seen).
Did I mention the emergency shutdown button on the wall behind covered with its nice plastic cover and the door stop that was next to it.
Well, as "someone" (not me) shoved the door, the door release was activated at that exact point by the card in their pocket, door acclerates into the door stop which promptly gets pushed into the wall behind it as it was only a weak partition wall, shattered the plastic cover and hit the emergency button.
Silence!
Off went the IBM mainframes, the ICL's, the VAX's, the AS/400, the HP's, the stratus non-stop (they stopped!), the System/38's and whatever other collection of stuff was in the room - all in the process of their overnight batch runs.
The only saving grace was it was going into a night shift - the evening shift were a little unhappy about having to stay for a couple of hours longer as everyone rushed to get it all up and running and restart batch runs at their checkpoints and other stuff.
The emergency shutdown was moved off of the wall behind the door and the door magnets fixed.
Fun times!
Re: Sound of silence
I've heard so many of these stories, and actually seen a few stupidly placed big red buttons.
So the thought is readily prompted..... Someone got paid real money for that decision.
Radio Keys
One DC I worked in had radio keys to control access to the DC. These, we about the size of a small mobile phone today and were generally kept in your trouser pocket all day if you required regular access to the DC. The Keys required charging every night so had to be locked into the charging station before you left for the day. It was a major pain if you took one home overnight and fessed up a lot of fuss was made. In reality if you didn't use it much it would just about last a second day. There was zero lag on the system so we got used to running through the door if there was an emergency (this was in the early 80's so they were horribly common) one of the ops found out that his key was dead when running into the DC. The door didn't open but he carried on, that's when he found out that the glass panel above the door wasn't shatter proof. He was lucky to get away with some deep lacerations to his chest. A post incident review lead to the glass being replaced with shatterproof wired glass and a relaxation in the rules around using a 'spare' key, whilst usage had to be logged and the users original key had top be present being charged it was no longer a disciplinary offence taking a key home by accident.
The operator was a resilient chap and actually was pretty happy a couple of weeks later, he'd been booked to go to Spain for 2 weeks and his holiday insurance company refunded the money as he could t take part in the water-skying and diving he had planned so he got a free holiday (lounging by the pool drinking as usual). His holiday was taken as sick leave and he got bunged a few thousand pounds in compensation for his injuries.
Paris as he apparently got a lot of attention from girls who were very sympathetic about his bandages.
put your hands in your pockets and leave them there
On a trip to support a very large organisation (where they had men with machine guns on the outer perimeter, all bags were x-rayed, and we were body searched on entry), we had to go into the inner sanctum operations room. We were told by my company "put your hands in your pockets and do not touch ANYTHING".
We could not touch a (dumb) screen we had to stay 2 inches from it. This means pointing to the error was a challenge.
Someone had been caught on video, pressing the "next" button to display more error diagnostics. This caused a huge stink, involving senior managers having to apologise.
As we kept our hands in our pockets we could not get out of the room unaccompanied. Even in the "unsecure" areas we were not allowed to go to the toilet unaccompanied, but at 0300 in the morning no one was too worried.
The magic loo-roll and your pockets are too big.
Two "red button" issues.
The big red button was at the end of a row of tape racks. One balmy after while the junior was putting away the tapes, they inadvertently bumped into the unprotected BRB - silence issued. The ICL's the PICK, and other assorted equipment goes silent. Rather than the flashy plastic, up-market flip cases all the "rich" computer rooms above seemed to go for, we put a trusty loo-roll centre over the button - this worked.
The best story, a hapless (and rather senior) tech support person was in the computer room having a chat, and walked past the ICL 2966 - WOPR (sadly not as advanced and blinkly lights as the War Games one) - just a monitor and keyboard station on a stand. And caught the reset switch with his pocket. Queue the requisite running around to get the borough back up and running so people could pay the rents and rates. The fix for this, was to remove the reset switch - that just slide out and leave it on the top so it could be popped back in when required.
Push me
Everyone who has been in the industry long enough has witnessed a [1]Father Dougal moment.
[1] https://forums.theregister.co.uk/forum/all/2017/11/24/on-call/#c_3356383