Standards-obsessed boss ignored one, and suffered all night for his sin
- Reference: 1706257808
- News link: https://www.theregister.co.uk/2024/01/26/on_call/
- Source link:
This week, meet a reader we’ll Regomize as "Laslo," who once managed a disaster recovery center – a specialist facility that housed its customers' emergency rigs with appropriately resilient and redundant everything, and even offered empty desks so that when trouble struck, clients' teams could assemble in their dedicated secondary workspace and carry on more or less as usual.
Laslo's facility also housed a few racks that contained servers used in production. And one of those racks included an IBM AS400 server and 42U of ye olde harde disques .
[2]
That enormous pile of storage made the rack too heavy for the raised floor. But the owner of the AS400 was one of the center's biggest and most prestigious clients, so Laslo and his crew had a local welder fabricate a steel plate to sit under the rack.
[3]
[4]
"We spread the load across ten raised floor supports rather than the usual four," Laslo told On-Call. This arrangement meant the rack stood out from all others in the facility – literally and metaphorically, and also because it was alone in having its cabling visible.
At about the time Laslo gave the monster rack the support it needed, the DR center's operators bought into ISO standards, big time. The management team was duly restructured, and the compliance operator given control of operations because it was felt his relentless pursuit of standardized perfection was just what the company needed.
[5]
This new manager duly visited the DR center and saw the AS400 rack, and decided it was both anomalous and dangerous. Someone could trip on the cables! And the bigness of the rack wasn't standard!
Laslo's perception of his boss's thoughts was that the rack "had to be rectified before the compliance police shut the whole place down for endangering life as we know it."
So the boss took it upon himself to sort out the situation by snipping the ties off the cables running to the rack to make enough slack that the AS400 and all those disks could be pushed back into line.
[6]Junior techie had leverage, but didn’t appreciate the gravity of the situation
[7]While we fire the boss, can you lock him out of the network?
[8]Tech support done bad sure makes it hard to do tech support good
[9]People power made payroll support in putrid places prodigiously perilous
That decision was taken without Laslo's knowledge, and at a rather late hour of the day.
When Laslo arrived the next morning, his manager did not appear in the standards-approved daily kickoff meeting which, in line with standard procedure, commenced with a review of overnight incidents.
[10]
Logs had recorded quite an interesting one: someone had gone into the datacenter, but nobody had come out!
An investigation of that anomaly found … the standards-obsessed manager, propping up the AS400 rack to prevent it from toppling over and almost sobbing with the effort of having done so for most of the night.
"It turned out he had wheeled the rack off the metal plate and as soon as the first two wheels rolled onto the raised floor, they had gone straight through," Laslo told On Call. The rack had therefore listed a little.
The standards-obsessed boss knew it was a production machine and that if it fell and stopped working, the result would be well and truly non-standard.
Which explained his epic all-nighter.
But as Laslo told us, it was not a necessary effort. As he set things right, he realized the rack would not have toppled beyond the slight angle at which it rested – because the non-standard steel plate would have held it in place.
"I still pass by the building from time to time. It's now an office," Laslo told On Call. "And whenever I do, I wonder if that manager is still there, valiantly and somewhat pointlessly, leaning on a rack."
"I hope he is."
Have standards made a mess of your IT? If so, [11]click here to send On Call an email and we may feature your story on a future Friday. ®
Get our [12]Tech Resources
[1] https://www.theregister.com/Design/page/reg-standards-converter.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZbOQ27KKzWZPVXzUFf-AMAAAAEA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbOQ27KKzWZPVXzUFf-AMAAAAEA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbOQ27KKzWZPVXzUFf-AMAAAAEA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbOQ27KKzWZPVXzUFf-AMAAAAEA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/01/19/on_call/
[7] https://www.theregister.com/2024/01/12/on_call/
[8] https://www.theregister.com/2024/01/05/on_call/
[9] https://www.theregister.com/2024/01/01/on_call/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbOQ27KKzWZPVXzUFf-AMAAAAEA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Re: EMC Symmetrix
At least a few of the raised floors dotted around Silly Con Valley would easily support an F-250 full of line printers, card readers, tape drives, and the like. You can guess how I know this.
IT Manager to the rescue!
I worked for a company that held a company ball in a marquee on the sports field (don't ask). The marquee was also the location for our yearly conference.
One year during the conference there was a storm brewing, and the wind was getting up a bit to much. So much so, in fact, that people were getting nervous sat inside what could be considered a very large kite. The wind got stronger and one of the guy ropes valiantly gave up and came pinging out of the ground. This obviously made the post it was holding in position lean inwards quite a bit. One of the IT managers bravely pushed against the pole to save everyone whilst an evacuation was performed. Once the marquee was empty was out (apart from IT - always expendable), the manager let go of the pole and ran. He needn't have bothered - the post didn't move in the slightest and was held in place by the half mile of canvas and various hardwood posts holding up the rest of the marquee..
He still thought he'd saved the day. We didn't have the heart to tell him
propping up the AS400 rack to prevent it from toppling over and almost sobbing with the effort of having done so for most of the night.
Well you gotta give him his due , he's taking reponsibilty for his actions pretty heroically
Well, it was on him. Quite literally.
He really leaned in to the roll.
Flooring
We used to supply rotating machinery supported on (usually) very strong floors. These floors would be practical but utilitarian so to make things more aesthetically pleasing some architect decided that 'computer-room' false-flooring be used to improve his poor initial design. Inevitably, this low but suspended floor resonated with the machinery; the vibration was irritating. Of course, as every engineer knows, you can have ten-million pounds worth of rotating equipment running smoothly, under superb automatic hair-trigger control in an unmanned building, but if the flooring rattles it will be a poor day for the manufacturer. We fixed it, but that false-floor would never be removed without welding gear.....
by the book
> relentless pursuit of standardized perfection
or idiot administrators with no clue about the real world, as they are more commonly known
At my previous employer we had a "server room" that was one of the retired professor's office, free as as the university's engineering department was in decline due to no new staff being appointed. A Sun rack in there with a 5kVA UPS, redundant server heads, and around 100TB of storage started to sink in to the wooden floor due to the high pressure around its feet and so we moved it live - literally running - to the other end of the room that was a concrete floor to allow the estates team in to replace the floor with a thicker plywood flooring in place of the modest chipboard originally used.
Then we moved the rack back again, slowly and with it still running (kept network cables in place, unplugged UPS feed), to it intended location once more.
When the facility was decommissioned in 2019 we had to dismantle the rack, as the newer lift was smaller and it did not fit in as it previously did on delivery 10 years previously, and I was astounded by how heavy the rack alone was, it must have been the best part of 150-200kg without equipment as each side needed two-three folks to manhandle it down the stairs.
Of course they are heavy! They have to carry quite a load sometimes ... The 2-post 19" and 23" telco relay racks down in the machineroom/museum/mausoleum/morgue are rated at 1,500 pounds. The uprights are made from 5/8ths inch steel,
Many years ago I worked for a security equipment distributor, I used to sell a lot of racks for DVRs, servers, storage and the likes.
We sold 3 different grades of rack castors, 300kg, 600kb, 1200kg.
Everybody bought the 300kg wheels, reasoning they would only be used once to wheel the rack into place, and that would be fine.
One time we got an order for a lot of gear that would completely fill a 42u rack, UPS, lots of storage, 4 servers, switches and power distribution units, about £250k worth of stuff to go in a casino
I totted up the weight from spec sheets and it was 1,100kg, so natural the customer decided to shave £100 off the £250,000 bill by opting for the 300kg castors.
Last I heard, they had to get a forklift to move the rack as the castors collapsed before they could push it into place...
Being a casino they probably figured they could take the chance ..... cos the house always wins eventually!
Mine's the one with the card counting device in the pocket!
Nice rack
Why sell a 600kb rack if most machines had 640kb?
Needless!!
"it was not a necessary effort. As he set things right, he realized the rack would not have toppled beyond the slight angle at which it rested – because the non-standard steel plate would have held it in place."
That's pretty hilarious!! I would have thought that in such a situation the obvious step, once the initial fall is halted, is to gradually push less and less to see to what extent the heavy object in question can support itself, in which case the standards-obsessed boss would have fairly quickly realised that he could abandon his post at least for the length of time necessary to call for help. I guess he was just in shock-mode and couldn't think it through
Re: Needless!!
Or he had the average analytical ability of the type of person that tends to gravitate to such a role, and was unable to think past "I'm holding this and it's not falling"!
Back in August of '79 ...
... we had a magnitude 5.7 earthquake on the Calaveras Fault near Coyote Lake, just South of San Jose. I was working for a laser company at the time, assigned to one of their manufacturing floors in Mountain View. When the quake hit, I was standing near a rack full of completed, but not yet boxed for shipping, laser levels. About 3 dozen of them. As everybody screamed and ran for the exits, I noticed the shelving was rocking precariously ... So I stupidly leaned on it, and managed to keep it from falling over.
I got a right royal public bollocking for it. I should have screamed and ran, I guess.
I also got bumped three levels in pay grade, not that anyone I worked with was allowed to know ... Seems that The Boss of the division knew about the shelving not being tied down after a company-wide insurance inspection, and had reported to The Board that it had been fixed. If the gear had hit the deck, he'd have been in deep shit. The following weekend, he and I bolted it down properly ... off the clock, but he bought lunch and slipped me a couple hundred bucks.
"All you kids who just sit and whine, you shoulda been there back in '79 ... " —TRB
EMC Symmetrix
Long ago when I first worked with EMC equipment, I was talking with a longtime EMC tech about various subjects while we were waiting for some longish self check to complete after he'd installed some new drives and I recall asking him why the Symmetrix wasn't designed to fit the same dimensions as normal racks. Its oddball footprint caused it to take up more space since it wouldn't fit in the standard rack/aisle plan. He said that was deliberate, they designed it to spread out across enough floor tiles so that at maximum capacity it would meet standard datacenter per sq ft weight limits (though as I understand it, some datacenters have needed to have their substandard flooring reinforced nonetheless)
Made perfect sense when he told me, but it wasn't what I would have guessed. Apparently it has installation guidelines that specify exactly how it is placed on the tiles it occupies to insure it occupies the correct number in the correct way. They even accounted for the weight of people standing around it when it is being serviced etc.
I haven't been around one for years, I wonder if they have a more compact version now that the world has moved on from heavy hard drives to nice light flash drives.