An arc welder in the datacenter: what could possibly go wrong?
- Reference: 1719818887
- News link: https://www.theregister.co.uk/2024/07/01/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Andrew" who told us about the fun times he spent in the 1960s and 1970s working as an IBM customer engineer – mostly for financial services clients of the sort for whom a few minutes' downtime meant a few million in losses.
One of those clients was a stock exchange which called Andrew to ask what IBM intended to do about the large crack that had formed in the metal frame holding a large high-speed impact printer.
[1]
The printer was a monster that produced a then-astounding 1,100 lines a minute but made a horrible racket while doing so. A huge, powered cover was therefore in place to reduce noise.
[2]
[3]
The crack in the frame meant the cover would not function properly, making the printer a mighty irritant for the stock exchange.
As luck would have it, Andrew's family ran a metalwork business. He had grown up in its factory, and become a handy wielder of an arc welder.
[4]
IBM and the stock exchange therefore hatched a plan.
A portable arc welder would be found and brought onsite. The stock exchange would open its doors on the weekend, power down its computers – and the fire alarm and sprinkler system – and Andrew would fix the frame without causing a meltdown.
"Come the day, all went as planned. I first cleaned all the oily paper dust that had accumulated, then duly fixed the frame. The cover worked to everyone's satisfaction," Andrew told Who, Me?
[5]Admin took out a call center – and almost their career – with a cut and paste error
[6]Techie installed 'user attitude readjustment tool' after getting hammered in a Police station
[7]Seething CEO shoulder surfed techie after mistaken takedown of production server
[8]Screwdrivers: is there anything they can't do badly? Maybe not
Years later, after Andrew had moved onto a different team, his phone rang.
"You just broke the stock exchange," a fellow Big Blue employee said, angrily.
[9]
Andrew hadn't set foot in the bourse for years, so was bemused as his colleague turned the air blue berating him for having caused the crash at a key client.
"After he had calmed down, he explained that another printer frame had cracked and one of the team who watched me weld the other one years ago thought he would do the same."
Unlike Andrew, however, this chap had never welded anything before in his life.
He did at least repeat Andrew's process of securing client permission, and having the sprinklers turned off.
But he forgot to have the alarms turned off.
And he did the job during business hours.
You can guess the rest: the fire alarms were tripped, all critical systems went offline, IT staff were evacuated, and the stock exchange took an expensive unplanned break.
And somehow Andrew was blamed for the mess – because years earlier he had left the portable welder onsite.
Really, how hard can welding possibly be, right? Have you ever assumed a skill would be easily acquired? If so, [10]click here to send us an email and we may share your experience in a future instalment of Who, Me? ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZoJ@P0westGcSQBcUTW@pgAAAFg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoJ@P0westGcSQBcUTW@pgAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoJ@P0westGcSQBcUTW@pgAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoJ@P0westGcSQBcUTW@pgAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/06/24/who_me/
[6] https://www.theregister.com/2024/06/17/who_me/
[7] https://www.theregister.com/2024/06/10/who_me/
[8] https://www.theregister.com/2024/06/03/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoJ@P0westGcSQBcUTW@pgAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
There is margin on those equipment purchases...
well...
Witless wonder wields welder when warning watsit working.
Re: well...
ale awarded
all alliteration always appreciated
Have you ever assumed a skill would be easily acquired?
No, but assumed a skill would not be lost. Recently, an electronic appliance started to malfunction, i.e. some relay didn't switch any longer. Having been fully trained to know my way around electronics, I went on with fault diagnostics. And I duly (dully!) fried said appliance, beyond repair.
safety goggles ->
Blame-shifting gone mad
Shifting the blame for a cock-up is of course par for the course in these situations. It's a bit rich to shift the blame to someone who did a similar job competently and wasn't anywhere near.
The story also brings back memories of the time I was teaching someone to solder electronic components onto a circuit board, but they kept referring to it as welding. I would certainly not recommend welding to attach even the more robust triacs (380V, 32 A, TO-48 housing) used for a theatre dimmer to any circuitry, so just to drive the terminology home, next time they said they were going to weld a component to a practice circuit board, I couldn't resist going "KZZEERRT" as the soldering tip touched the component. Gave them quite a shock, but they did find it funny. Their soldering was actually pretty neat.
Re: Blame-shifting gone mad
I did encounter some dodgy circuit board where I wished that the components had been welded on rather than soldered. Then again, I've also had some (stainless steal) welding, that did let go.
"You just broke the stock exchange"
Proper answer : Fuck you, I wasn't there and you're incompetent.
Frankly, it beggars the mind that someone could actually think to call a person who isn't even in your team any more and blame them for your cock-up.
Re: "You just broke the stock exchange"
I'd have said "Noah, the arc welder is your problem"...
Re: "You just broke the stock exchange"
It is the very strong SEP field that keeps local blame away.
Having rebuilt a couple of them ...
... I'm here to tell you that you don't just fire up and use an arc welder after it's been sitting around doing nothing for a few years. Such equipment attracts rodents, reptiles, avians and insects, all of which do their own particular damage. The worst is rodents which gnaw on everything, and then pee all over it, just to make sure it's completely toast.
On top of that, after sitting around for a few years, the cables will have gone walkies, there would be no stick/rod to be seen, and all of the other little bits & bobs would be similarly missing. Including the helmet, which I heartily endorse for arc welding.
And then we get into the damage caused to the old-school data center. Drives would have been crashing left and right for weeks because of the shit that welding kicks onto the air, tape read/write heads would get trashed, etc. etc. One quite simply doesn't weld inside a datacenter, not without precautions ... and the story about that is a tall tale unto itself. (Picture a two-layer "shed" made of 10mil clear visqueen, with it's own HVAC and filtration, built around a rather monstrous (but broken; forklift accident) iron casting that was used to measure particle location after running the beam through an experiment at SLAC ... ).
Other than that, cool story, bro.
Re: Having rebuilt a couple of them ...
[Voice of Sean Bean] "One quite simply doesn't weld inside a datacenter"
But rodents near the stock exchange's data centre? I'd assume they'd have bigger problems than a non-functioning welder. And if it wasn't near the dc, it's highly unlikely that someone would have found it in the first place.
So the second guy was less Andy with the welder?
Welding in a chemical plant
Peripherally related to this item:
Many years ago a chemical plant had a scheduled shutdown for maintenance and the installation of some steelwork, by welding. When they wanted to start the plant up again they discovered that the stray currents from the welding had burnt out a rack of instrumentation, which took several weeks to replace :( Very costly loss of production.
When arc welding you have to connect the ground/return close to the point where you are welding. If you use long cables and have a considerable distance between the grounding point and the welding point you can get nasty stray currents. And I imagine the long welding cables carrying a high current could also induce current in nearby cabling.
Apparently any further extensions of the steelwork on that plant were done with bolts or clamps, not welding.
Re: Welding in a chemical plant
'It was started by stray sparks from a welder' seems to me one of the most common causes of industrial fires.
I set fire to the bin in my workshop once when welding nearby... oops!
I mean, the correct response would be to ask why they are processing GDPR for hanging on to my details for that long... I've always found that most useful at shutting up long dead and vacated employers coming back to demand unreasonable stuff...
Nowadays, yes, but back in the days the story was set, GDPR wasn't a thing.....
If you reread the story then you'll see that he was still employed by IBM, only in a different team.
Let's start a fight with the Welders
"Really, how hard can welding possibly be, right?"
Welding in and off itself, just getting a basic, functional weld, isn't really all that hard. You could teach a child to do it. In a lot of situations, people care too much.
That said, knowing when to attempt a weld, what precautions to take, whether you need pre-heat or post-heat on that particular material, whether a material is weldable with what flux/shielding/rod and then putting down a testable, clean bead on something critical, that is where the skill is at and when you might want to hire someone with actual certs, training and experience.
typical IBM engineer, charges the customer for a bunch of extensive tools that they only ever use once, and leave in the datacentre, making them the customers "problem", and refusing to take any responsibility for them, but they MUST be left there, or else!