Ask a builder to fix a server and out come the vastly inappropriate power tools
- Reference: 1698391748
- News link: https://www.theregister.co.uk/2023/10/27/on_call/
- Source link:
This week, meet a reader who asked to be Regomized as "Bob" and told us he used to work for a company that sold kit to radio stations – some in remote locations.
"I wrote the software but it was a small company so I also sometimes had to double as Linux support, as I was the only one who could setup the servers the company sold," he told On Call.
[1]
Most of this involved staying in his nice warm office and fixing things remotely. But some of his clients were in Africa, where a studio was being expanded – without the budget for a new server.
[2]
[3]
Thankfully, Bob's employer usually installed a backup server. So even though it wasn't ideal for the expanded studio to run without a backup, that arrangement was reluctantly chosen.
And then regretted, because the backup server was discovered to have failed – and nobody knew for how long!
[4]
Bob was duly despatched, along with a bag of spare parts.
"I got to the customer, and it was literally a building site with new bits being added and the roof being redone," Bob told On Call.
The first server he saw was running but with "a horrible noise from the fan and a lot of clicks from the discs." The backup was utterly dormant, so Bob removed it from the rack, opened it and found "the worst case of dust I've ever seen – two or three inches deep and really clogged."
[5]
At this moment, the station engineer who chaperoned Bob "barks an order at an underling who grabbed the server and said 'We'll sort out the dust'."
Bob watched as the station employees spoke to the construction crew through the nearest vacant window hole, and then as a leaf blower was passed through that aperture.
"Before I could shout 'Noooooo!' the station engineer blasted the dust out," Bob told On Call, before pausing the remind us that this story took place on an active building site – so a leaf blower was likely to push more dust or other material into a server!
[6]Making the problem go away is not the same thing as fixing it
[7]Workload written by student made millions, ran on unsupported hardware, with zero maintenance
[8]Police ignored the laws of datacenter climate control
[9]Techie labelled 'disgusting filth merchant' by disgusting hypocrite
Then came danger from above.
"Someone on the roof had finished with a large bucket filled with murky liquid and asked themselves 'Why carry it down to the ground, when you can just empty it over the side?'"
The resulting deluge just missed the server, sparking much shouting from the station engineer about the need to take more care of delicate electronics.
Bob was eventually able to bring the server into a clean and safe space, peered inside, and was "amazed that all the components were still on the board" – such was the force of the leaf blower's output.
He was even more surprised when he installed a new power supply and the server booted up as if nothing had happened.
Next came sorting out the other server, before installing software, racking and stacking, and then heading home.
"I left a delighted station owner who was oblivious to how close he came to a big bill and delayed opening," Bob recalled.
What's the least appropriate tool you've seen used to maintain hardware? [10]Click here to send On Call your story and we may harvest your story here on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZTuKSFtwHNF9Zr2Y-QIRqgAAAcY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTuKSFtwHNF9Zr2Y-QIRqgAAAcY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTuKSFtwHNF9Zr2Y-QIRqgAAAcY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZTuKSFtwHNF9Zr2Y-QIRqgAAAcY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZTuKSFtwHNF9Zr2Y-QIRqgAAAcY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/10/20/on_call/
[7] https://www.theregister.com/2023/10/13/on_call/
[8] https://www.theregister.com/2023/10/06/on_call/
[9] https://www.theregister.com/2023/09/15/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Shocking!
To the best of my knowledge and belief I have never seen a bit of equipment killed by static electricity.
When you hear about things like this it tends to reinforce my feeling that i'm unlikely to.
Re: Shocking!
You've never handled 4000 series CMOS chips then. At times it seemed just looking at them was enough to pop them.
Re: Shocking!
I've killed a Soundblaster CD-ROM drive after walking across a carpeted floor and just touching it I gave it a good zap. I was working in a small computer store at the time.
Re: Shocking!
These days you really need to try hard to kill a piece of kit with just static electricity, but a lot of these stories are set in the early days of computing where this was a serious concern. Especially proper zaps like a dusty leafblower can deliver!
Re: Shocking!
When I worked with testing and fixing electronic equipment as day job then static used to an issue. We'd work on rubber mats for protection and movement on those could generate quite a bit of static (sliding back the chair to change something in rack would generate a painful charge).
When have I done a bit of electronics at home static has never been an issue
Re: Shocking!
One of the things about damaged caused by static is that it doesn't always cause an instant failure.
The over-voltage caused by a static discharge can punch a tiny hole in the insulation between the layers of a chip that allows metal and other ions to migrate over time, and this eventually leads to the device failing as some unpredictable point in the future.
Re: Shocking!
I did once corrupt the firmware in a print server circuit board with static. Walked across an office carpet and took the board from a colleague and saw the spark jump as I reached out.
Reapplied firmware and all seemed well again. Guess that's why they call it flashing firmware...
ChopChop
Once had to do a site refresh at a remote office that was getting new servers, switches, UPS, the works.
But we also had to replace the little 12U rack with a full height one, so it was appropriately ordered and delivered, and waiting for us when we got there.
Problem was the network cable people had routed all the cables through a little hole in the top of the rack, and wired them into the back of the patch panel, meaning we simply couldn't get the patch panel out of the rack without ripping all the cables out and having to get it all re-terminated.
One trip the site workshop saw the application of a power saw to simply cut the top/rear of the rack out so the cable loom could be removed with the panel intact.
Luckily the new rack had a full size slot on top we could easily pass the patch panel through.
Re: ChopChop
"Problem was the network cable people had routed all the cables through a little hole in the top of the rack, and wired them into the back of the patch panel, meaning we simply couldn't get the patch panel out of the rack without ripping all the cables out and having to get it all re-terminated."
I'm guessing that simply walking off and demanding that the project manager "fetch the buggers back" to do the cabling job properly wasn't an option.
Re: ChopChop
Is it ever?
Re: ChopChop
Oh i remember those patch panels. The newer ones where you terminate the cables into a jack and just click them into the panel are a godsent.
Ouch!
Very hair-raising experience
I once had someone want to weld components to a printed circuit board of a stage light controller I had built and maintained for a student drama society. I suggested soldering was the better option. As she was doing some practice soldering on a little spare PCB, she kept confusing the terms, so when she next called the soldering iron a welding tool I couldn't resist going "KZZZEERT!" when she next put the soldering tip on the PCB. Apart from confusing the terms, she was very capable, and did an excellent job. I kept up the occasional "KZZZEERT!" until she got the terminology right.
Re: Ouch!
Only the BOFH's allowed to use KZERRRRRRT...
I'm hoping for a BOFH eppysode on this sunny friday...
Re: Ouch!
The PFY has also been known to apply the KZZEERRRT!
Could have been worse - by design
I had to clean out the computer that controlled a digital printer that could put an image on 12ft wide rolls of fabric. The manufacturers had placed the computer directly under the print head parking and cleaning point.
Just a quick manicure.
We had a guy that solved a desktop stability problem with nail trimmers.
See, the users in that department were slightly technical, and had figured out they could overclock the Pentium 100's they'd been issued by changing a jumper. Sure, their build times dropped, but the machines became a little flaky. You'd think that folks making unauthorized modifications to company hardware might keep it quiet, but not these folks. They openly admitted to overclocking and *whined* about their applications crashing.
On about the third or fourth trip over there where he found a machine behaving badly because they'd changed the clock speed, he noticed a pair of nail clippers on a desk across the way and was inspired.
Snip! Off came one of the pins in the jumper block, and with it, the ability for the motherboard to be clocked to 120mhz.
Re: Just a quick manicure.
Take this pint for taking me back to the era of turbo buttons and jumper overclocks.
Re: Just a quick manicure.
turbo button
Or as we knew them, Slow Down Buttons! That's what they were in reality, originally fitted because some (mainly games) software relied on the IBM PC 4.77 MHz clock speed.
Most-Inappropriate Computer Repair Tool
At one large company I teched at, I was warned by my supervisor about a particular department-project rep who ate erring techies for lunch. I did a few projects for her, she was reasonable, and we got along fine.
On one major office remodel/offices move/many-PCs replacement project she supervised, I was working on a PC which had a 5.25" floppy diskette drive and a 5.25" streaming backup tape drive. These were mounted on drive rails specific to the old computer, and which were not compatible with the new computer. Attempting to unscrew the screws holding the rails in place, I found I could not.
She saw me walk out of an office, look around, walk over, pick up a very large hammer, walk back into the office, heard WHANG-WHANG! ... WHANG-WHANG! , saw me walk out of the office, put the hammer back, and return to the office.
She didn't say a word. I guess she'd learnt she could trust me -- or was too flabbergasted to think of what to say.
(Icon for potential tool-misuse hazard.)
Re: Most-Inappropriate Computer Repair Tool
Engineer's solution to a problem: if it doesn't work, hit it with a hammer; if it still doesn't work, fetch a bigger hammer.
Re: Most-Inappropriate Computer Repair Tool
The skill of Engineering is knowing the correct size of hammer to use on the exact right spot.
iMacs...
Ex apple genius here....
The first aluminium iMacs, circa 2010, I think, had 3 very large capacity fans inside (power, Logic board, HDD), and if you're going to blow air out, you gotta suck it in.... So, have seen my share of dusty (and other flotsam and jetsom) Macs.
However, those Macs had a pice of polycarbonate infront on the screen, but with a gap that wasn't sealed.... this meant that air circulated here.... Match this with people that had ash trays infront of their macs, and you had customers coming in wondering why there was a yellow tinge to their display.
Cleaning it only made things worse: we'd have to charge them for a new polycarbonate
Shocking!
It's a miracle the server survived, dust + leafblower = static electricity!