The power of Bill compels you: A server room possessed by a Microsoft-hating, Linux-loving Demon
- Reference: 1599808504
- News link: https://www.theregister.co.uk/2020/09/11/on_call/
- Source link:
The mystery takes place in the early years of this century, when "David" (which is not his name) was toiling away as a hardware engineer for a third-party maintainer.
His story begins with a jaunt to a client in the city of London and a seemingly innocuous request to drop off an IBM ServeRAID CD requested by a customer.
"Apparently," he told us, "their Exchange server wouldn't boot past the POST screen. It would hang with a blinking cursor in the top left-hand corner."
If nothing else, a reminder to cloud refuseniks that on-premises email has its downsides.
The theory was that the RAID configuration was somehow borked, but booting the CD (running Linux) showed that all was well. It must therefore be Windows that had gotten its knickers in a twist.
"A Windows Server CD was procured, and we attempted to boot it to see if it could be recovered," recalled David.
Alas no. Things seemed to boot up OK from the CD but then hung in the same place.
With things starting to sound a bit like a popular [2]Bernard Cribbins song from the 1960s , David began working through the hardware to find the problem. It was obviously Windows accessing some failed RAM or other bit of faulty chippery not used by Linux, right?
"Over the next two days, I replaced every part at least twice," he sighed. "I called every support engineer in the company, then every other engineer hoping someone had an idea and I lost 2kg stressing."
Still Windows refused to boot. The scenario was simple – if there was a valid Windows partition on the RAID, there would be no booting. If there wasn't a partition on the RAID, or if it was on a non-RAID SCSI controller, Windows would boot. "If you started from a blank RAID group, you could load Windows until the first boot off the new partition when it would hang," added David.
Linux was fine, and would boot up in any configuration. Windows, however… not so good.
The hardware passed every diagnostic test David threw at it, but nothing would persuade Windows to behave.
By the end of the second day, David decided to throw in the towel. He returned to the workshop, built a duplicate system (he certainly had enough hardware now), configured the RAID and loaded up Windows. Everything worked.
Everything was still working the following morning when he disconnected it and took the newly built hardware to the customer.
Upon his arrival he was met by the customer's IT manager, who had come up with a theory of his own: "The server had been possessed by a Microsoft hating demon and it needed an exorcism."
Laughing at such a ridiculous suggestion, David popped the new server into the rack and fired it up.
"My completely new [and fully tested] server does exactly the same thing," he told us.
Fortunately for the IT staff preparing to rush out to find the computing equivalent of [3]Father Merrin , the penny dropped for David.
"They had never heard of an [4]earth loop and didn't believe me until I took their old server out of the server room and put it on a desk in the office where it booted without a problem."
"The first time I saw a problem like this," he told us, "we eventually found that the earth connection inside the power supply was corroded and causing high resistance between 0V and earth, which means the potential of the 0V rail changed as the load on the system changed. Something at that point in the boot process was drawing enough current to trigger a fault."
"I will say that in nearly 40 years of being a hardware maintenance engineer," he added, "I have only seen this sort of thing twice and if I hadn't seen the first one, I would probably be calling the exorcist myself."
A swift call was made to get an electrician on site, and Windows server (NT4, David reckoned) happiness was restored.
Ever taken on your own Microsoft (or Linux) demons, only to find some dodgy wiring behind the mask? Send an email to [5]On Call to share your moment of enlightenment. ®
Get our [6]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://www.youtube.com/watch?v=Ge_4SlJWfl0
[3] https://www.youtube.com/watch?v=M6GIbICDY0s&ab_channel=LindaBlairFan
[4] https://www.emcstandards.co.uk/earth-loops-ground-loops-and-hum-loops
[5] mailto:oncall@theregister.com
[6] https://whitepapers.theregister.com/
Re: This is why...
What a shocking thing to say!
Re: This is why...
That’s a (wire-)wrap.
I once destroyed the internet.
It's ok though, I had a backup on a floppy disk stuck to the side of the case by a magnet.
-Me.
Re: I once destroyed the internet.
Manny moons ago, I witnessed a field engineer open the back of a piece of equipment, pull the diagnostic floppy (8", just to date myself) off the inside of the door where it was affixed with a magnet ... and the fucking thing still worked! Observing my surprise, he just shrugged and said "I know. I don't get it either. They did it this way for years before I got here. I don't ask questions, I just go by their playbook and collect my pay." He claimed to have seen several tens of these things, and the disk was only dead once.
Re: I once destroyed the internet.
It could be that in order to re-magnetise the disk surface, the magnet needs to be moved tangentially to the disk's surface. Moving a magnet perpendicular to the disk surface, as you would do when sticking the disk to a metal surface may have a lot less of an effect than expected.
From my dim and distant memory of A-level physics, it's the change in a magnetic field that will induce the electric current, and thus the magnetic field in the disk surface. That's why putting magnetic tapes anywhere near hi-fi speakers is a bad idea.
Not met a demon
But I used to have an IT Boss who had a pathological hatred of all things made by the beast of Redmond. This was before Red Hat and Ubuntu so he ran an Ultra 5 with CDE. Email - pine was what he used. I do still miss my Sun boxes though.
Re: Not met a demon
I'm fairly certain your Sun boxen were absolutely chock full of miscellaneous daemons, of varying degrees of power.
Re: Not met a demon
I used an Ultra 1 with Solaris for many years as my desktop at my first IT job (university department). Ran like a dream. I had some horrible Frankenstein windowing environment using CDE and Window Maker, but it was stable and by the time I left, I'd been logged in from October to March (just locking the screen at night) without any issues. Getting Windows to stay up that long without a reboot (let alone logged in) was pretty much impossible...
Re: Not met a demon
I ran a SPARC 2 as a desktop for several years. Lovely machine for the time, stayed with SunOS 4 though. Eventually though I persuaded the boss's boss to spring for a PC, that then ran Linux.
Re: Not met a demon
Getting Windows to stay up that long without a reboot (let alone logged in) was pretty much impossible...
Of course it was impossible, there was the same bit of code, that ran a 16 bit centi-second timer, in all versions of windows. It rolled over at 37.9259259 days and BlueScreened all desktops and servers. It took over 3 years of emails to Microsoft from various engineers at the BBC, who explained the problem before Microsoft even acknowledged that this was a serious bug. I'm sure hundreds of other corporate users with more than 50 copies of windows would have noticed the same problem and reported it. BorkZillas's response was always make "sure you have regular downtime planned". The trouble is that many users have 24/7 operations and can't afford to keep rebooting servers just for the fun of it, especially since most servers couldn't even make it to 31 days of use because of lack of garbage cleaning in other areas of the OS.
I haven't used Windows for more than 1 session every 4 months to update my Garmin, in over 20 years. Do you still have to run defrag every few months of continuous use?
Re: Not met a demon
Not sure why this was downvoted - I found it an interesting and quite believable story. Certainly my 486DX100 Redhat server - in 2000! - had more continuous uptime than every Windows machine in the house combined. My 9 months record was only stopped due to a power outage.
Re: Not met a demon
"Getting Windows to stay up that long without a reboot (let alone logged in) was pretty much impossible..."
Why would you wanna? thats just cruel
Re: Not met a demon
sun were Ok until Oracle got their grubby hands on it, everything went downhill from then on, apart from the cost of support and licensing which went UP hill!
Trumpet
Did David hear any trumpets?
Because I am sure I remember something about trumpets and demons walking the Earth.
Re: Trumpet
Isn't a Trumpet the spawn of Trump?
Re: Trumpet
A Trumpette certainly is.
Re: Trumpet
I thought a Trumpette was one of his cheerleaders.
Re: Trumpet
If we're talking Windows, does trumpet winsock count?
Power!
Once had a similar issue. Desktop wouldn't boot up properly, worked at the location for a while, and finally told the user I needed to take it to my office for deeper inspection. Damn thing booted while sitting on our workbench. Moved it back, and no booty! Back to the workbench, boots fine! Turned out to be either a power cable, or a dodgy power strip, because I took the power cable from the work bench, and went straight to the mains and bypassed the other cables. I took the cable and powerstrip and chopped the plugs off and binned them. You don't know how many times we've binned cables, only to have the cleaners and facilities guys bring them back and ask why? Now we destroy them.
Re: Power!
Cables of any sort!
Dodgy RJ45 cables are my pet hate and I always cut them in half, otherwise someone will always drag them out of the bin!
Re: Power!
And depending on where it's broken and where it's chopped, you may even have a chance of one of them working if someone still decides to resurrect them and put ends on your two halves!
Re: Power!
I used to do that !
why throw 60m of cable away when you could lose only 30?
..might even get a useable 15 out of it too
Re: Power!
Cat5e cable turns out to be just right for snaking out limescale deposits in the water channel in the rim of my toilet bowl. With the useful side effect from association with toilets that I no longer feel any temptation to try resurrecting the other 19m from the unreliable cable I butchered!
Re: Power!
I (like most commenters here, I expect) have a hefty box of used Cat5e lying around. I'll have to try this. Thanks for the tip!
Anyone else have unusual uses for Cat5e?
Re: Power!
Faulty keyboards and mice, where the fault is not immediately obvious (e.g. missing/broken keys) get the same treatment SNIP!!
Re: Power!
I don't destroy them, I have them scrapped out of inventory and marked accordingly, then keep them locked up where the proles can't access them. One never knows when one might need to fabricobble something non-standard ... and it's always easier to grab a scrapped cable or six than it is to request stuff from stores.
Re: Power!
Jake, your post is MUCH more amusing if you replace "cables" with "cleaners". The thought of you dismembering them, locking the bits in a cabinet, & pulling out random body parts to make a "FrankenWidget" just made me LMAO.
*Hands you a pint*
Drink up, it's Friday!
Re: Power!
> I don't destroy them, ...
Arrrrgh! Absolutely DO destroy them, unless you like hardware Heisenbugs. I nearly had a fist fight with some idiot coworker about this. From then on I chopped those cables into very short pieces.
Re: Power!
Rationale: cables that have obvious failures (a sharp bent or a loose end) tend to have non-obvious problems as well. I learned that the hard way, wasting enough time I could have bought twenty replacement cables and still save money.
If you have a proper cable tester (and the time), repair and re-use as you like.
Re: Power!
I always make sure cables are cut before scrapping.
Back in the days before WEEE when you could still chuck PCs in the general rubbish skip we used to remove the HDs first and dispose of them more securely (as in meet with a big hammer). We couldn't believe it when somebody not only removed a couple of machines from the skip but put in a complaint that, "They don't work, will you fix them?"
Re: Power!
"...You don't know how many times we've binned cables, only to have the cleaners and facilities guys bring them back and ask why? Now we destroy them."
Do you mean the cables or the cleaners? The first is just good sense. The second is cathartic. =-)p
Re: Power!
happens all the time people diving in to the electrical waste skip! Best one or worst depending on how you see it, I've seen was a boffin at the lab I used to work at decided to use a vacuum pump that had failed its PAT test but was still on the bench with a FAILED PAT TEST DO NOT USE sign on it (really should have been chucked in electrical waste but hey ho). He was a bit of a menace and always buggering around with stuff he shouldn't and not following correct procedures etc, a total lose canon. Anyway bloody thing caught fire, its the only actual proper incident that I witnessed at the lab in 20 years! Had a few false alarms but because its a lab anything that trips the house alarm automatically calls out the fire brigade. The same feckwit was also caught disposing of the contents of his bio reactors down the storm drain in the front yard of the lab!
Re: Power!
I learned early in my EE career to be suspicious of connectors (of any kind); they are one of the most likely failure points. Minimize them in your designs and suspect them first during debug.
I'm not sure I would have spotted the problem as quickly as "Dave" did. Well done, have a [virtual] pint!
No FreeBSD daemon?
Amazed you didn't reference https://rmitz.org/freebsd.daemon.html
Re: No FreeBSD daemon?
I'm waiting for the curtain-twitching hand-wringers & namby-pambys to notice the near 45 year old [1]Beastie [0] and raise a stink ... It's going to happen, it's only a matter of time.
[0] Where do the years go ...
[1] https://en.wikipedia.org/wiki/BSD_Daemon
Re: No FreeBSD daemon?
That reminded me of a time, yonks ago, when I ended up having supper with three guys, who, at that time unbeknown to me, were some Evangelical Christians. The discussion arrived at music and, inevitably, at my favoured hard rock and metal. Oh lord!
They came up with satanic backwards messages in rock songs and I, still, thought it was all fun and stuff. Until I started to realise they were all serious and moments before lynching me. Obviously, they didn't. But I made a rather quick escape.
Re: No FreeBSD daemon?
Reminds me of my time in Silicon Valley. I described a bunch of code as "evil" (thousands of lines of spaghetti C) only to be lectured by an angry American colleague that Satan is evil and to call code evil was "offensive" to his beliefs.
Re: No FreeBSD daemon?
Lynching you?
Crucifixion, more likely!
"Always look on the bright side of life..."
Re: No FreeBSD daemon? @Evil Auditor
Do what happens if you play Country music backward?
You get your house back, your wife back and your chickens start laying :)
The frisson of Y2K
Not earth loop related but similar....
Back in 1999 (remember then? heady days, full of optimism..) we ran multiple tests on a duplicate rig to test everything prior to the 'big day' -- some interesting effects on licence servers winding dates back and forth but no problem.
So, on the last working day before Christmas we started our regular annual shutdown (only real time the server room could be cleaned thoroughly and other maintenance done).
Come the first working day of 2000, the IT team were in early to turn everything back on ready to welcome the joyous hordes back to their labours. Everything started up apart from the Windows NT file & print server. It refused to boot. No system found.
Start of controlled panic. Several retries - yes the disc lamp flashed, yes there was a comforting sound if you put your ear to the case. Get the backups from the safe. Get the recovery boot CD. Start the preparations to do a controlled restore when, about 45 minutes into the story, in walked the youngest member of the team who popped out a floppy from the drive saying "I was looking for that!". Two things happened then: one - the system booted OK :-) and two - he was persuaded that he was on tea & coffee making duties for the whole week to calm down his colleagues.
After that, a scheduled downtime was set to disable boot from floppy in the BIOS of all Windows servers -- especially as our boot images were all on CD anyway.
Sparky
HP (IIRC) laptop power supplies had an issue a few (probably 8-10) years ago... the clover-leaf plug that came with them had a manufacturing defect which left an air bubble between live and ground. eventually the plastic deteriorated and it sparked across. Left quite a good flash burn in the socket, along with a very surprised customer!
Thankfully replacing the lead was all that was required, but it made quite an interesting root-cause investigation. Affected a sizeable batch too, but took a while to emerge.
Windows boot power draw?
So Windows takes more power to boot than Linux? Why is this?
Genuine question.
Re: Windows boot power draw?
It's partly due to the required Nyarlathotep invocation.
Re: Windows boot power draw?
It does initialisation stuff in parallel, whereas before systemd Linux didn't as much.
There's also the fact that Windows loads about 1000 drivers and services whether you need them or not, whereas Linux will only load what it needs to do the job it was configured for.
Re: Windows boot power draw?
Not more power, different timing when bringing up the various hardware subsystems..
Re: Windows boot power draw?
I once worked at a place with a machine that had an underpowered PSU and five hard drives (don't ask why...). It would only boot if the drives were connected to particular SATA ports on the motherboard. Theory was that this made them initialise in such a way that it staggered the peak power draw. Funnily enough, the PSU in that one eventually blew completely...
Visions of foriegn nightmares
Twice in the 80s for some poor Vaxen where a stake in the ground was the building earth and it needed watering. First time took a week to hunt down, second was deja vu.
the demon earth
Many years ago, in India, we had mail appliances fail every 2 months.
Yep, mobo burned, replacement was painful, because of customs, and it happened so many times, vs. many other sites where the same *never* happened once.
At one point, I asked the dudes where they plugged the double PSUs setup. Of course, to 2 different power sources like is best practices.
After asking them to plug both PSUs to the same power source, the issue never re-occurred.
Yep, the 2 earth from the 2 power source were very different and it repeatedly killed the mobos *every* single time.
In the same way, I had great flashes when working at one building, years ago, between my hand and the coffee machine, even after touching the other coffee machine 2 meters away. Yep, 2 connected buildings, and the earth was not connected properly, and have 100 V give or take voltage difference.
Earth is a bitch.
This is why...
...hardware engineers should always be well grounded.