Watt's the worst thing you can do to a datacenter? Failing to RTFM, electrically
- Reference: 1694417595
- News link: https://www.theregister.co.uk/2023/09/11/who_me/
- Source link:
This week meet "Nikolai" who spent some time in the mid-1980s working on some fancy bespoke communications solutions. On the project in question Nikolai was employing two very expensive microprocessor emulators – because you need one at each end to demonstrate a comms solution, of course.
Underlining the rarity and expense of these emulators, Nikolai tells us there were only three of them in the country at the time, and he had two thirds of the supply. He doesn't spell out what the other one was doing – probably not comms, we're guessing.
[1]
The project had not gone well and had been delayed several times by the time this story starts, when Nikolai's client delivered an ultimatum: show us a demo or the deal would be cancelled.
[2]
[3]
The day before the scheduled demo, one of the emulators began behaving strangely – "not obeying its own instruction set at all," Nikolai lamented.
[4]Sure, give the new kid and his MCSE power over the AS/400. What could possibly go wrong?
[5]Polishing off a printer with a flourish revealed not to be best practice
[6]Lesson 1: Keep your mind on the ... why aren't the servers making any noise?
[7]The price of freedom turned out to be an afternoon of tech panic
The search began to discover the cause of the problem, and a potential culprit was found: "the 5V rail on the emulator was checked and found to be only 2.5V" according to Nikolai. What could cause such a strange anomaly? Nikolai vaguely recalled a colleague fiddling with a power supply earlier in the day, and investigated.
Sure enough, the attached power supply was set to 2.5. Why? A prank maybe? Not a very funny one. Nikolai cranked the power supply back up to 5.
At which point the magic blue smoke escaped from the emulator. One of the two emulators Nikolai had. Of the three in the country.
[8]
It turned out that the setting on the power supply was not at fault at all. A broken wire in one of the cables that connected two boards within the emulator was the culprit.
So why had Nikolai's colleague fiddled with the power supply? Well – you'll laugh – it turned out he'd switched the display on the power supply from volts to amps, and Nikolai hadn't noticed. The power supply had been correctly outputting 5V at 2.5 amps, or 12.5W. By doubling the amps, Nikolai had sent 25W to the emulator, blowing its little mind.
Thankfully, there was that third emulator in the country, which was very hastily obtained. Nikolai did what he could with one emulator overnight and, when the spare arrived, all worked correctly. And Nikolai learned a lesson about reading the units on a power supply.
[9]
If you've ever jeopardized a major project with a moment's inattention, we want to hear from you. Send [10]an email to Who Me? detailing your misadventure, and we'll make that moment last forever.
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZP7lQID4kp@FpBt45W0@5gAAANE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZP7lQID4kp@FpBt45W0@5gAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZP7lQID4kp@FpBt45W0@5gAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2023/09/04/who_me/
[5] https://www.theregister.com/2023/08/28/who_me/
[6] https://www.theregister.com/2023/08/21/who_me/
[7] https://www.theregister.com/2023/08/14/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZP7lQID4kp@FpBt45W0@5gAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/systems&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZP7lQID4kp@FpBt45W0@5gAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: But surely
You just couldn't resist could you?
Re: But surely
I'm laughing so hard it hertz.
Re: But surely
Inducing spasm capacity?
Re: But surely
There is so much potential for punnery in this one
Re: But surely
Watt the hell are you on about?
Re: But surely
I'm currently feeling wired...
Re: But surely
I'm currently feeling wired ...
Don't worry. You're fine. Just solder on. You still have some spark left.
Re: But surely
It's worthy of induction into the El Reg Punnery Hall of Fame.
The means by which that recognition is triggered is, of course, the Hall effect.
Re: But surely
But not when the capacity plays a role.
Re: But surely
I feel this pun may have volted the fence...
Check the power supply
A friend told me that she had once been testing a dozen brand new, very expensive, monitors. They were the first batch off the production line and we needed to check they had been correctly built. After the tests were done, she sent them back to the manufacturing plant in Scotland for repacking and distribution. They decided to do their own basic test that the displays did still work so they could be sold on.
Plugged the first one in, and it didn't work. Plugged the second one in, and it didn't work ... All the way up to 12.
At which point they finally checked the specs and found that they had used 240V power instead of the 120V required by this model. And that they had managed to wreck all of them.
Re: Check the power supply
Gotta love it when people don't stop and wonder if something else might be going on...
I suggest designing all electrical appliances to ensure they fail spectacularly when you plug in the wrong power supply.
Re: Check the power supply
I'm sure that if they had just used a slightly more expensive power supply that supports the whole 100-240 voltage range, this wouldn't have happen.
Region locking bites another ass in the ass.
Re: Check the power supply
Personally, after the first 2 didnt work, I'd have stopped testing until I'd passed the buck up the tree to say "Something isnt right here. They say everything's fine, but the first 2 I've tested failed. What do we do?"
To do 12 in a row - Either someone was very stupid (so why on Earth where you letting them test new kit!), very afraid of asking questions (management fail), or it's a rather subtle version of sabotage. Since I dont think there are any more monitors being manufactured in Scotland, I guess the sabotage succeeded!
Not on expensive kit but entirely wilfully (and in a very distant past), when some electronic circuitry didn't work as intended, it could only mean one thing: it didn't have enough juice. So we ramped up the amps. And if one bench power supply (usually around 30V / 5A max) wasn't enough to make the magic smoke visible, we would stack them. And, before you try this at home, be reminded: 90V already delivers a slightly sub-pleasant sensation through the fingers.
I I be a-goin there, I be-n't start from here
To me, part of the blame lies with the manufacturer of said very expensive kit.
Anything of that nature that requires external power should have at least basic safety 'crowbars' fitted. It's not difficult or expensive.
Re: I I be a-goin there, I be-n't start from here
But also why the hell have the option for double the power straight into a device that on the face of it, not only doesn't require the ability to switch to it, nor had any safety components to stop it self-destructing.
Even if it was an "off the shelf" PSU across different models - big sticker over the switch with a warning not to adjust it.
Re: I I be a-goin there, I be-n't start from here
"But also why the hell have the option for double the power"
Never used a bench-top power supply, I take it. Handy bits of kit ... but capable of getting a neophyte into all kinds of trouble.
Re: I I be a-goin there, I be-n't start from here
A proper bench-top power supply should be able to display both volts and amps at the same time .
I just tested a hand-built assembly over the weekend. The little bench supply I have on hand can do 30V, 10A, but is set to 24V and I get to read the live amps to see actual load taking place (0.1A on one circuit, 0.24A each on a couple others).
Re: I I be a-goin there, I be-n't start from here
I barely have a working knowledge of electricity, but I always thought amps were "drawn" rather than pushed. I've been told (for example) that adaptors need to match voltage, and need to have an equal or higher amp value than the device you're powering.
Re: I I be a-goin there, I be-n't start from here
"I barely have a working knowledge of electricity, but I always thought amps were "drawn" rather than pushed."
Traditionally, that's entirely correct.
But this seems to have been kit which could be set to deliver a certain current. So when it was set to deliver a higher current it probably increased the voltage (which, assuming the resistance of the load remains the same, will also increase the amps drawn) until the selected higher current was delivered. The resulting higher power then destroyed the kit (which as someone else pointed out, could have done with a bit more protection).
When I was at secondary school one of the teachers built a sizeable analogue synthesiser - and then one day connected the power supply the wrong way round leading to loss of a lot of magic smoke :( . So now if I build something significant (rarely) I try to remember that and provide adequate protection.
Re: I I be a-goin there, I be-n't start from here
Some bench supplies can operate either in constant voltage or constant current mode. In CV you set the voltage and the load will pull whatever current it requires (I=V/R). In CC you set the current and the PSU will push that current through the load and the PSU voltage will adjust to whatever is required by the load (V=IR). CC mode is rarely used - I only used it once in my career for biasing some weird microwave diodes. Most bench supplies have a current-limit setting that prevents the sort of problem seen by today's protagonist. It's good practice to set the current limit to a bit above what you expect the system to draw, especially if you're dicking about with it.
In this example the PSU was in CV mode and the supply was set at 5V but the display was showing the current drawn by the system, not the voltage applied. When the voltage was increased the current drawn by the system increased (I=V/R), so it looked like the voltage was being turned up. Until the kit blew up.
Re: I I be a-goin there, I be-n't start from here
Exactly this. Note that it also means that at the magic smoke release point the voltage was significantly higher than the 5 volts claimed and thus the power calculation (V*I) that the article makes is wrong - it would've been a lot more than 25W
Re: I I be a-goin there, I be-n't start from here
"To me, part of the blame lies with the manufacturer of said very expensive kit."
Yes, and no. Remember, thus was the mid- 1980s. The assumption was that only people with clues would have access to such kit. This is the time when many stories of switches set to 120V were wired into 240V mains, with a resulting release of the magic smoke. It's also the era when we started seeing units that would automagically adjust for most household voltages world-wide.
Mid 90s. Early in my career as a chemical engineer, so my memory of details is hazy. Had a job to do monitoring the organic chemical emissions to air of some plant or other. Had a whizzy bit of kit to log the emissions: a flame ionisation detector, which would output an electrical signal, and a datalogger in the shape of a Psion Organiser 2 - the big chunky calculator one, not the cute microlaptop Organiser 3. There was, IIRC, some kind of interface thingy on the top, and we were supplied some cableage to connect the two devices. Again, IIRC one of the plugs was an old 9 pin D plug like an Atari joystick. And the same connector was used for a power supply...
Long story short, it was physically possibly to connect that up incorrectly. Which of course at some point I did. And at that point the magic blue smoke started coming out of the Organiser. So I didn't do any monitoring that day. The organiser and the cables went back to the supplier, and were returned, free of charge I think, with some of the pins snapped off one of the plugs and some of the holes glued up on the socket... making plugging the wrong thing in there physically impossible. An elegant solution, and just a shame they hadn't foreseen the possibility before handing the thing to a wet-behind-the-ears numpty like myself.
Plus ca change....
>> And the same connector was used for a power supply...
Oddly enough the sea trials of a very expensive piece of naval hardware were interrupted because someone saw a 25 pin D connector and assumed that it would be OK to plug in a serial cable...
I guess its one of the hazards of having a multipurpose connector - it can have many purposes, one of them inevitably being "finding out who actually read the fine manual before connecting random cable A to connector B"
Re: Plus ca change....
If I have to use cheap connectors, I prefer automotive ones like the Deutsch DT series. At least you can specify keyways to prevent connector mismatching.
I just built a box with (1) 8-position and (3) 12-positions (A, B, C). As long as I wire up the other side correctly, you can't swap them, so no worries.
Re: Plus ca change....
once had a job to supply a bunch of remote sites with a router and a modem for backup.
Both the same tubular power connector... unfortunately one was 9v and the other 18v... many 'dead on arrivals'... rapid deployment of colour-coded WARNING labels for plugs and sockets
(luckily it was the el-cheapo modem they were frying, so they had service while we rushed them YET ANOTHER modem)
Silly Mistakes
Silly thing to do. I can't be the only person who could never use a multi-meter on an incorrect setting....... If only.....
Yesterday, I tried to measure the voltage on a small battery and wondered why it got rather hot. Fortunately, no-one was watching so it didn't happen.
Not involved-->
Re: Silly Mistakes
I never got a sensible reading when I (accidentally) tried to measure the resistance between live and neutral of my mains supply. The end result was similar to that of a University colleague who had the bright idea of getting a ~500v supply by putting two adjacent 240v sockets in series.
I'm sure I have posted this before but when working at an about to open fruit store, we discovered the hard way that not all TVs have universal power supplies. It was for a window display and had been shipped from the US *with* a power converter but was plugged directly in to 230V. The most remarkable thing is that it lasted nearly a whole day before failing. A replacement was sent overnight from another store at great expense.
Doubling the amps?
> By doubling the amps, Nikolai had sent 25W to the emulator, blowing its little mind.
Huh? This makes no sense. The ammeter is measuring the current drawn by the device, and while the PSU doubtless has a maximum limit (possibly even configurable on a bench PSU) you can't force the device to draw more current than it wants to by changing something in the PSU (notwithstanding a PSU in CC mode, but (a) if it had been set to CC mode it would have almost certainly have blown the device up already, and (b) in CC mode you're still changing the voltage (albeit automatically), not the current.)
Presumably what actually happened is, not realising the display was showing the current draw rather than voltage set, Nikolai turned up the /voltage/.
And yeah, I remember working with wretched ICE CPU emulators back in the day. The things would flake out at the drop of a hat, and more time was spent wiggling hundreds-of-pin connectors to try and make the damn things work than doing any actual development. (The ones I used to work with came in stacks of 3 or 4 instrument units each connected by thick bus cables - I was always told, possibly apocryphally, that this was so they could be shipped as separate units each of which would fit down the hatch of a submarine. Why you would want a CPU emulator in a submarine is left as an exercise for the more sceptical than I was in those days...)
Re: Doubling the amps?
P=(I^2)R, so doubling the current would be four time the power...
To double the current requires (usually, in circuits) more voltage.
If you've ever jeopardized a major project with a moment's inattention
A moment's inattention is all it takes for the majority of issues in my experience. That's why you should have proper review procedures.
Even Agile likes "pair programming" for instant review.
Once, many years back, had a colleague set a desktop PC PSU from 230V to 110V to see what would happen. "What happened" was the PSU emitting lots of sparks and magic blue smoke when I plugged in the power lead, in close proximity to my head. Replacement PSU and the PC was fine, and from that point on I was always paranoid about any power supply with switchable voltage...
But surely
An ohm is the place an amp lives?