News: 1593411304

"I also survived not being provided with enough UPSes to cover the load,"

Did he survive not calculating the required load first, before destroying a UPS by overloading it? Surely the first thing you do when asked to move a large number of systems is talk to the facilities group to make sure that adequate power & cooling is available. Before plugging things in.

“ Surely the first thing you do when asked to move a large number of systems is talk to the facilities group to make sure that adequate power & cooling is available. Before plugging things in.”

We called them a site survey, normally any company doing work for another will look for any opportunity to up the costs of the job, I.e is there enough room in the rack, Ethernet ports, fibre, this server series has a max draw of 500w at 5amps you need a bigger ups, despite the server ordered having max 200w power supply and nothing inside it that could consume all 200w. Seen it many times, been overruled many times and watched, usually government Departments, projects burn money they didn’t need to and also add chunks of time unnecessarily, often keeping the pm contractors on for months extra time but there is no link there.

Shouldn't an overloaded power supply shut down safely, or at worst blow a fuse, rather than bursting into flames?

Yes, but UPSs have always had their own rules.

Like shutting off when the remaining minutes countdown reaches 29.

Like shutting off when the remaining minutes countdown reaches 29.

In my experience, once it has beeped to indicate that it's now active a UPS generally shuts down in the time it takes to utter the first four words of the sentence "I'm glad I've got UPS"

UPS vans were brown...

Now I know it. Brown hides all the [redacted] in your pants when things go TITSUP.

Mines the one with a roll of soft tissue in the pocket just in case things get a bit short.

Following a planned power off for electrical work, 06fartooearly on a Monday, I come in, fire up all the network kit, lights come on in the right order, everything goes green as it should, lovely. Go to the server cabinet, start them all up, all the usual whirry noises, fans, lights, etc. Fantastic. Then my ears pick up a regular beeping noise, one not normally heard. WTF?? While my half awake brain processes this, all the servers shut down and there is silence again. Mild panic ensues. Press power again, nothing. Beeping continues. Finally look at UPS at the bottom of the rack....oh-oh, red lights all round. No power. Phone electrician on call...…

After the inspection one of the RCDs hadn't been replaced correctly, so the entire rack had no power. All the servers fired up on UPS power, which then promptly shut them all down gracefully before it ran out of battery power.

Lesson learned, check everything for red lights. Even the stuff right at the bottom...….

No, because I've also gone for ones with overload protection...

Yes because of an internal fault in one of our UPS's, it tripped the utility power and went to battery.

We got alerts immediatly via email that utility power was down. It was connected to a fully populated IBM Blade Centre I went to check as it covered 1 of the 4 2KW powersupplies and if it went down we would have no power redundancy on the left hand power domain (Which also powered internal fibre switches and management).

I went and stabbed the button to put power back on as I couldn't see anything wrong.

Sparks immediatly came flying out and the 16AMP C19 cable melted before it re-tripped.

Never been so scared in my life (Circuit had 50AMP fuse).

+ 1 Was wondering if someone might mention a simple thing such as overload protection.

I used to work for a CCTV distributor, as well as selling kit to installers we provided pre & post-sales tech support, which meant we had to have one of everything in our demo suite.

We kept warning the powers that be that we were overloading the available circuits, but the PHBs kept ignoring us.

That is until the advent of networked CCTV with a need for beefy dual Xeon based servers to process dozens of HD streams.

We managed to blag a server and SAN off one of our vendors and we made space in one of the racks, slid them into place, plugged the power supplies for server and SAN the into the power distribution unit, turned it on and got post, then the SAN spun up and there was a very big bang and all the lights went out.

Not just our lights, as we went outside to see where the acrid smoke was coming from we saw all the trading estate was out, and that smoke, it was from a now deceased sub-station.

Yep, we'd not just overloaded our circuits, that Dell MD1000 array was the straw that broke the sub-station's back.

Electricity Northwest brought a mobile generator round a couple of hours later to get the estate back up & running, then they delivered a much bigger substation a couple of weeks later, which we managed to knock out the next year, but that wasn't completely our fault, the company next door had a new machine delivered that sucked way more power than our entire building, no idea what the machine did, but the company only made printer ink and laser toner...

Elec-trickery isn't the only utility that can get knocked out by industrial kit. Some years back a dyer had a new gas-fired boiler installed and until they got their own supply they connected it to the domestic supply main. Then fired it up the first time in the middle of winter. The initial surge took the pressure down so low that pilot lights went out. It took a whiles for the gas engineers to work out why the pressure was so low in the mains.

a new machine delivered that sucked way more power than our entire building, no idea what the machine did, but the company only made printer ink and laser toner

They doubtless needed a very high-power machine to count their profits.

Many years ago I was debugging a very expensive bit of electronic kit with a few design issues on one particular board. eg under-specced ceramic wirewound resistor that gave an instant blister burn when lightly touched, and could melt its solder attachment to board. All hinting that a hefty power supply was lurking behind the scenes. Cue a more senior work colleague rushing through a revision and with a flourish hits the "on" button. My slow brain was registering that such haste was possibly not appropriate. The lights appeared to come up, but a few seconds later the magic smoke fairy emerges from a transistor can: actually probably more than one fairy. Although I was the first to observe it, the words of alarm froze in the brain. The only "f"-word was "aphasia".

Back in my student days (in nineteen-hundred-and-frozen-to-death) I earned part-time cash doing TV repairs. The workshop I was normally in was quite basic but, as long as you remembered not to touch any metal without first checking, it all worked well (and no accidents). I was loaned out to another shop for a few days, as their regular repair tech was on holiday. All went well until, a couple days in, I needed to check it the line driver was working on a TV. This was back in the days of valves and the usual check for the line-drive was to draw a spark from the cap of the HT valve using an insulated screwdriver - a 1/4" spark told you all was well. Unfortunately, this workshop had all its kit running off an isolating transformer and ELCB. The small current being drawn for the spark test (which passed through me to earth, totally pain and harm free) was enough to trip the circuit and plunge the whole shop into darkness. Apparently, they had a separate circuit that bypassed the isolated system when needed, but nobody thought to let me know.

Fortunately, nothing permanent and all back running ten minutes later. Their normal tech, it turned out, didn't know about the quick line-drive test and rigged up an AVO each time he needed to check; effective but we only did that if we needed to know the actual voltage for further diagnostics.

That is a sure sign of insufficient planning.

I'm not an admin, but I'm pretty sure that these days, when you're planning to install/move several servers, the checklist includes enough power for each and UPS for all and you don't go through with the install/move until you have those boxes checked.

On the other hand, as always, things like that have had to happen in order for today's admins to have that complete checklist to sign off on. Because there are very few humans who are capable of thinking everything through and envisioning all possible issues. We learn to plan because we've hit a snag, or witnessed a UPS go bang.

As the saying goes:

"A wise man learns from his mistakes, but a truly wise man also learns from other peoples..."

Got the soot stained t-shirt...

In this instance it was a series of 1000w output UPS's with each server connected to two through redundant PSU's running in parallel rather than one hot, one cold.. Apart from that one new server.. Which wasn't the 420watt PSU as the rest but a shiny pair of platinum rated 750watt PSU's.

It all ran fine until a power outage whilst I was in the server room moving other kit around. Lights go out, aircon dies, UPS goes bang with smoke, fire and flame (mercifully brief but enough for it not to be as dark as it should have been).

Cue panic to shut down various other servers before the other UPS died from its rapidly depleting power reserve (when you can see the leds dropping before your very eyes you know you have issues).

Small mercy that it didn't take the DB server with it, just the staff proxy server so no great loss.

"UPS goes bang with smoke, fire and flame (mercifully brief but enough for it not to be as dark as it should have been)."

Oooh, a UPS with built in emergency lighting?

If the light is lit, it's definitely an emergency!

My favourite ever support call to Dell: "The PSU on this server's failed, can you send a replacement?"

"Sure, we just need to run some diagnostics and collect some logs...."

"Don't need that, just send a new PSU"

"Well how can you be sure it's the PSU without collecting the logs?"

"Cos when I connect a power cable blue sparks and smoke come out of it"

"....OK, I'll just arrange a replacement now...."

At the start of this century I was a part of a team developing a highly specialised network simulator for a customer. The network that the simulator supported also included RF (SATCOM) components, and the simulator was designed to calculate the intermodulation products that all of the various RF carriers generated. The simulator was actually quite a beast; written in Java with a lot of work put in to optimise it, but with some settings you could more-or-less see the JVM (and the PC it was running on) scream.

The customer originally was planning on installing the simulator on a desktop machine running a mid-range single-core Intel processor. However when they saw the simulator running in the lab they decided to upgrade the target platform to use a dual-processor SMP motherboard with higher-specification Intel processors. This was 100% a customer decision, we did not recommend the upgrade, or (to be honest) gave it any thought at all - our responsibility stopped when we delivered a working simulator.

Eventually the day of the initial delivery dawned, and I went to the customer site, installed the simulator, and ran the delivery checks that verified that everything was OK. Customer sign-off of the delivery was obtained, and everyone was happy at that point.

About a week later we got an e-mail from the customer say that the simulator was causing their desktop machine to blow up! Off I went to the customer site, to be greeted by a very irate representative who proceeded to demonstrate how, when they ran a test scenario on the simulator, the desktop machine it was running on would shut itself down with extreme prejudice after about 20 seconds. I checked the simulator software that we had delivered - no problems. I then asked, casually, whether they had upgrade the desk machine's PSU when they upgraded the processors - I got about 30 seconds of silence before a muttered "Oh shit ..." was heard.

We never had any further problems with the delivered system.

"The PSU on this server's failed, can you send a replacement?"

The Dell rep did exactly the correct thing based on the information given, Hell desk people have to deal with the full gamut of callers from total numpties to people who actually know what they are talking about. S/he did the right thing in not assuming you were correct without garnering further information from you.

I worked for a large government department and our team were the first to develop PC based solutions, the department's vast majority of work being done on ICL mainframes. We bought our own development server (Novel Netware), wired up our own network, and got ourselves a UPS to protect the server. We used to test the UPS every 3 months, doing a controlled shutdown, and everything was hunky-dory.

About 18 months later we moved to a new building and our server was moved into a new server room and plugged into the new network. We got it up and running very quickly and things were going well. Until, that is, we decided to run our regular UPS test. We switched off the mains supply, then watched as the software on the server detected the power out and shutdown everything gracefully. At this point screams came from another team that their system had gone down.

When putting their server into the server room, that had spotted a spare outlet on our UPS and decided that it would be a good idea to hook into it, but they had not made us aware of this. Doh!

Ain't the laws of unintended consequences wonderful!

Bet they didn't do that again. The question that remains is...

Did this other team get their own UPS or did they freeload off another one. I've seen that happen before.

They banned Dilbert?

Well, to be fair, it's racism. Just not the sort of racism that makes it into US and western news feeds :-)

When I was working for what remains of the British Aerospace industry, we had a visit from a couple of gentlemen from across the pond. Apparently one of their other projects had included working with a company from the land of the rising sun, and some of the visiting employees had been heard to repeatedly make nasty comments comparing the lazy and incompetent Americans to the diligent and hard-working Japanese workers.

Bearing in mind these were less enlightened times and the Japanese management refusal to stop the comments (did the American workers arrive an hour or more before their shift started? Did they show "proper respect" to their bosses and the Company?), some of the factory floor workers made some "motivational posters" of their own - the words "Built by lazy and incompetent Americans, Tested by diligent and hard-working Japanese" - with a picture of the Enola Gay...

Pre-dating the days of IT (well, OK there were a few BBC 1st gens knocking around) when I were just a lad earning some extra pocket money working weekends and holidays at the local car-wash (yes, it was called "Jeeves"). Me and my mate were considered trusty enough to open-up, cash-up and even drop the takings off at the bank when the manager was away. The start of the day involved powering up the whole plant, separate relays for brushes, rollers, heaters etc, and a certain order these switches had to be turned on/plugged in. Bit of a faff running back and forth, so Pete and I had worked out a system where he did one end and me the other, suitably timed so that the power-up sequence was maintained via synchronized yells. Worked really quickly and efficiently, until the day we managed to get out of step. I plugged in the last connector but unfortunately it was already live at this point, resulting in a big, black smoke-filled bang. Needless to say I jumped back a socially-distanced 6 feet from the billowing socket, but with a charred and completely crusty-black hand.

"This is going to hurt soon" thinks I, as Pete comes scampering along to investigate the loud noise. He goes a bit weak-kneed and has to sit down when he spies my hand, which surprisingly was still attached to my arm and hadn't started to register on the pain threshold yet. I tried to flex a finger, and the skin cracked along the joint. I say "skin", but it became obvious that my hand wasn't completely barbecued, it was merely covered in a thick layer of carbon that had spouted out of the electrical socket. A few dabs with another finger to confirm this theory, then a swish under the tap confirmed that everything was indeed still in working order.

And the surprising thing was, after we'd re-set the master relays and re-powered up it all worked, even the exploding plug. Lessons learnt - until Pete and I had a go at fixing the VI form vending machine, but that's another story.

Just a week ago Sunday, while driving from home to church to play drums -- a distance of only 3 miles -- the missus called and said there were power issues at home just after I left but after a few flicks up & down it stabilized.

But the church building was a different story. I walk in to the sanctuary where only half the lights are working and all the outlets on the stage-left side* are dead. This included the remote audio I/O unit that connects to the main PA board by Ethernet (Allen & Heath system), the PoE that runs two A&H personal-monitor units (one is mine), the Roland digital drums I play (but the church owns them), the Hammond organ, and all the musician's powered monitor speakers, plus the main monitors at stage front.

And on top of that many of the HVAC units were dead, so the environment was not horrible but temperature was slowly climbing; the building's internet was also gone so the Facebook livestream was in question.

This being the first Sunday we were trying two services to keep attendance down and social distance up, this was NOT a good omen overall. (The livestream was going to use the second service.)

Lighting aside, the main A/V booth survived -- graphics computer, audio, and main speaker amps -- as did the large flat-screens (two for the congregants and one in the back for the praise team). We lost organ, drums, and audio interface to the separate digital keyboard, but still had piano (direct-wired mic), guitar (wireless pickup to the main board), horns, and my personal pair of bongos and were a little more "unplugged" for the 9:00.

Seeing as the outlets actually on stage were working, which are only occasionally used for LED PAR wall-wash lights, someone grabbed some extension cords and we got the drums, monitors and audio/PoE units working again just before the second service. They still did the livestream, probably by turning on tethering on someone's phone; I heard from the missus that the feed wasn't as stable.

Here's what really happened: the building has a 3-phase utility connection at the main road; different parts of the building tap into different phases. The power flicks** took out 1 phase blew the fuse on the roadside utility pole. Thankfully we have a congregant sparky (electrician) who quickly diagnosed it and helped turn off any 3-phase loads (like the HVAC) quickly. (I think they tried some cross-over switches trying to restore the dead circuits by tying them to a working phase, but that kept tripping also.) In the end, nothing in the building was harmed and it was all-systems-normal when I was there Thursday evening for practice.

* Yes, I worked it to say "left side" as an allusion to Marvin and you found it. Pat yourself on the back, you nerd.

** I was also watching our utility's outage reports via their own app. I think there was an issue with a substation very close to home and it took out some nearby neighborhoods completely. Wouldn't have been the first time. I blame the squirrels.

Many moons ago I could smell a strong burnt cabbage smell and remarked that it must be nearly lunch time as the canteen was on the floor above and the kitchen just above where I noticed the smell.

Turned out to be a selenium rectifier burning out in an elderly 50v telecom PSU rather than the dish of the day

So who's he leaching 'net access from?