I still blame them for not testing before the blackouts happened.

It's one thing to not upset your network for no good reason, but knowing that it's going to happen and not making sure the failover works is not acceptable.

The rule is: if you haven't tested it, it won't work.

And if you have tested it then it still won't work but for a different reason, that you hadn't previously thought of

The rule is: if you haven't tested it, it won't work.

So what you're saying is that you need a plan and you shouldn't just try to wing it...

"you shouldn't just try to wing it"

Oh, please, don't you think that comment was a bit cheep ?

The DR plan I wrote for a data centre explicitly did not address nested failures. Would I have got away with it?

> The DR plan I wrote for a data centre explicitly did not address nested failures. Would I have got away with it?

As long as you didn't crow about it...

These puns are just fowl

>These puns are just fowl

No, they are eggcellent...

Those higher up the pecking order should take responsibility.

As it was, it's a complete flock-up.

I'm not saying I'm going to steal borrow that joke from you, but I'm definitely robin it.

However, some of the comments are a bit hawkish

Another reason for backup power failures I know of is they test the generator by having it start and run while the system remains on mains power.

Yes, you know they don't actually cut the supply and verify that it switches all of the required over to backup, and that on switch over there is not such a surge it stalls the engine.

Oh and let us not forget the other deal-killer that follows from that sort of a "test" where not everything that is needed is powered by the backup power. For example the A/C (yes, really!) or a couple of network switches around the building where the related local UPS comes off a non-switched branch of the power system.

Back when I worked for Bigger Blue (late 1970s), on Fabian Way in Palo Alto, we'd kill the mains power at 3PM on the last Friday of every month to ensure that the battery would carry the load long enough for the genset to warm up enough to take over. It ain't exactly rocket science. (In the event of failure, everyone went home early with two hours pay. It never happened while I worked there).

Keep in mind that this system ran the entire campus, not just the computers. (OK, nearly the entire campus ... The fine folks in the Faraday Cage over on East Meadow Circle and across Adobe Creek on West Bayshore had their own solutions ...).

While plausible, how long does it take for enough poop to build up to diable the switch?

Fundamental backup gear like generators should be inspected and tested monthly at least.

I visited a switchroom where the control panel was covered in pigeon poop. It was deep doh-dah on the top, the front had solidified 'poop-falls'. The floor was crunchy, the air was acidic, the steel enclosure was rotting away. Truly disgusting.

Incredibly, some of the machinery was still operational. Any further fault investigation was going to have to wait for a deep, deep clean or probably shot-blast.

We were never called back. Entry was forbidden due to a bio-hazard.

When wiring up your new emergency generator do make sure the 3-phase is correctly connected on your control panel. 415V in lieu of 230V on the desktop PC ring main is quite spectacular.

In the early sixties, when the grid was not as robust as it nowadays is in central Europe, a mishap of this sort happened, only on a much larger scale. It was a very, very cold winter and every inch ounce drop SI unit of electricity was needed. Those times it was not uncommon for factories, streetcar operators and the like to run their own insular electrical systems which could be connected to the mains grid when necessary. All coordination was done by telephone. It was in the steel works where my grandfather worked, that the operations received a phonecall pleading for their "phase in" - the steel works power plant first needed to be phase synchronized with the grid, which most of the time it was not, for it to be connected as an emergency source of electricity. This phase-in was to be the first one after some refurbishment works on the main transformer. Then the phase-in would be done using a bulb connected between the steel works mains and the grid mains. As soon as the bulb went completely cold and dark, the phases matched. Only then this would be done with one bulb only and the technicians usually saved time and effort by not checking other phases. There was no way the other two phases would be out of synch when one phase was synchronized. Or so they thought - and you know very well where this is going. The bulb went dark, another phonecall dutifully came in, the senior technician pulled the lever, and the whole steel works together with the whole surrounding area went completely dark and silent, except for the glowing hot steel cooling slowly inside the production line and the atershocks of the generator trying to reverse-step its waltz and now dancing around the generator hall.

For the next phase-in, after the whole power generation station has been duly rebuilt, three new bulbs were procured, for the old one was pronounced unreliable by the management.

All hail the magic smoke!

In one case, the generator ran exactly as planned during every test run, except there was one small issue that was not detected. The fuel feed was for some reason fitted with a multi-way tap. This allowed the engine to draw fuel from a local header tank or the main tank. The reason for this set up escapes me at the moment. Of course, all those test runs had not exactly refilled the small header tank. When the mains power went on holiday, the backup generator kicked in, ran for a few moments before spluttering to a halt. Diesel engines do not like being run dry of fuel, so even when the problem tap setting was found, it took a while to prime the engine and restore matters.

I saw an issue where the fuel tank wasn't vented, so it would run for a while, then the vacuum would stop the fuel draw.

"This allowed the engine to draw fuel from a local header tank or the main tank."

At bigger blue, the second smaller tank was to provide enough fuel to run the system should a power failure occur right in the middle of the twice-yearly flush & fill of the main tank. The "small" tank was only 300 gallons, the large tank was in the 25,000 gallon range (both numbers from memory). We had a small fleet of trucks in to pump the old fuel out, and then the new fuel in. Took pretty much all day.

Most modern diesel generators have self-bleeding fuel systems. Thankfully.

can even cause much wider blackouts, by causing flashover of insulators. If that happens at a crucial substation there'll be dark times ahead.

It definitely sounds like something from the random BOFH excuse generator to me.

Bird Output From Height ?

That sounds like an excellent punishment for any management trapped below.

"Or perhaps the facilities engineer used the antics of his feathered friends to cover up a cock-up of his own."

Yes. The switch gear for this kind of thing is protected from the elements, and birds. The automatic versions don't have any moving parts on the outside[0], and all access is (supposed to be) sealed. Mine don't even get mud-wasp nests in them. More likely he forgot to wire the automatic bit.

[0] Ive installed many manual versions that have a three position switch that can be thrown from the outside, usually labeled "PG&E", "OFF" and "Generator" from top to bottom. Each position has space for a lock, to keep prying hands from killing you. This switch disappears into a weather (and bird) proof box, where all the mechanical bits are contained, safely away from bird poop, insect damage, ice storms, and the like.

My thoughts exactly. This sounds like bullshit - sorry, BIRDSHIT :-D

Not quite the same thing but once worked for a company and the air con in one of the server rooms failed at some point over the weekend and wasn't detected till everyone was in the office on Monday (we had several server rooms and we weren't allowed to know which server room held which servers so might have been developer or tester only stuff and thus not important enough to have real time monitoring). Do know everything in the room was fried and had to be replaced. Expensive lesson in cost saving/disaster planning.

Sometimes it really is better when the shit hits the fan!

One financial institution I worked at many years ago decided to do a generator test late in the evening midweek - I was on shift that evening.

The failvoer goes Mains -> Battery -> Generator

On this test it went something like Mains -> Battery -> Generator (fails to start, everything goes black) -> Back to battery (everything comes on) -> back to generator (everything goes black) -> back to battery everything comes on -> batteries give out -> everything goes black and stays black.

No emergency lighting in the DC where the bridge was it turns out and no outside lighting to shine in thru the windows as there were none.

The hilarity, the mainframe batch runs that were in the progress at the time, the chaos - fun times.

So issues with the generators and lots of issues with failed batteries, lots of root cause analysis, lots of beefing up batteries, failover procedures etc...

There was fun at one bank where the generator fuel tanks spung a leak and flooded the basement canteen to a depth of 3 feet, they never did manage to paint over the diesel tide mark enough and their was a odour of diesel for the rest of the time I was there which never quite left.

I heard this story from a line technician with a local distribution company. They would register a very perculiar failure on one of their local 22 kV lines around the same date every year. It was a sudden potential drop very early in the morning, not correlated and not looking like any other typical failures they would see. It went for some years with inreasingly more targeted observations every year (and correspondingly more intense expectations). Once they got a young hunter on the team, and he agreed to actually go and observe one suspect section of the line at the usual time the potential drop was expected to happen. He would go and camp right next to the line for a week, before he would be presented with the clearest of explanations. The line became a temporary home to a huge flock of migrating birds who spent the night sleeping on the line. At the very instant the sun rose, every bird woke up at exactly the same time and took a dump. The number of those birds along the line was so huge that the, erm, potential drop they collectively undertook actually registered on the line diagnostics.

Birds and power wires, not always a happy combo. Sometimes the angry pixies have had enough and get extra angry: https://youtu.be/tqvVYWBGucU

> that old favourite of the HR axe-wielder: "restructuring."

Why are they the only department that never gets staff cuts?

Happened at a company I worked at. We went through two rounds of redundancies, dropping from 120 people to about 80. And then two weeks later to 79 when our HR person was made redundant "as the size of the company meant we could no longer support full-time HR". HR was then provided remotely by the HR person at a sister company elsewhere in the country.

After having been told by the head of PR that my lack of experience in SQL meant I wasn't suited for new role A and my application for a HoD post was aspirational and that the only role I would be considered for carried a drop of over £17kpa I sadly waved goodbye to the organisation pocketing a £12k severance and having used the prolonged restructuring period to go job hunting. I was one of the last out of the door which did unfortunately mean the leaving party and whip round amounted to no party and a £5 book token.

Around 8 weeks later I bumped into former head of PR in a tube station. She asked how I was and if I had found a new post. I told her I had - I was now HoD with special responsibility for repairing and optimising a SQL solution that the external developers had made a total mess of. Plus I had taken a £14kpa uplift in pay and had a 5 year guarantee on my contract. Being polite I asked how she was. They'd got rid of her post two weeks after I had left, was unsuccessful in applying for the new post but didn't mind as that was a drop of £23kpa and she was just heading for an interview for another post as a deputy head of HR at the same pay. I didn't see her again and don't know what became of her. And frankly I don't care!

HR not PR. Auto correct!

This sounds like a scene from High Anxiety... a bird turd encrusted switch is comical, a fried bird that was on the switch when the power flipped would have been Hitchcock

https://www.youtube.com/watch?v=scgO8Vfh1qU

I think Mel needs the coat!

Many years ago - pre-IT days - I was working at a local radio broadcasting station. The place had lots and lots of lead-acid glass cells in a huge battery room to maintain DC supplies in the event of mains loss, and a backup mains generator of 30kW or so built onto the back of a Land Rover.

In the event of power loss, first talk to the power supply company to see if it was likely to take some time to fix, and then drive the Landie round to the power inlet and plug it in.

Came the day of a test, and the junior engineer appointed to this task disappeared for some time and returned to announce that neither he, nor any of his immediate colleagues, could start the Land Rover... oh dear, bit of a problem there.

After the post test analysis, a placard was affixed in front of the steering wheel announcing "This vehicle has a choke. Pull out before starting".