A few years (and companies ago) we had to run the site on the generator because someone had stolen the busbars from the nearby substation. How they did that without getting fried I don’t know.

It was realised that as we were waiting for the power to come back on we were a bit short of fuel for the generator. I think I remember hearing it got down to 20 minutes life, and this bit is a bit vague as it was years ago they might have driven the minibus to the nearby petrol station (which still had power) filled it up and come back and transferred fuel to the generator tank whilst waiting for the fuel truck to get there.

It could have happened to us.

Previous office location had had dodgy power in the early days so a fixed diesel generator was installed. It was never needed in anger from when I joined the company (the building works that caused the problems were long completed) but someone duly tested it every week or so, until it failed through lack of fuel.

Nothing more was done until we were holding an important seminar when 40l of diesel were duly purchased.

As far as I'm aware testing continued until we moved out, without any further fuel supplies, so I've no idea what the run time would have been if we'd ever actually needed it.

I was recounting a fuel empty generator story to my daughter last night. In our case it lasted about 2 minutes before it spluttered to a halt!

I know of a cinema in Reading about 20 years ago that had exactly the same problem.

Eighteen months after the site had opened they get their first big power cut. The cinema goes dark, then the generators start up, the lights come back on and the feature film continues. Ten minutes later the lights go out again and stay out. Site services had been running the generator for a few mins every week for that eighteen months, but never replaced the diesel they had been using up!

I remember a site where a digger managed to fracture the aircon main pipe between DP room and the aircon plant. Having dual power supplies etc didn't matter if the kit gets to hot to use.

I also heard in France where a company had two external network. One went via the north of France, and one went via the south of France. Which worked fine, till the telecoms company put a major backbone across the middle. So the northern network went north, then south to the backbone, across France, then north, then south to their office (and the mirror image for the southern network). This all worked well until there was an outage on the backbone, and both networks were impacted.

I've related this story before, but in the '80s, I was working at a campus site some distance from the nearest large town. The power was always a bit flaky there, so they had a decent UPS with diesel generator that would even keep the Amdahl mainframe running.

So one afternoon, there was a violent thunderstorm, which lasted several hours. There were repeated brown-outs on the power, and each time, the UPS cut in and initiated the diesel generator start, but the power returned before the generator was stable enough to provide power.

This does not seem like a problem, but one of the things that happened when the UPS cut in was that the air conditioning in the machine room stopped (it was not on the UPS by design), and then took several minutes to restart when the power returned.

So each time there was a brown-out, two things happened. The available power in the UPS started dropping, and the machine room temperature started rising.

Eventually, it was decided to shut the machine room down, because it was getting too hot, and would completely exhaust the UPS. And everybody except the core operations staff were told to go home, as they could not really do any work without the systems.

All this could have been avoided if there had been a manual override on the generator, to start it and leave it running, but nobody had thought that that would be required when the system was designed. It was soon added afterwards, though.

> Backups were performed and rotated and once a week

Necessary, but not sufficient.

Did they ever check that the backups could restore their systems?

In practice a much more difficult task than many would imagine as you need a complete identical set of servers (plus other infrastructure) to restore to.

I remember one customer who was decomissionig their Mainframe about Twenty years ago, and they very carefully took backups of everything to tape.

Tapes for which the only available drives in a 3000 mile radius were the very ones that were packing off to the landfill the week afterwards!

They were doing this I may add after their support contract & software licenses had expired, so strictly speaking, wehereas they owned the hardware they shouldn't heve been using the operting system. Their fix for this little legalistic problem was to bar all employees of the mainframe provider from the site until they were finished.

Presumably there was a separate migration to a new system so the backup would be belt-and-braces for an extreme situation. If there were suitable drives almost anywhere on the planet this could be acceptable.

Way back in the mists of time working on DEC Vax we had a total loss fire. Fortunately we had backups so a temporary replacement system was procured and located in a site hut near enough to the comms room, which had survived, for the hardware people to run the necessary cabling.

Unfortunately the replacement was specced (not by me) with compatible drives not genuine RA81s* which turned out, when formatted, to be ever so slightly smaller meaning most of the restores failed. The supplier ended up putting in larger compatible drives to resolve the issue as the 21 year old wet-behind-the-ears IT guy (me) refused to sign off their 6 figure invoice.

*For the youngsters, 456MB, 19" units, 3 to a cabinet IIRC.

If you use the services of a DR company the greatest need is to check restore on their H/W. The experience of your first test - or rehearsal if you prefer - can be quite informative. The first time I tried it we ran out of time just about at the point where we had a file system system we could log onto as /etc had been backed up after a lot of stuff it would have been less urgent to restore. Things were changed round so that next time we were in a position to start restoring the database fairly quickly.

I remember our big outage -- phone alert -- rushing to the site and knowing most of the battery life would have gone by that time I got there. I start shutting everything down cleanly until I suddenly realised that the lights were on again. (It was during the day so there was some excuse.)

I landed a contract to install two big, garage sized, Memorex tape backup robots at a large number-crunching outfit once. Before I bid on the job, the VP of operations gave me the grand tour. He was proud of all his redundancy. He had two power lines coming in to two separate rooms, with a motor-generator, a large battery consisting of dozens of telco-style lead-acid batteries, a generator, and monitoring systems for each room-full of gear. The 48 Volts was switched by a box at the corner where the two rooms met, brought into the main building via a 5" conduit, where it was switched to two separate computer rooms. Even the links between outlying offices were redundant T-1 and T-3 lines. There was a third "data center" that was dark, to be used for spares "just in case". It was designed to provide non-stop operations, and it did a pretty good job of it. Even the Halon had built-in redundancy.

Until a semi-truck carrying some of my Memorex kit backing into the receiving dock went off course & cut the 5" conduit. The security cameras caught the sparks quite nicely.

Two weeks after getting the tape robots installed and signed off, I had a proposal for a more geographically diverse version of the same thing on the VP's desk. I didn't land that contract, alas.

Even the best-considered systems can fail due to the improper use (or absence of use) of the the simplest piece of equipment.

I was told this story yesterday.

Once upon a time we had a county Council*. They used to use snow ploughs to clear rural roads* in winter. Because some of the roads are steep and snow ploughs are less likely to get stuck coming down hill they decided to park one at a convenient spot at the top of a hill. When the sown came some poor council worker fought his way through the drifts to the top of the hill. And found the snow plough''s diesel had been stolen.

* This now seems like a fairy story, hence my use of the traditional opening..

Rural folk don't steal from their neighbours, after all we're all in the same boat, but council gear was paid for us all via the rates* so it's fair game.

* or poll tax or council tax, depending on how far back we're going.

"Rural folk don't steal from their neighbours"

Transplanted city folks sometimes do. Once.

In my head there's a banjo playing. For the UK readers I live not far from Royston Vasey.

I'm sure it happened somewhere.

I had someone steal the fuel out of a riding lawnmower once. I had topped the thing up (three wimpy US gallons), then headed back to civilization, intending to return the following weekend to mow the verges of our access road. I got back two weeks later and discovered the gas cap removed and the tank empty ... but everything else was exactly where I had left it. This was down a 5 mile driveway, behind three locked gates.

Judging by the tire marks, it was three kids on dirtbikes.

"three kids on dirtbikes"

Does your hunting licence cover these?

.... until your genny explodes mid use.

Back in the day we experienced a power outage while on a cig break. The clue was in the lights going out and the genny exhausts (about 3ft behind us) bellowing out fumes.

By the time we walked back in , all was well. Servers running, Management suitably impressed.

Panic over, latte/cig and smug grin time, then BANG!

What appeared to be a large cap had blown, taking off the genny door and the passenger window of the car parked opposite.

Usually see “practice” used as a verb (accepted use across the pond, of course).

Don’t ever recall seeing “practise” used as a noun. Another Register first?

Have one of these ->

I was told this story some while ago, it may be an urban myth but there are elements that sadly ring true.

I major data centre in the US of A went all out for redundancy. Dual incoming power supplies, with diesel generators for backup, and UPS capable of handling the full load for about 20 minutes just in case the generators were slow to spool up. The generators were tested each week and (shock, horror) the fuel tanks checked and kept topped up. The one problem is that the fuel tanks had to be located about 100 yards away for fire safety reasons, but this was solved by installing fuel pumps that kept the generators fed very well.

Of course the inevitable happened any one day both power supplies failed (the story discusses the rank incompetence of the power company in great length using may pointed words, but I digress). The UPS initially kicked in as planned, then the generators started up and took the full load. About 30 seconds later they spluttered and died, with the rest of data centre following once the USP batteries had drained. It turned out that someone, in a fit of unbelievable genius, had wired the fully pumps so that they could only get power from the mains supply.

I'll leave the resulting management reaction to your imaginations.

Worked for an a managed service provider and had a suitably impressive DC with lots of redundancy, all top notch, including diesel generator that was managed by the local power grid provider (Who used it to smooth out power supplies when demand was high). All great stuff.

However the UPS batteries as you will be aware do need some tender care and attention to keep them in great condition, this was done on a regular basis but unknown to everyone is that we have a defective battery (That past inspection) and during peak daytime running suddenly failed and produced an impressive amount of smoke, this triggered the fire system which duly shutdown the DC. Had a bit of fun trying to stop the local fire brigade from axing their way through the doors, we managed to show them the doors were unlocked (As part of the fire system released all locked doors when activated).

The damage was limited to the UPS battery and some associated kit, nothing dramatic, but then the fun started with explaining why the UPS/Generator et al did not keep the DC up - It would seem obvious that in the event of a fire that the DC power would be shutdown, but this seemed to be a surprise to senior leadership and a number of the customers .......

Amateurs...

The other common failure is a flat battery, see/heard about this so many times from those who don't test them regularly (you know, the not very important places like hospitals).

LOL, we had two UPS's and servers with twin PSUs

And several racks powered exclusively from one UPS.

And that UPS had a defective battery. Can you say recover SQL cluster?

I was a member of the ops team in a large multi site organisation. The primary site had redundant data links to the other sites. Two shiny new high capacity links provided by a large telco famous for many, many other things too....

I was called in one day to check the cross site links. They were all down. Famous telco swore blind their network was fine, just not getting data from us.

So it must be a problem at our end ;-( We're up, the links are up, just not getting a response from them on either line.

The two independent links left our site, by two different routes, and disappeared into the famous telco's network. Both went into the same famous telco's building, in the same room....

Turned out it wasn't a problem at our end ;-)

Many years ago I worked in a large department store that had a frozen food concession on the ground floor with a huge walk-in freezer in the basement,

The shopping centre gave us advanced notice that they were cutting the power in a week for essential work, so I was sent up to the roof to check the big Cummins generator.

It fired up OK, but upon dipping the tank I found it was all but empty, so we ordered a barrel of diesel which was duly delivered the next day and connected to the pipe in the loading bay to be pumped 4 floors up to the roof, except it's that long since we'd used the pump it had seized up.

We put the barrel in the goods lift, took it up to the top floor, then had to carefully drag the very heavy (about 200 kilos) up the last flight of stairs to the roof, where yours truly had the wonderful task of siphoning the fuel from barrel to tank.

Genny tested again, all good.

A week later, the shopping centre gives us the good news that they could carry out the work without cutting our power...

I was told this by a risk assessment specialist (on a project that was building railway stations).

Fire alarm systems get tested periodically, but if there's an integrated sprinkler system, you obviously don't want to actually cover the building in water. So, there's a diverter valve (or several), which can be set to channel the water to a test outlet, enabling a test of the active parts of the alarm-plus-sprinklers.

And the interesting risk here is that there's a non-zero chance that after the test, someone forgets to re-divert the output back to the actual sprinklers!