I was taught that there are no tested backups, only restores. Even in retirement, each week I still rename a different unimportant file and then restore its original with Time Machine...

"To believe in one's backups is one thing. To have to use them is another."

http://www.taobackup.com/testing.html

After this year's Christmas, I wish there had been an emergency at work needing my attention. It was painful watching the family argue in front of the kids.

It always is.

It's disheartening to witness the sheer amount of families who just can't keep their shit together for one day in the year.

You had Christmas? My in-laws didn't even put up a tree, or buy gifts... I've made sure the boy has had a good Christmas.

Long ago, in another life, I looked after an inherited NT4.0 file server built around the infamous EDS.

No RAID setup or adequate supply of back-up tapes, for reasons best known to the beancounter director.

What could possibly go wrong with that?

One day Murphy decided that a corrupt EDS and a worn out tape at the same time made for a lot of fun.

As a result, I had to stay in a Thursday till 03:00 and the succeeding long week-end to fix that.

O.

Which "infamous EDS" ? Google and Dudckduckgo show lots of possibilities, and only about 20% are IT related...

> What was to be done? Would Bill need to abandon his family, and his holiday?

It seems to me that the people who caused the problem are the ones responsible for fixing it. If they cannot, then it might be time to replace them with others who could. Or better: with true professionals who would avoid the problems in the first place.

I have worked for outfits that have absorbed other companies. Including their less capable IT departments. It would not surprise me if those depts. lack of ability and (frequently) their antiquated systems was a factor in their parent companies owners wishing to get rid of the problem.

Whilst I agree with the sentiment, sometimes its the bean counters who made the problem or short sighted Directors who wouldn't approve the fix before it was needed.

In the case of having the equipment but not bothering to set it up sensibly or not asking for the correct solution, then yes hang them high.

I've been 'lucky' enough to inherit several failing tech support teams over the years. In both cases case it was pure bad management, in the first the guys just needed to understand the basics of change management and logical problem investigation in complex environments, it took 18 months but I turned them into a world class team with completely stable production environments and we went on to implement several major technology changes (change from novell to windows server,implementation of Unix servers, email implementation then migration from group wise to exchange etc) successfully after that. In another case the team was split between the incompetent, the lazy and the ill informed. It took longer to get rid of the incompetent who had been the 'team lead' but I managed to bring the other two up to speed by putting my own team to work mentoring them and helping them to develop, all they needed was exposure to a good team culture, one their previous manager had deliberately poisoned

"I turned them into a world class team"

How did you know? Does the olympics have an IT admin competition?

I doubt your world class team was even nominated for the Montgomery Burns Award for Outstanding Achievement in the Field of Excellence.

upvote and downvote for you!

I had one boss that was great. Sure I got called for a few things, but he knew when to chuck in the towel, and when to keep the upper manglement off our backs when we had to recover from a case of intense idiocy... Funny, it's only been a year since I worked for him, some days it seem like 20 years ago, and some days it was only yesterday.

I once spent an hour and a half at the bottom of a slope watching all my mates ski on (initially) pristine snow while I did remote support for a piece o SW I'd had the misfortune of taking over mid-development some weeks earlier. Of course none of the other members of my team, who were not on holiday, could take the call...

Adding insult to injury, when I came back, I spent days knocking on doors until someone at my company "magnanimously" signed-off to pay me back the roaming charges

I've never, ever come across a situation where a senior manager said thanks carry on with your vacation. During my time in tech support / diagnostics I've been forced to carry on trying to recover data off a completely dead server, performing pointless reboots until 3rd line support arrived ad 07:30, driving into the office a 3:AM just so 'someone's working on the fault', had to baby sit an engineer through dissembling a £1m mainframe to get to a faulty fan, he hadn't received any training as 'field engineers wouldn't need it these machines are so reliable'. I was a software specialist in the diagnostic center, alone on nights but had the hardware manuals, he was alone on site with no documentation, he'd just managed to get to get to the part when a trained engineer arrived 10 hours later who managed to complete the work in 2 hours as he actually knew which order to assemble parts in).

It always seems senior managers can sleep better if they know a technician is awake and 'working the problem'

Been there, seen it, lived through it.

Only my backup nightmare was due to someone changing a UNIX backup device from st0n to st0 at some point, without change control, causing portions of data to be erased. Long time ago, before there was any commercially available backup software from what I can remember. I remember finding out the backup was corrupted, affected multiple backups going back weeks, and being summoned to the boss's office to explain. Luckily for me I was able to point out what the problem was and that it wasn't anything I'd done!

Fun times.