The thing is, with daily tasks, they just get done automatically. Monthly, OK there's a month end process. Yearly tasks are only ever triggered if they are literally scheduled for Jan 1st (or 2nd if it's a manual task). One can so easily imagine the batteries being installed in June, someone putting a calendar entry for the following June and everybody forgetting about it. Then the first person leaves the company, the batteries continue on their merry way for 3 years, and before you know it, there isn't even anyone on the team who remembers what that box is and what it does.

Similair applies to server certificates with multi-year validity. Shorter renewal cycle could be more of a PITA, but actually work better for reliability.

Completely agree. Yearly tasks are a pain to schedule, especially when the responsible person leaves the company (whatever reason).

I'm thinking there should be a yearly calendar process for business. The business needs to know these things, and they cannot be tied to the knowledge of one person. How many times has Borkzilla been caught out because it forgot to renew its own certificates ? And it's far from being alone in this. If there was some sort of shared calendar showing yearly actions and who is responsible for them, then I think this kind of issue would be almost resolved.

At a business level, that should be easy to organize. Create a user named Yearly Renewal, share his calendar with all upper management, and have HelpDesk insert the proper data and the names of people responsible. When one person leaves, check Yearly's calendar for all events under that person's name and figure out who to reassign them to. Then make sure everyone is checking Yearly's calendar at least once a month.

That should work, no ?

modern email/organisation systems (as they are more than just email if they contain a calendar system) should have a Shared Calendar facility so you don't even need a fake user to handle it, which could expire or be purged in "tidying up" by accident..

The CAB have pretty much killed off certificates with multi-year validity, but since getting bitten by a certificate expiry I have a weekly task which sends me a report of certificates that are expiring in the next few weeks. Now the only way that certificates cause me support headaches are the users in third world countries who use 10-year-old Android phones whose root certificate lists need updating, and we only get one or two of those a year.

Once you've realised that this thing needs batteries changing yearly or $things will fail, why not replace (or supplement) those batteries with an external power brick? Plugged into the UPS on the basis that if the UPS dies, that clock will be the least of your worries.

I guess the change request of :

"I wont to cut the dc plug off a 9v answering machine psu and solder the wires to the battery terminals in the Network Clock time-getter"

might seem a bit too Wallace & Gromit.

Smoke alarms start to peep when the battery gets low and then squeal when it gets really low. Perhaps a 'fail loud' modification to critical battery operated equipment?

someone putting a calendar entry for the following June and everybody forgetting about it.

Isnt the point of a calendar entry so that you *can* forget about things and be reminded of them when the time comes?

> "I was immediately summoned to explain what had happened"

There is a crisis in progress and one person in the building who can address it, so naturally you call that person to your office to tell you what they would be doing about it if they weren't talking with you. Why come does it happen that way around so often? Would it hurt to go to where they are dealing with it and ask for a time estimate?

"Are you working on the current outage / issue?"

"Yes"

"Thank you, carry on, I will let everyone know not to disturb or interrupt you"

That is all it needs.

"Anything I can help with?"

As my dad was fond of saying - "you can help by staying out of the way".

It begins at home. I've lost count of the number of times SWMBO has stopped me getting ready in a hurry because we are late, to tell me we are late and I need to hurry. SMH²

Anon 'cos I'm attached to my testicles and I'd like it to stay that way.

I once rather stunted my career prospects within an org because of this.

There were two large (but separate) outages, affecting two large customer networks. It was during the later hours of our night shift, but around peak time for them. There were only 2 of us on (due to cost-cutting leading to understaffing).

The two of us took a network each and started investigating.

Except... the phone kept ringing, with our CEO demanding updates/ETA etc. This wasn't driven by customer demands - they were well aware we were working on it and that we'd give updates as/when they became available, it was just the CEO and his micro-managing ways.

Around about the 5th time that the phone interrupted my train of thought, I answered only to hear our CEOs voice, again asking for updates and ETAs that couldn't yet be given. Rather than explaining again that we weren't yet in a position to provide these, I said "at least 8 hours".

He was *beyond* dismayed and asked what could possibly have gone so badly wrong that it needed that kind of recovery time. So, I explained that the team were unable to effectively investigate and recover services because they kept being interrupted by pointless calls asking for status updates.

He didn't phone back again, but did phone my boss later in the day (who backed me up).

During the process for a later internal job-move (and promotion) though, the hiring manager needed approval from the CEO, who denied it.

Should have got your boss in earlier...

Had a self proclaimed paper pusher manager at a previous role, and they were brilliant.

They had a great memory for "the team saw something maybe similar seven years ago, here's the first and last couple of emails - does it sound relevant" and was excellent at being a management shield when needed.

I only had one "event" where I ended up on three calls first thing in the morning - the third was meant to be a progress report from the first - but the time between them had been taken up with the other call from a different set of management.

Now both sets of management did need to be updated, and updated properly (and that really meant by me), and the incident wasn't getting any worse (as we were now running correctly and had backups of all the raw incoming data from the event) - it eventually took two full weeks to repair and reprocess the data (which had been sent in the wrong timezone by a handful of systems, there was a nice overlap of timestamps at one end of the data which took more effort to resolve than I care to remember). But the first thing I said on the third call was "no progress, and there never will be if I don't get off these calls now."

Doubly so when they insist on PowerPoint presentations, pretty graphs and all the other nonsense required to prettify the data enough that their brains can parse it.

Or perhaps not even then, and you have to spend extra time explaining the basics, and then when another manager joins half way through then doing it all again from the beginning.

Leading to the oft repeated mantra of "Do you want me to do this update, or just waste my time actually fixing the issue?"

Ah! But MS Office's genAi CoPilots stand ready to solve that problem for one and all, in one short synthetic heartbeat. Complete with face-saving hallucination if need be ... that tech is a godsend for all On-Call Who-Me's (unless that's the very tech that titsupped itself of course!)!

Having finished some work for a bank, I was due to fly home Sunday lunch time. Sunday 0600 the phone rang. It was the boss saying they had a problem. They had tried to fix it.but the backups didn't work. Please could I help. He said anything I need - let him know. I quietly worked on the phone with the customer's guys, getting it sorted. Every 10-15 mins I messaged my boss giving status. I said we needed a networking guy... and 10 minutes later their top networking person was on the phone.

We got it fixed by lunch time - just in time for me to get my flight.

Afterward my boss told me the extent of the outage. Half of the bank was down - it was on the national news. He said he fielded the calls from every one- including the head of the bank's disaster committee who wanted to talk to the people fixing it. My boss said to him, if they tell you the problem and the solution, you will not understand it - trust us.. He didn't tell us the scale of the problem (although we guessed) because he didn't want us to rush and mess it up.

He was a good boss - he handled all the flak, and he trusted us to get on with it.

I used to work in the broadcast world, and came across a couple of such stories where the big boss wanted to know what was going on ....

1. In person. "What are you doing to get this fixed ?" Reply: "Talking to you"

2. On the phone, after rather tetchy reply from the fixer. "Do you know who I am ?" Reply: "Yes - do you know who I am ?" "No" Reply: "Good !"

I remember those toys. I can't remember why the local authority I was working at couldn't just allow port ntp through the firewall but there was a "reason". It wasn't batteries that caused the outage though. Window cleaners had dislodged the network cable.

icon because it is nearly that time

I used to manage a network with a couple of those (Galleon Ntp servers) one was fine but the unit at one office had problems.

The first antenna used the time signal from rugby which worked until they moved it to Birmingham (I think) at which point a large row of trees blocked the antenna completely.

After this we swapped to a gps unit mounted on the roof which worked until the enclosure sprung a leak and fried the electronics.

The final time I heard about it was about 2 years after I left the company when some builders cut through the cable as it snakes under the floor and they called me to ask where the thing was routed to….

When nobody was trying to sabotage it they were brilliant at their job and ran for about 15 years - they may still be working away…

Is that a British "Colin"?

Or the American "Colene"?

I thought the question was ALWAYS African or European?

Is a PITA!

I was seriously considering an atomic clock setup for one of our servers as, for some reason we couldn't fathom, it was consistently 3 minutes slow despite regular NTP calls and irrespective of the time server used. Had I been sure it would have resolved the issue I'd probably have gone ahead but in the end our support company managed to fix it with an obscure patch.

Did 3 minutes really matter? Whilst not as critical as the time for financial trading yes, it did! The factory time clock updated from the server weekly (and when the clocks changed), that three minutes was the difference between some of the operators catching their bus home or waiting an hour for the next one.

Not work but we had an issue with linked electronics (it was a proprietary ring bus system) on a yacht where the time on the chart plotter (which didn't have it's own RTC) had to be manually reset at power up when it was supposed to take the time from the GPS. Having finally managed to track down a USB interface for the network we went into packet sniffing mode and it turned out the scenario was:

Chart Plotter sends Time Request Good so far

GPS responds to Request Still good

Chart Plotter responds to Request Eh? Why is it responding to its own time request?

Chart Plotter receives time from GPS and updates Correct time

Chart Plotter receives time from itself and updates Overwriting the correct time with the default start up time

We were able to update the chart plotter config to stop it responding to time requests but why the heck it was ever enabled to do so was beyond us.

I had to fix an outage a few years ago where the Acopia fronting the storage used the PKI servers for its clock and everything else used Active Directory and the PKI clock became inaccurate the Windows boxen wouldn't talk to the Acopia any more!