When you're On Call, only you can hear the silence of the clicks
- Reference: 1602832509
- News link: https://www.theregister.co.uk/2020/10/16/on_call/
- Source link:
Cast your mind back to The Before Times. The mid 2000s, to be precise, where our protagonist, "Bob", was gainfully employed within the walls of a major financial institution (which must remain nameless for Reasons.)
Bob was at the human end of a lengthy incident response chain. Well-designed systems monitored the applications in this well-known institution and, if something happened that didn't match expectations, an alert was generated.
This alert would normally fire off a SMS to whoever was unlucky enough to be carrying the on-call phone. SLAs dictated an acknowledgement within 15 minutes, and failure to do so would provoke robo-calls. Continued failure to respond would see those robo-calls redirected to the bosses, something which would likely prove somewhat "career-altering," as Bob put it.
On the night in question, Bob had the dreaded phone.
A transaction being conducted in Japanese Yen managed to burst through the maximum allowable value in the system. This was not particularly uncommon "due to a combination of the exchange rate and the size of the transactions" with which the finance house was entrusted.
Alas, perhaps because Bob had imbibed a tad too many adult beverages that evening or perhaps because he was just a really sound sleeper, the chirping of the "help me" SMS did not wake him up. It took the robo-calls to rouse him and realise that he needed to do something or face the wrath of the bosses.
The borkage was easily identified, and Bob dutifully followed SOP: split the transaction into two separate trades with values that would not blow through the ranges. Blearily, he checked his work and then hit the "restart batch" button.
Nothing happened.
He blinked, admitting later that he might have been a "bit fuzzy" having been so rudely yanked from dreamland, and theorised that he'd simply missed the button. Again, he clicked "restart batch"
This time "batch restarting" appeared, signifying all was well and Bob went back to bed, a job well done and a pay packet due to be inflated by an overnight call-out.
Bob's disturbance payment was but a trifle compared to the mayhem he had accidentally unleashed, which only became clear when the figures were totted up at the end of the month.
It transpired that the first attempt to restart the batch had actually worked. "A glitch in the UI meant that the confirmation message wasn't displayed," he claimed.
Uh huh. Right.
His second click of the button had also worked.
"The outbound payment batch process ran twice and the system happily paid out over $100 million to various financial institutions, high-wealth individuals, etc. - twice."
After much gnashing of teeth, wailing, and general keyboard bothering from the bean counters, Bob was eventually invited to explain how he'd managed to overpay $100m.
"It's not my fault," he protested, pointing out that running such critical processes in parallel was a bad idea and the screen didn't provide a response the first time round.
He failed to mention he also had a good few units of Stella Artois's finest still coursing through his veins. Probably irrelevant, right?
The unfortunate account managers were set the task of clawing back all the money, which they mostly managed to do. There was the small matter of $5m paid to a bank that had collapsed in the meantime, but for an outfit the size of this financial institution, such an amount was little more than a rounding error.
And Bob? He wasn't fired, but found himself "firmly at the wrong end of the bell-curve when the performance reviews came around." No pay-rise or bonus for Bob.
Nowadays Bob can be found toiling away in a huge infrastructure provider.
He doesn't do on-call any more.
Ever indulged in a bit of bad behaviour while entrusted with the On Call phone and gotten away with it? Or got caught out? Share your million dollar mistakes with an email to [2]On Call .
Get our [3]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] mailto:oncall@theregister.com
[3] https://whitepapers.theregister.com/
Re: Classic Errror
Very frequently I find myself also on call with someone who has 'really poor' phone signal in their house. Or, as they work in the City, any calls that come in whilst they're underground travelling home come to me as the next in the line.
Unfortunately for them as I'm not trained in the same products as well they stack up ready for when they do answer the phone.
Re: Classic Errror
In principal I'd agree with you but as IT Manager for a multi-national with a very small team I'm sort-of on call 24/7 (although the New Zealand office did have the decency to wait until 06:00 our time before ringing me recently). The plus point is that as I'm not actually contracted to provide the coverage I can, and will, ignore them if I wish, which I'll do if I've had a few beers or if the person ringing is an a******e (like the manager at a previous employer who called 3 times in the early hours of the morning because he had a spreadsheet problem)..
Re: Classic Errror
try doing out of hours planned maintenance with someone else remotely who has alcohol issues..........
Re: Classic Errror
Not IT related, but my experience with a pager. It was back in the late 1980's and I was on 24/7 call to arrange vendor inspection for a large multi-national oil company. I received a simple pager - it just beeped - no message facility back then. Also, no mobile phones (bricks installed in cars were just becoming popular show-offs). In rural Scotland, there seemed to be very few dead spots for that pager - out in the middle of nowhere with the family and I'd get the dreaded beep-beep-beep! Then it was a case of trying to remember where I last saw a phone box (or a pub - probably nearest but usually so noisy I'd struggle to hear anything on the phone). The rule was to phone back to the company emergency desk where I'd get a message regarding what was needed. Then a second phone call to find out what was really needed (all too often an arbitrary delivery deadline - kit must be on the Sunday boat - was no longer urgent as the boat had been delayed until mid-week). Finally, if needed, a third call to one of the on-duty inspectors.
If anyone wonders why I took the liberty of going gallivanting whilst on call - I'll reiterate it was a 24/7 setup and I didn't get any extra pay for the "privilege". I did it because the inconvenience was less than the mess I'd have to clear up when back in the office (because a vital bit of kit actually *was* needed on the rig and the downtime was costing $100,000 per day)!
I enjoyed that job (for five years before manglement decided the problems they'd hired me to fix had been fixed - needless to say, the problems started to return a year or so after I left because I'd been the fix - knife and forking the issues that didn't fit the norm). I learned a lot and made contacts that stood me in good stead for the rest of my career.
Re: Classic Errror
We had a consultant, he didn't have a home telephone, but the company put him on-call anyway...
So he would go home at 5, then toddle off with the dog at 9 o'clock to the next telephone box and call duty op, to see if there were any problems. If there weren't, he'd toddle off back home and go to bed.
It's not Friday the somethingth of Marchtober
Today is Friday, September the 9908th, 1993.
Re: It's not Friday the somethingth of Marchtober
Eternal September ended almost 16 years ago: https://betanews.com/2005/01/25/aol-pulls-plug-on-newsgroup-service/
Re: It's not Friday the somethingth of Marchtober
That's odd ... I didn't notice an improvement in the Usenet signal to noise ratio 16 years ago. Nor that of The Internet in general. In fact, I'd say it has been getting steadily worse Internet wide as time goes on.
So no, I completely disagree. It's still September.
Re: inflated pay packet due to an overnight call-out
During the Y2K project I was in, my team was correcting an application consisting of almost a 100 programs chained together (just as COBOL likes)
We were made aware that that particular batch chain would brake every single night, meaning an hefty on-call bill. And is had been going like that for years.
So I analysed the failure point and it was seemingly simple to correct - IIRC there were numeric conversions that would fail under certain conditions, conditions that would almost certainly be met once or twice per run.
That correction was deemed out-of-scope by our client rep, who, curiously, was also that application's IT boss.
I screwed the pooch with the OC pager.
I "accidently" set it to vibrate instead of making noise. So when I set it on my desk & turned my music on, it's no wonder I didn't hear the burr as it rumbled on the wood.
My first clue that the shit had hit the fan was my personal land line (before those newfangled cell phoney thingamajiggers) from my work... at 3AM. Oopsie!
Pick up & get an ear full of angry manager wanting to know WTF. I lied & told him that my kid had gotten sick all over me & I'd been in the shower. I had just gotten out when I heard the phone ring. "What's up, Boss?" I asked like a certain cartoon bunny. I think he tried to explode. Needless to say I got to the office a few minutes later (I had to whip those horses into a right froth, lemmetellya!) and got to work fixing the issue.
It didn't take long to sort the problem out & I stayed long enough to make sure that the fan wouldn't be collecting any more feces. I also remembered to return the pager to noise making mode.
For some strange reason I wasn't given the OC pager for the next few weeks. Woe is me!
Re: I screwed the pooch with the OC pager.
A colleague used to set the on-call pager to vibrate then clip it to his underwear when he was out dancing when on call. I always gave it a full de-tox with proper screen cleaner ( 1,1,1 Trichloroethane) when I was next on the rota.
Big financial institute blaming some poor sod for a failure of process? Par for the course.
Well Worded Oncall Agreement
I worked somewhere that had an on-call agreement that was so well worded that it said 'if the on-call engineer receives a call late in the evening and does not feel comfortable with their ability to drive, they use a Taxi or Private Hire service and be reimbursed by the company.' We all took this to mean 'if you have had a beer or 2, call a cab on us'!
Ah yes the joys of Pagers
In another industry back in the mid 90s, a former boss\owner of the company would visit our office a few times a week. My office was small, and my assistant had to use it too. My former boss was a walking pig pen. After two minutes in the office, he would take off his coat, empty his pockets on my desk, grab a beverage and generally disrupt the office. One day he visited, and my pager was on my desk, he routinely dropped his crap everywhere, and when gathering his crap also snagged my pager! I didn't see it happen, and I didn't miss it at the end of the day, but when I got home, I felt naked! I tried calling him at home, as few people could afford mobiles then, to let him know, no answer... Fast forward to 2am, and the cops needed someone at the office( a garbage truck backed into our building), my boss paged me, I would have loved to be there when he realized that he had my pager!
NO Entry ......
I had in 1993 an analogue Vodac NEC P4 (Which had replaced a trusty pager) as I was told being reachable 24/7 whilst on call would improve service etc ......
Apart from the concern that this thing was literally going to microwave my head, it suffered from the issue of a lack of masts for service coverage where I lived at that time, so would have variable service connectivity. However one night at about 11:30pm I got a call from Ops team that the trading system had borked, the software house had made a change during the day and it seemed it had unintended consequences - The software house was of course unreachable, so I was off to site in my trusty RS Turbo Fiesta :-)
*Could not use my 9600 baud modem (Connected via a VT terminal) as this particular system was not cleared for remote access.
As I drove out of the town I lived in, the local plod switched on their blues and twos and stopped me - As they explained it looked suspicious that a relatively young man driving a expensive car at that time of night (!!) - Could I prove I owned the car, that it was insured and show my driving license ..... Of course all I had on me was my company ID badge, which was not cutting it with them nor that I was on-call and responding to an urgent call. At this point the mobile phone rang (It was in my jacket pocket), I answered it and with mobile phones still being expensive and relatively uncommon I was then viewed even more suspiciously as I had all but confirmed I was a drug dealer in their eyes.
On the phone was the ops team asking for an ETA as they new that if the overnight run was not fixed in the next hour or so it would be very very bad for everyone (As lots of money would have been lost etc). I asked the ops team to call the IT director and get him to phone me to help explain to the police I was who I said I was.
10 minutes of the police doing checks and finding out the car belonged to a leasing company (And not me directly) it was looking like I was off to the police station for a more detailed interview, my phone rang and it was the IT Director - Not known for being particularly knowledgeable on IT, but it seemed he played golf with the Chief Constable and told the local plod he was going to call him to get this issue resolved.
I was was given the go ahead to depart but with strict requirement to bring all documentation to the Police Station in the next 24 hours - I managed to get to site, fix the issue with the overnight run (Which was classic software developer stupidity).
But first time I was grateful that I had the mobile phone and not the pager, plus was able to add in an additional hour for the chat I had with plod :-)
Classic Errror
Never, ever drink on call, you can guarantee that will result in a call, a pretty nasty one too.
I used to like our on call procedure, we had pagers, but, when out of network, they didn't receive any notification, even when they re-joined the network, so you could show the pager and be like "didn't get it boss".
Was a good couple of years I got on call bonus whilst living in a network deadspot..