How do you call support when the telephones go TITSUP*?
- Reference: 1638520266
- News link: https://www.theregister.co.uk/2021/12/03/on_call/
- Source link:
We're back in the 1990s for a tale from "Frank" (obviously not his name), who found himself dealing with a problem when all the phones in his company failed. Sure, it meant the On Call phone was quiet. Too quiet.
The company had its own PABX (Private Automatic Branch Exchange), enabling users to call each other and, most importantly, place calls to the outside world. Except for some reason that morning phones were dead.
[2]
"And in those days," Frank explained, "that meant the company was defunct. No email existed except internal."
[3]
[4]
Although nobody could directly place a call for help, Frank's team immediately began investigating. It took half a day, during which time those used to having handsets welded to the side of their heads had to find other ways of looking important (doubtless via the clunky mobile telephones of the day), but the issue was eventually discovered.
"The problem we found after half a day was..." said Frank, pausing for effect (dun, dun, dun ), "the protocol printer for printing all calls and the fees being generated off them had run out of paper."
[5]
A delightfully low-tech problem, and one that the team had taken a while to understand because, quite frankly, it hadn't been fully understood that this was how accounts tracked people and just how significant the printer was. Someone used the phone and the call was logged and the cost spat out by the printer. However, if the logs could not be printed then the designers of the PABX had made the decision that it was better that nobody be allowed to use the phone at all.
[6]Swooping in to claim the glory while the On Call engineer stands baffled
[7]A lightbulb moment comes too late to save a mainframe engineer's blushes
[8]The Ministry of Silly Printing: But I don't want my golf club correspondence to say 'UNCLASSIFIED' at the bottom
[9]Say what you see: Four-letter fun on a late-night support call
Pop in some more paper, and everyone was happy. Sort of.
There was a flaw in the system, one Frank discovered quite by accident. As he inspected the printouts from the recovered printer, he found no record of his own telephone activities. Even after a particularly expensive call (he did not elaborate), there was no record.
It transpired that while the PABX would stop working for all phones if the paper ran out, it would only actually print fees for the first 100 phones in the company (requiring an expensive upgrade to do so). Phones beyond that first 100 would still work (assuming the printer hadn't dropped offline). But there'd be no incriminating printout. Frank's phone was one of the lucky over-100.
"I kept that to myself, though."
[10]
After all, nobody wants to make the walk of shame to the beancounters' office.
Ever fixed one problem, only to find another that you kept very, very quiet about? Or summoned support without access to phone or email? Tell us all about it with an email to [11]On Call . ®
*Total Inability To Service User Phonecalls
Get our [12]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yan43gyg86w6OKVqUE210QAAAIw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yan43gyg86w6OKVqUE210QAAAIw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yan43gyg86w6OKVqUE210QAAAIw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yan43gyg86w6OKVqUE210QAAAIw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2021/11/26/on_call/
[7] https://www.theregister.com/2021/11/19/on_call/
[8] https://www.theregister.com/2021/11/12/on_call/
[9] https://www.theregister.com/2021/11/05/on_call/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yan43gyg86w6OKVqUE210QAAAIw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:oncall@theregister.com
[12] https://whitepapers.theregister.com/
Re: Divert
What a strange thing to do. Other than costing the company money, why would she have done that?
I worked at an insurance company where it was found out that some of the call handlers were calling a silent number to make them look busier than they were. Apparently if you rang it, it connected but as the name suggests was silent.
Re: Divert
Sounds like she would set her phone to divert to an overseas number, then that would allow her to make an overseas call by calling her own local number, and presumably only have to pay for a local rate phone call? ¯\_(ツ)_/¯
Re: Divert
Yes, it cost the company money, but conversely she got "free" overseas calls to her family and these calls apparently lasted a good 2-3 hours. This was the early 90s when overseas calls could be quite expensive - especially ones lasting 2-3 hours!
I never knew about the silent numbers, but a pretty sh!tty way to annoy your hard working colleagues...
Re: Divert
To be fair, your colleagues will only be annoyed if they find out.
Reclaiming Private Call Costs
I had the joy of managing a Count Council Tech Support team, when the PABX's came under my remit we were saddled with the process for reclaiming private call costs. This was an honesty based system which relied on the staff ember remembering to report it then one of the switchboard team would search for the call to that number and work out the call cost by hand.
This cost 10X more than the cost of the call. I managed to automate the searching and tariff calculation and managed to reduce the cost of the process to the point where it only cost 3X the cost of the call. we had already barred premium rate and overseas calls for all but selected staff members and my own view was that if a team member was spending hours on the phone for personal calls than this was a disciplinary matter about 'theft of time' rather than an issue trying to recoup 45 pence for a long call to a local number.
I was put under pressure to develop code to spot repeated calls to 'personal' numbers but I refused and cited the rare personal calls I would make to call home and tell my wife to put the kids to bed as I was working late.
I pointed out that if they tried to charge me for that call I would just leave work, go home and put the lids to bed rathert than work late, again, and miss eeing them.
Re: Reclaiming Private Call Costs
We had an honesty box system. Every month a printout would come round, and for each phone there would be a list of external numbers etc which were direct dialled.
For our phone, there was always about 10+ hours a month to a national number. Both me and my office mate(Martin) denied it.
We phoned it ... and when someone replied, Martin said "oh hello Mum.. we were just checking the number, I hadn't realised it was your work number".
Re: Reclaiming Private Call Costs
I worked in a university office in Edinburgh which had a fax machine. Ask the 'rents, kids.
As was traditional there was a jotter beside it in which one wrote down number called, pages sent and, crucially, work or private. I had to send a renewal notice to my insurers for proof of NCB so I faxed it and, very honestly, wrote down the details in the jotter. Two pages including cover sheet, UK number.
A month later I got a manually produced invoice for 4p. I decided to wait and see what happened.
A month later I got a manually produced reminder. Still 4p.
I sent them a cheque, for 4p.
They paid it in.
Re: Divert
I've had that issue much more recently. A company with an IP phone system contacted me over their bills. They were being billed for a lot of overseas calls. When I checked the CDRs I found that all the calls were from one number to a number in Dubai.
So I checked the setup on the particular extension that was making the calls sure enough it had a forward set at the handset level to a Dubai number. The interesting thing was I could see no outgoing calls from that number other than the forwards suggesting that the phone was not in use. So I asked the company who that extension belonged to. "You must be wrong" says the customer "that's Bob's phone and he can't be making any calls. He's seconded to another branch at the moment."
You can guess what came next can't you? The branch in question was in Dubai and the number the calls were forwarded to was Bob's phone in the Dubai office. Bob's email signature had his UK number on it so he decided it was easier to forward his phone than get his email signature changed.
The problem for the company was that they took their UK phone system from a provider in the UK and their Dubai system from a provider in Dubai. So the calls from the UK broke out onto the POTS network in the UK and were therefore billed at international rates. And the best bit? A lot of the calls being forwarded were originating in Dubai and many were made by Bob's colleagues in Dubai. So the company was being billed for two international calls for a call being made locally in the Dubai office.
Needless to say Bob was not very popular with his management.
Cost savings
Back in the dim and distance past, I was IT support for a company that provided services for Oil Companies with offices around the world.
Most of them we quite small with less than a dozen or so people. We tended to use small Phillips PBXs for them, wonderful things.
One place in Africa had HUGE phone bill, even more than we were expecting, so I came up with a plan...
Everyone would have a PIN code that would report back the use of each code AND set the guard hut up with local calls only.
I never did get the reporting system to work properly, it was too flaky to be reliable, BUT just saying that the call billing would be logged to your pin and restricting the calls worked, and the costs dropped to about a 1/3 of what they were, and the first month savings paid for the ticket and time to do the work...
Turns out because the guards previously had unrestricted calls, they would supplement their meagre income by selling international calls.
A few years ago, I got called to see my boss. Apparently, the switchboard had notified him that I had made a *lot* of calls to india, which were very expensive. Can't remember the exact details, but there where hundreds of pounds worth of calls made from my desk phone.
I was called to explain this. I explained I did not make personal calls on my office phone (I have a mobile for this), and that I did not know anyone in India. While I did have international calls enabled on my desk phone (by default, international calls were disabled on the company phones), this was purely so I could deal with some of our US based suppliers. I did not call any other country, and only called the suppliers on rare occasions. He didn't really believe me so I asked if he could give me a print out of my previous calls, so I could look into it.
He did, but had to request it from the Switchboard, so it took a couple of days.
When I got the print out, I spent about 5 minutes going through it before I noticed something odd. All of the non US international calls were made when the building was closed. While I often used to work late, even if I'd have wanted to, I wouldn't have been allowed in the building when it was closed. The only people allowed in were the security and cleaning staff, so I highlighted all the calls made while the building was closed, and took the printout back to my boss. He apologised, and said he would deal with it.
Never heard any more, but I did notice that a security guard I had considered a bit of a friend vanished after that date, so I suspect it was him.
Have heard of something similar...
A cleaner worked out that managers had international call access, so had started by calling home... and, because the manager had 2 phones, then progressed to acting as an 'exchange' for friends and family using the old skool method of simply holding the 2 handsets together
At my first ever job, the office had recently got a newfangled telephone system, where you could set up your phone to forward calls to another extension. One day, someone in the office wondered..."If I set up my phone to divert calls to you, you set up your phone to divert calls to Bob, and Bob sets up his phone to to divert calls to me....then Alice tries to call me...I wonder what would happen?"
The answer turns out that it would break the PABX for the entire building, resulting in an expensive call-out for an engineer.
In Ye Olde Days I used to while away boring evenings by dialling 031 ... 041 ... 031 ... 041 ... 031 to see how many it took to tie up all the lines between Glasgow and Edinburgh and get "number unobtainable". The answers was "surprisingly few". Generally between 12 and 20.
A setup like that must have been ancient even in the nineties, as such I can't believe the issue had never occurred before.
Or brand spanking new. I used to deal with early Cisco callmanager stuff that couldn't do a lot of what long discontinued PABXs could handle fine.
It was like some dudes had just hacked together some code that was a bit like a PABX, bigged it up massively, and then sold the company to Cisco to handle the fallout.
Also time critical shizz on windows 2000? Do me a favour!
Beancounters not on the job
So the printouts only handled the first 100 phones installed (or whatever) and the beancounters never noticed that the invoice from the phone company was more than what they had records for ?
How did they explain away the difference ?
Sloppy job, there. Very sloppy.
Re: Beancounters not on the job
You assume that there was some procedure in place to actually look at the printout.
I've worked in places where they would have had formal, documented procedures to retrieve and file the printout religiously each week, but probably not to act on it.
Actually....I can think of one place that probably would have had a procedure to take the printout, make a photocopy of it, and then file both
(yes, I used to do quite a bit of contract work for local government departments)
Beancounters
Back at around that time, when Premium Rate meant calls made before 1pm, the Head Beancounter used to come round every month with the call log printout requiring an explanation for any calls made during the morning. I'm sure the time spent doing this cost far more than any saving in bills.
When we moved premises and I specified the new Nortel phone system it was "too expensive" to include call logging!
"...during which time those used to having handsets welded to the side of their heads had to find other ways of looking important (doubtless via the clunky mobile telephones of the day)..."
I've often wondered how office staff would continue to look busy nowadays if their computers weren't working!
Divert
I know someone who would set-up the divert function on a Friday night on her desk phone, and during the weekend, call her phone which diverted her call overseas. Obviously she was found out but she still worked for us so I expect it was more of knuckle rap than anything more severe.