IT phone home: how to run up a $20K bill in two days and get away with it by blaming Cisco
- Reference: 1679301130
- News link: https://www.theregister.co.uk/2023/03/20/it_phone_home_how_to/
- Source link:
This week meet a man we'll Regomize as "Elliot" whose story comes from a long-ago almost forgotten time when ISDN was the latest and greatest way to send data long distances at speed.
A refresher: ISDN was often mocked as standing for " I ntegrated S till D on't k N ow, but the Integrated Services Digital Network offered blazing-fast speeds up to 128 kilobits per second. Yes, per second . Can you even believe such a magnificent thing was posisble?
[1]
Unfortunately, Elliot tells us, in the golden age of ISB all that speed came at a cost . Providers would charge flagfall per call and also charge for the length of the call. That's OK – cost of doing business, you know – but only if everything works as it's supposed to.
[2]
[3]
Say, for instance, your enterprise is using an ISDN line as backup for if its dedicated WAN fails over. If everything is configured correctly, the WAN fails, the ISDN springs into action and connects, and the IT guy fixes the WAN and switches it back. ISDN goes back into standby node and nobody even notices there was a problem.
If things are not configured correctly, the WAN fails, the ISDN springs into action, fails to connect, tries again, fails again, tries again … you get the picture.
[4]
The rig Elliot oversaw could do that twice per second, at five cents flagfall per connection attempt. If no-one stops it, it can keep doing that for a weekend. You can imagine how it can add up.
[5]Techie wiped a server, nobody noticed, so a customer kept paying for six months
[6]Don't worry, that system's not actually active – oh, wait …
[7]Backup tech felt the need – the need for speed. And pastries and Tomb Raider
[8]A tip for content filter evaluators: erase the list of sites you tested, don't share them on 100 PCs
Elliot doesn't have to imagine, because he twice – twice! – witnessed such "ISDN storms" and saw bills in excess of $20,000 as a result. Not a fun Monday at all.
Naturally, his verdict each time was that it was a fault in Cisco's IOS, and that an upgrade was clearly required.
Smooth.
Less easily excused was a colleague of Elliot's, who realized that a customer's 56K leased line between the US and Germany wasn't nearly as fast as the ISDN backup line (56 being notoriously less than 128).
[9]
This colleague would deliberately switch over to the ISDN (which was meant as a backup ) every time he felt he needed faster internet access. Flagfall was not so much an issue, but the international ISDN calls at four dollars a minute added up pretty quickly.
At least that one wasn't Elliot's fault.
Ever managed to create a ridiculously enormous bill with a seemingly trivial mistake? Don't worry, we're not here to judge – we just want to tell everyone about it. Drop us one of those [10]newfangled email things to Who, Me? and we'll use your misfortune to brighten everyone's Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZBg805PzPYAb5cAjbP43xwAAAE4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg805PzPYAb5cAjbP43xwAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg805PzPYAb5cAjbP43xwAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg805PzPYAb5cAjbP43xwAAAE4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/03/13/who_me/
[6] https://www.theregister.com/2023/03/06/who_me/
[7] https://www.theregister.com/2023/02/27/who_me/
[8] https://www.theregister.com/2023/02/20/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg805PzPYAb5cAjbP43xwAAAE4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Every good deed....
So you're saying the current readings are higher than before
"Flagfall was not so much an issue, but the international ISDN calls at four dollars a minute added up pretty quickly."
Tell that to kids these days and they won't believe you.
Maybe they will if they travel.
I just noticed that the packages of Orange in Belgium include free local calls, and free calls in Europe, but with a little evil catch:
Call from Belgium to anywhere in Europe are NOT free. None of their packages offer it, but it's phrased in such a way that it appears as if they're talking about out-of-Europe calls so your first incling of this deception is when you get an alert of out-of-package spend, and when you get the bill it's hard to work out where exactly that occurred - it's one of the most interesting games of obfuscation I've ever seen.
I haven't had time yet to report this, but I think I will, just to see the excuses fly back and forth (and to see who they have to bribe to keep this scam going as it's Belgium, after all).
They might believe it if they try to visit Japan. Visitors from UK on Three network:- £2/min for calls + £3/MB for data.
I managed to rack up a 1k ringgit bill during just one week in China from my colleague calling me to see how a job was going. Reimbursement came out of his budget within an hour of me getting back to the KL office...
Needed a few of these --->
Actual EU benefit
It has long been a standing recommendation when visiting east asian countries to buy a local burner SIM card instead, which gets vastly better pricing than whatever roaming service you choose.
EU-roaming price ceiling has been one of the (few?) tangible benefits of EU regulations, which limited the price gouging when travelling within the 27 states.
I had the reverse: a legit bill being refused
At the time, I worked for a consultancy who had developed a product to be installed in some test pharmacies in the US.
On arrival there I did a final inspection of the gear and found it had a virus infection. I spent quite some time on my company phone with our support guys in the UK getting rid of the damn thing (because it was test gear there wasn't a backup, and because I was supposed to only install them I didn't carry anything with me for a backup myself and it was the weekend). There was no way I was going to install a virus infested system into a HIPAA compliant network, so it either had to be fixed or I was coming back to the UK with the units.
This was in the days of the mass ripoff of phone customers, so when I returned I ended up with a £1k bill, which the company refused to pay back - taking the opportunity to get some payback for me being in the last weeks as I resigned (yes, they were that petty).
Revenge is best enjoyed cold, though, as I am now looking at some very large projects, so guess who doesn't get to bid? Saved a grand, so now lost a few million.
I can be petty too..
Headline
I'm slightly sad that the headline wasn't "WAN things go bad" or "WAN things get expensive"
Mobile dongle and ISDN
I had a sales colleague back around 2008/2009 who loved tennis.
He happened to be abroad at the time of Wimbledon so used his USB dongle to connect and watch hours of tennis.
Shortly after arriving home, he was confronted with a £27,000 mobile bill. He just said "well no one told me I couldn't use it abroad and I can't afford that so you'll either have to pay it or sack me". Since he was their top earning salesman they sucked it up and paid it but as you can imagine, there was a rather strongly worded email came around shortly afterwards.
Even earlier than that, back in the mid/late 1990's we had a couple of sites that used ISDN for backup. They'd been in years and in fairness rarely ever used.
We came to do our annual failover test and...nothing.
A bit of investigation later and we found that the head of personnel who got all the phone bills had looked at them for some time and decided "hey these expensive ISDN things are never used so I can save some money for the business..." and had them terminated. Of course, he never asked anyone...
Mind you, I will raise a glass to him and his team because they were a true personnel department - they worked on behalf of the staff not the business. In fact, I think that was the last place I worked that had a personnel department before they all became HR departments.
Re: Mobile dongle and ISDN
My work is in a place where you can get three countries' phone signals. When iPhones were new and shiny, people loved doing new and cool things like streaming videos. Those working near the border often had their phones flip over to another country's network leading to some massive bills...
Re: Mobile dongle and ISDN
@Korev
A co-worker got some complaints about his international use of a phone, when as far as he knew he had not left the US. He had gone on a boat tour along the Detroit River, and there were spots where his phone connected to a Canadian cell rather than an American one. Our boss grumbled, but did approve the bill.
Microsoft "Active Desktop"
I had a similar problem, sometime about 1997. Microsoft had introduced [1]Active Desktop , which "allowed users to add HTML content to the desktop". This content came over the Internet, even if there was nobody sitting in front of the computer overnight. Our ISDN bill shot up until we worked out what was going on.
[1] https://en.wikipedia.org/wiki/Active_Desktop
Ah, yes.
Known as "3D phone-bills" in a marketing message I put out, back in the day.
I'd just started out freelancing doing various networking stuff, and a lot of companies were being sold ethernet bridges with ISDN dial-on-demand to link sites together. Which resulted in exactly this, if you didn't very carefully set up the network devices to block broadcast and multicast traffic.
I made a good living following around after bigger networking companies and fixing this sort of cock-up. Simpler, far-off days...
GJC
Thanks for the Monday pick me up
I'll be phoning home now, Elliot!!
+1 for the Extraterrestrial reference! And have a bag of Reese's Pieces for later!
Not just in the office
I had ISDN at home and a new laptop configured by my employer. I worked from home a couple of days a week, and often left the device switched on at evenings/weekends. After about a month the postman knocked and handed me an A4-sized parcel about two inches thick. It was my itemised phone bill. The laptop had been sent out with IPX/SPX installed which was firing up a data circuit and trying to make a connection every minute. Or perhaps it was every second. In any case it led to many thousands of very short calls, each of them billed at a far higher rate than their length would suggest due to a minimum connection charge.
I can't remember how large that bill was but it was sufficient to get the company to reconfigure networking on that PC very quickly indeed.
The good old days :)
I remember installing a device from 3Com called an ISDN LANModem at home, which was a four port hub with a built-in ISDN modem, capable of dialling the ISP on-demand. HUGE risk of massive phone bills, but it was amazing being able to use dialup internet across home and work PCs at the same time.
On the phone bills - there was the engineer who installed a server under his desk, hooked up to an ISDN-2 line that was installed to dial in to customer locations. He and his buddies ran up HUGE bills dialling premium rate porn BBS' via that server before it was taken away from them. It was also completely infested with virii. And of course there was the guy who worked out he could forward a random desk phone at the office to his girlfriend halfway around the world and call her for cheap. Of course, he didn't factor in access logs to work out who was setting/clearing the forward...
Anonymous to spare my former employer blushes ;)
Re: The good old days :)
Years ago we had a story go round the company that one of the cleaners had discovered that manager's phones had international calls enabled (something barred to the vast majority of us poor plebs) and had started calling his family back home. This snowballed in to him acting as a 'telephone exchange' for other expats, with him holding the handsets of 2 phones together
Happened to me
We were opening a new office and the boss didnt want to pay for a leased line, so decided to do ISDN. Before the new office was commissioned we decided to test between 2 local offices. Said boss marched down and said 'what's this £5k bill for'. When he found out it was for call charges, we got our 2Mb line. Oh, and a fibre for the local offices.
Anyone remember Sonix Arpeggio plus/lite? Happy days.
Ah, the good'ol days..
ISDN was often mocked as standing for "Integrated Still Don't kNow, but the Integrated Services Digital Network offered blazing-fast speeds up to 128 kilobits per second.
Never heard it called that, but more usually "Invention Subscribers Don't Need". First encounted it back in the late '80s while working for the Phone Company. Engineer turned up to install a line in our comms room. No idea who ordered it, or why, untill discovering that every site was to get an ISDN line because nobody else was ordering them and it made the installation numbers look a little better. We got told we could use it for a Group 4 fax machine. Then told we couldn't requisition one of those because nobody else was using them. When I left, it still had a post-it over the socket saying 'Not a POT'.
I don't think the UK had a flag charge, but we had a thing called 'telegraphing', which was to abuse the D-channel signalling to send data that the billing system at the time didn't measure. A well-known POS credit card terminal provider used that loophole, despite grumblings from the Phone Company, and I think later, Oftel. Other shenanigans came from the good'ol 0845 ISPs. At the time, 0845 numbers had a revenue sharing agreement between wholesaler and service provider. That had tarriffs for short and long duration calls, partly as an anti-telegraphing measure, and from memory, the outpayment was something like 2.4p/min for long duration, and 4p/min for short.
Ever wondered why there used to be so many authentication failures on good'ol dial-up ISPs? Call connects, billing triggers, RADIUS says 'nope', and collect those pennies..
Also used to hate ISDN backup solutions, especially when one large telco decided it would be a GoodIdea(tm) for us to pick up the costs of customer's ISDN calls when their leased lines went down. Took a while to convince the product manager that this was a risk, and to add an alert to the fault management system when ISDN lines were activated. Also had a lot of fun with unmanaged 'solutions', but quickly came up with a boiler plate response for users complaining their bills were so high. This was long before MS's auto-update attempts to phone home, or spam telemetry, but MS has always been very chatty with broadcasts and various discovery packets hitting dial router interfaces and bringing up the ISDN lines.. Which is still a big issue with telemetry spam, but mostly a cost issue for mobile Internet devices. Still amazes me that regulators haven't stomped on that one, but then mobile operators make a lot of money when they charge for data usage, and don't give customers control over their devices to disable all that junk.
ISDN...
Back in the day we had a demand from senior management to connect a Dutch office to our UK AS/400 (via 5250 terminal emulation on a PC). We did warn them about the costs but it "must be done"!
The only viable option at the time was ISDN, there was no way they'd provide the funding for leased lines at each end. Despite being told otherwise they'd naively assumed the connection would only be up whilst the user was actually typing.
3 Months later the Dutch were restricted to only connecting for a couple of hours a day due to the high bills, which didn't last long, they found an alternative workaround.
Many, many years ago when the Internet was much fresher, we dipped our toes with LAN access using Demon Internet and a device called Instant Internet. It shouldn't have cost too much as the line was supposed to drop after five minutes of inactivity.
However, either Demon or the Instant Internet (I can't remember which) was sending a keep alive or was checking for email every five minutes and so the line never dropped and the call charges were much higher than expected.
Took quite a while to track it down.
Not ISDN...
Enjoying the ancient "oops I've run up a 5-figure++ bill by accident" from the times of ISDN and early roaming days.
I'm wondering how many simlair modern stories there are involving badly configured cloud implementations....
Every good deed....
I went to commission some large machinery for a waterworks, drawing water from a river for treatment. All the important people were there, including the end-user and his consultants. All went well and the consultants remarked how good their concept had been, hardly using any energy. I went to have a look at the electricity meters which indeed were turning much slower than expected. A gentle 'tap' on the side of the meter sent them spinning away at a more likely rate; much to the embarrassment of the consultant and dismay by the end-user/payer......
"I wish you hadn't done that."
Further percussion did not revert the rate. Good job for us though ---->