Suits ignored IT's warnings, so the tech team went for the neck
- Reference: 1699601290
- News link: https://www.theregister.co.uk/2023/11/10/on_call/
- Source link:
This week's tale comes from a reader we'll Regomize as "Bruce" because his story concerns his time working for an Australian bank's internet infrastructure team.
Bruce joined when internet banking was in its infancy, so his team was growing fast. So was their workload.
[1]
"We controlled/owned/looked after everything from ISDN link to the cable that was connected to the network team's internal switch," Bruce explained.
[2]
[3]
But Bruce and his colleagues still had time for planning. And one day he was assessing current user numbers vs demand vs link capacity, then the trends for future numbers, and realized it would not be long before that ISDN link was more than half saturated.
With internet banking taking off, the risks were obvious. So Bruce and his mates put together a position paper explaining why ordering another ISDN line was a good idea and gave it to the CIO for approval.
[4]
This was vastly sensible – in those days ordering an internet link could take months, so Bruce knew that getting this order in ASAP was essential.
The CIO forwarded the request to the bank's Executive Committee, which promptly rejected it because it was clearly wasteful to buy an extra link when the current one wasn't even half-used!
Bruce and his colleagues were scolded for their wasteful ways.
[5]
Once usage ticked past 50 percent, the IT team again made a pitch for an ISDN order. And were again rebuffed – with instructions not to ask again until utilization was nudging 100 percent.
Bruce knew what would happen at that point: the connection would max out, user experience would become dire, and IT would be blamed for the bank's failed entry into the world of internet banking.
He also imagined being dragged out while on call to fix the inevitable network errors. And then being reamed for the extra cost of having the ISDN link installed as a rush job.
[6]Ask a builder to fix a server and out come the vastly inappropriate power tools
[7]Making the problem go away is not the same thing as fixing it
[8]Workload written by student made millions, ran on unsupported hardware, with zero maintenance
[9]Police ignored the laws of datacenter climate control
The IT team hatched a cunning plan to make this the executives' problem.
"The suits and their associated hangers-on all resided on one floor of Head Office with their own subnet of IPs," Bruce explained. That network topography made it a simple task to throttle their connection.
"The first week, we throttled them down ten percent. Each successive week we withdrew another ten percent," Bruce confessed.
After a month of these shenanigans, the IT team again requested the additional ISDN line.
It was approved – along with the extra payment for expedited installation, and an explanation that if the suits could understand the frustration of slow connections, customers would also chafe.
Once the new line was connected, the IT team removed the throttle, "and left the suits to congratulate themselves on fixing the 'internet problem'."
Have you used the powers of IT to fool the non-technical? If so, [10]click here to send us an email and we'll share your secret to brighten up a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZU4NV1VCSkg3zO2C1h4bSQAAAIw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZU4NV1VCSkg3zO2C1h4bSQAAAIw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZU4NV1VCSkg3zO2C1h4bSQAAAIw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZU4NV1VCSkg3zO2C1h4bSQAAAIw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZU4NV1VCSkg3zO2C1h4bSQAAAIw&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/10/27/on_call/
[7] https://www.theregister.com/2023/10/20/on_call/
[8] https://www.theregister.com/2023/10/13/on_call/
[9] https://www.theregister.com/2023/10/06/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: The interneet
We'll all have to Jen up on that one...
"The first week, we throttled them down ten percent. Each successive week we withdrew another ten percent," Bruce confessed.
After a month of these shenanigans, the IT team again requested the additional ISDN line.
So the execs got board waiting?
similar
Except that we added a summary of sites being visited by the suits..........
Even in day of Jennicam, no one wanted that to become common knowledge in the company.
Re: similar
Ah, the heady days of Jennicam and the BOFHCam...
Sadly, those days are long gone... :(
Quite a few years ago, I used to be the IT Manager for my company. One day I got a call from the CEO saying that all of his e-mails had disappeared from the server, and I needed to get it sorted out PDQ. This was a completely unexpected problem and I had to investigate the root cause of the problem before I could fix it.
The root cause turned out to be easy.
The CEO also owned a flat in Majorca and had gone out for a week or so of R&R; however he had taken a brand-new laptop with him so that he could stay in touch electronically. While he was out there he had set up the email client on the laptop, but had decided selected POP3 as the protocol instead of the (mandated by IT) IMAP (we did not use Microsoft's e-mail server having had a bad experience with it). not surprisingly, the POP3 client then sucked everything down from the server, deleting the server-side copies as it went. When the CEO came back from holiday, he left the laptop in Majorca so that he did not have to carry it back and forth.
Once I had realised what had happened I (a) restored the CEO's mailbox from a backup taken before he went off on holiday, noting that anything that had come in afterwards was stuck on his Majorca laptop; and (b) told the CEO what had happened, whose fault it was (his), and gave him strict instructions on how to not repeat this screw-up. Thankfully he listened!
Excellent, except that I find that last sentence rather hard to believe!
The power of controlling the internet
As a charity we used to have board and trustee meetings where it became apparent people weren't paying attention (in the course of one meeting I received an email stating we had policy which contradicted what IT had recommended and I knew hands on leadership were pushing for and as well hadn't been voted at that point.) Very awkward for us getting the job done and telling recalcitrant volunteer workers to comply. From that point on I got the GDs permission to have a wifi outage during meetings. Attention increased, and under table chat was curtailed.
Re: The power of controlling the internet
Really? Turned of the wifi? We used to sit spread out on the back row and play a networked fps.
Anon because while I don't work there anymore, they still do.
Not just IT. If you want to get a problem solved there are two options:
1) Persuade manglement the solution you need is their idea, or
2) Make the problem theirs.
Either way, they’ll end up pushing for the solution to be implemented and, unless you manage to really screw up, you’ll earn Brownie points.
The interneet
This story reminds me so much of the IT Crowd and 'The 'Internet' episode.
Thanks for the chortle.