You're wrong, I'm right, and you're hiding the data that proves it
- Reference: 1718955073
- News link: https://www.theregister.co.uk/2024/06/21/on_call/
- Source link:
This week, meet a reader we shall Regomize as "Brian" who told us he once worked for "a large, international company with a web presence that we shall call 'Circus Maximus'."
Brian managed an application group and worked with an IT manager he called "Spartacus."
[1]
"Together, Spartacus and I worked to satisfy our corporate end users in the US and in multiple international locations," Brian told On Call.
[2]
[3]
The applications Brian oversaw ran on servers in the US, where users accessed it via a browser. Brian's message to On Call suggests it was quite a lightweight affair – not much passed over the wire other than HTML and what he described as "very small icons for a handful of buttons."
Brian and Spartacus ran a tight ship, so users were pleased with service levels.
[4]
But one office – which Brian described as "Gaul" – was not happy.
Despite the app's modest nature, users in Gaul claimed their WAN link was maxed out and Brian's code was to blame. "They kept reporting terrible response times and complaining the app was bad, the network was bad, and this meant that they were being treated poorly and they could not meet their productivity metrics."
Brian therefore put his app group to work reviewing code and looking for needlessly chatty communications.
[5]
None were found – as you'd expect, given that all other offices were problem-free.
As the dispute with Gaul raged, Spartacus was sent there to investigate the situation.
"He joined the chorus whinging loudly about the crappy app," Brian lamented.
[6]We need a volunteer to literally crawl over broken glass to fix this network
[7]I didn't touch a thing – just some cables and a monitor – and my computer broke
[8]Thanks for coming to help. No, we can't say why we called – it's classified
[9]Bad vibrations left techie shaken up during overnight database rebuild
Rather than put up with that, Brian decided to gather some actual data to inform the conversation.
"I challenged Spartacus to put a data analyzer on the WAN to see what kind of traffic was moving and what was causing the slowness before the whinging reached the Emperor," he told On Call. “
Spartacus agreed to do so and, a week or so later, the problem went away.
So did Spartacus.
"He went silent and would not respond to my emails," Brian told On Call. "So I hunted him down, cornered him in his office, and grilled him."
Spartacus grudgingly revealed that the analyzer identified that an IT guy in Gaul was routinely downloading copies of Linux binaries over the WAN during business hours, thereby saturating the link.
When the Gallic techie was told " Arrêter de faire ça " he did, tout de suite .
"I never did hear if the French admin kept his or her job, but Spartacus and Gaul stopped complaining about application performance over the WAN," Brian wrote.
Have you proven a colleague wrong and emerged triumphant? If so, [10]click here to send On Call an email so we can say "bravo” and bring your ballad of beneficence to readers on a future Friday.
Remember: On Call always needs ideas, and is closing in on a 500th column! You can help us get there – and beyond – by sharing a story. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZnVPPHqCXA7nJD2VbyEW5QAAAIQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnVPPHqCXA7nJD2VbyEW5QAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnVPPHqCXA7nJD2VbyEW5QAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnVPPHqCXA7nJD2VbyEW5QAAAIQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnVPPHqCXA7nJD2VbyEW5QAAAIQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/06/14/on_call/
[7] https://www.theregister.com/2024/06/07/on_call/
[8] https://www.theregister.com/2024/05/31/on_call/
[9] https://www.theregister.com/2024/05/24/on_call/
[10] mailto:oncall@theregsister.com
[11] https://whitepapers.theregister.com/
Re: Brian may not have been the messiah ...
But it was a mess allright.
Re: Brian may not have been the messiah ...
Look on the bright side, he didn't get crucified by his users.
"more often than not it's users and managers who are in the wrong when IT goes awry"
That's because the users and managers keep their job and are free to mangle things again.
An IT guy who fouls up is more often than not fired from the company, so he doesn't get much chance to foul up again.
Slow WANs
Back in the day, as they say, slow WAN links could be a pain in the proverbial.
I inherited a 2mb dedicated internet connection which was supposed to be sufficient for Citrix traffic from two remote locations, all our internet usage and replicating backups to off-site. It was sufficient, just, until the backups kicked in late afternoon so first job was to shift that traffic off to the failover ADSL connection.
One day I'm suddenly inundated with calls about the system going very slow at the remote locations so I logged in to the main firewall and sure enough the WAN connection was being completely swamped. Further investigation showed the excess traffic was coming from an akamai server. Once I located the recipient PC and had strong words with the user concerned we isolated the problem.
He was trying, not unreasonably, to update his TomTom SatNav but there was an issue with the transmissions from the server. The update was split into packets, if the server didn't receive a response for a packet it would send it again, however due to the slow connection very few packets were ever acknowledged in time so the server just kept sending more and more data until the line was totally overloaded. Having killed the update at the user end, and told him not to try again, I blocked the relevant servers on the firewall and eventually the packet flow stopped.
The same user also decided to have the Sky website as his browser homepage so he could keep up with the football news. The multiple videos also caused serious congestion on our connection (something which also occurred on our Parallels RD servers thanks to Microsoft Edge's default home page - easily tweaked with a GPO).
Re: Slow WANs
In that scenario I'd say: Everything which is not Shitrix, i.e. the target IP/subnet, gets offloaded to the ADSL line be default. Just a simple change of the routing table. With some adjusted metric settings if you want the normal internet to fall back to the 2 MB line on a ADSL outage.
Did the nerd burn the ISOs with Nero Burning Rom?
That might have been a bit fiddly?
Reminds me of this comic
on [1]Commitstrip .
[1] https://www.commitstrip.com/en/2020/09/24/not-the-same/?
Slow WAN leads to trouble
Long ago I worked for an organisation which had three issues: a slow WAN, a junior techie who thought he knew everything, and ill-disciplined users (academics).
WAN was slow; sometimes stopped because junior tech "fixed" it. But I got it stable, and locked junior out of the router, which connected us to head office and through them to the world.
Wonderful - break for xmas. Come back to work to a flurry of messages from head office - why so much data use on xmas day? Quick investigation, look at IP numbers and I knew it was "Ulrich's" machine. What was going on? He decided that on xmas day the WAN would be all his, so he went in to the office and spent all xmas day downloading pr0n. I found 100s of gigabytes of it on server and workstation drives.
He was irreplaceable apparently - so he stayed. Under very strict conditions about hours of access.
Months later I discovered that junior tech who was told to clean up the server drives that Ulrich had used had made many copies of really bad stuff for his mates. He was not irreplaceable in the slightest.
Why did/do so many people endlessly download Unix binaries other than to check their connection is achieving the claimed speed by keeping it maxed out?
Linux binaries?
Nonsense.
It was a C* downloading pR0n. Every time I have to break out a sniffer[0] to analyze slow traffic, it always turns out to be a C* downloading pR0n.
[0] Not a real antique Network General Sniffer, but a laptop with a pile of code I have built/adapted to my needs over the years.
Brian may not have been the messiah ...
but he certainly wasn't the naughty boy in this case
Sorry, couldn't resist