Tech support fill-in given no budget, no help, no training, and no empathy for his plight
- Reference: 1737102729
- News link: https://www.theregister.co.uk/2025/01/17/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Ronnie" who admitted he has never been an IT professional, but used to fill that role when he worked at a non-profit which was affiliated with a university and shared offices with its academic collaborators.
Despite a lack of formal tech training (his background was in law and conflict resolution), Ronnie enjoyed his IT responsibilities because he wasn't bound by the university's rules and standards, but was able to ask its IT staff to fix stuff he couldn't fathom.
[1]
"Everything was humming along fine until the academics moved out of our building to another campus and our network immediately slowed down to a crawl," he told On Call.
[2]
[3]
Crawl is too kind a word: Ronnie told us keystrokes took eight minutes to appear on screen.
"I called university IT, and they came out and informed me that when the academics moved, they took their server with them." Ronnie's team was therefore now connected to a server shared by the population of the whole building. IT wouldn't help to speed things up. Finding a fix was Ronnie's responsibility.
[4]
But Ronnie wasn't entirely sure if he could accurately describe a server, never mind explain why it might be slowing things down.
He did a little digging and learned that his little non-profit was not the only tenant suffering. Apparently a professor was running a workload that overwhelmed the server.
Ronnie asked what could be done and was told he could buy a server. Between his lack of knowledge and the non-profit's lack of budget, that wasn't an option.
[5]
He suggested the professor could buy a server for their own use, which seemed fair as the boffin clearly needed one.
IT bluntly said that wasn't going to happen.
[6]Devs sent into security panic by 'feature that was helpful … until it wasn't'
[7]Tech support warrior left cosplay battle and Trekked to the office
[8]After a long lunch, user thought a cursor meant their computer was cactus
[9]Techie fluked a fix and found himself the abusive boss's best friend
Ronnie despaired. "Our office was completely stuck in inactivity; it was up to me to save my job and those of our other staff."
So he went online and learned about network attached storage (NAS) devices, which he came to understand were simple to operate and could house and serve the files his org needed - freeing it from the slow University server.
Would IT let him have one? Sadly not. They knew nothing about the machines and – for once – stuck to their rules by prohibiting such a device being connected to the university network.
"I bought one anyway for about $300," Ronnie admitted to On Call, and figured out how to install it on the university network. In the dead of night, when the professor wasn't thrashing the server, he snuck in, connected the NAS, and moved all his non-profit's files to the storage appliance.
Ronnie also configured the box to back up to the university server – he'd learned to do that, too. Nobody at the uni knew. But Ronnie and his crew did – because they were back in business!
"Sometimes it is easier to ask for forgiveness than to get permission," he concluded his mail to On Call.
We're sure you have stories of a tech support triumph, so why not [10]click here to send On Call an email so we can share your experience? We never use your real name and take care not to identify your workplace or otherwise endanger your career: This is a safe space to swap fun yarns with your peers! ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z4o4XO8-7pcEO11KTVUr3QAAAJY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z4o4XO8-7pcEO11KTVUr3QAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z4o4XO8-7pcEO11KTVUr3QAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z4o4XO8-7pcEO11KTVUr3QAAAJY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z4o4XO8-7pcEO11KTVUr3QAAAJY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/01/10/on_call/
[7] https://www.theregister.com/2025/01/03/on_call/
[8] https://www.theregister.com/2024/12/27/on_call/
[9] https://www.theregister.com/2024/12/20/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
What the story does not tell: Did he ever tell anywhere else that thing existed, and if, really got forgiven? Naaaaaaaaa...
"Sometimes it is easier to ask for forgiveness than to get permission,"
The technical term is fait accompli which translates as "the only way to get things done".
You're not Ron...
When I was a nipper, I read everything the local library could offer. This included stuff like the Jennings schoolboy series, Biggles of course and Bunter. Didn't understand half of the latter - what, they don't go home at the end of the day? - but I do remember Billy getting upset after he (thought) he overheard one of his school mates refer to him as a fat accomplice. I chuckle everytime I read the proper phrase.
27 years later
they found a NAS walled in still humming along. Fan was broken long ago, but it still worked.
I love how he did not give up to get something in a working state.
Universities are the absolute worst for shadow IT
Shame the IT dept. were basically not supporting the users properly but I'll bet that thing chugged along till it didn't, which was probably after this chap left and then was dumped on IT with no information whatsoever but had to be fixed.
Seen that before (and much much worse), glad I no longer work for a Uni..
Re: Universities are the absolute worst for shadow IT
Agreed.
From what I've read (here and elsewhere), universities are the absolute most dismal IT environments that exist. No budget, so-called "administrators" that have nothing but demands and no knowledge of the requirements, and no flexibility when it comes to finding solutions.
It's obviously a lot easier to shield oneself with a half-baked procedure document than actually think about how to make things work.
Re: Universities are the absolute worst for shadow IT
A customer of mine used to specialise in network audits, which were usually carried out prior to upgrades or introduction of new tech,
He got a call from a large UK university who wanted to install a new networked IP camera system across the main campus building.
When my customer carried out his survey he found out the university already had a full IP camera system, running to hundreds of cameras and multiple recording servers, that the site management had installed a couple of years earlier, failing to tell the network department.
Because the network department hadn't been informed this system was running on their network and their switches, it wasn't even on a VLAN, and many of the students had already discovered the recording server and were merrily deleting any footage of after hours shenanigans.
Anyway, my customer utilised the existing kit and reconfigured it all so it was on a secure VLAN, this meant I didn't get a sale, but we had a new case study to show to other Universities and large colleges to ensure they installed kit properly in future.
Re: Universities are the absolute worst for shadow IT
A classic example showing how important audits are! I have similar cases on and on... Not only regarding networked equipment, also regarding the "we have a perfect windows update statistic" where a relatively small powershell script, only scanning "Windows Server*", exposes the reality.
Re: Universities are the absolute worst for shadow IT
Not at a University, but a few years ago (around 2007) I got a call from a user about a PC stuck in a boot loop. After a couple of minutes conversation with the user I'd established there were no identification labels on the machine - asset tags, PC id etc. - they then mentioned the make and model which was by then a defunct manufacturer which we'd never used. Further dialogue revealed that the user had been told to use this PC for their work when they'd started a year previously, and after I'd asked them the check the back of the machine it wasn't even plugged into the network. After asking them what was displayed on the screen when booting they said, "Windows 98." So this was just a random machine someone had brought in years before and started using and it had become a fixture in the office multiple people had used over the years.
I had to end the call by telling them to speak with their boss as the machine wasn't covered under any support for either hardware, software or user.
Re: Universities are the absolute worst for shadow IT
In the very early days of laptops I wanted one but our IT department didn't and wouldn't support them. I was PM for a project that used a PC for a control and management terminal. The customer asked us to upgrade the CMS to a laptop, so we had to choose a laptop and then give it a part number on the company system so that it could be procured on the project. I had loads of bunce in the project so I bought myself one. IT didn't get a look in - I raised a purchase req and one appeard a few weeks later. I gleefully set it up - insert disc 3, etc. - and, in my naivety, plugged it in to the the network. Nothing happened. I messed about a bit and nothing kept on happening so I called IT. They turned up, took one look at the laptop and left. I transferred it back into stores for the project and when, eventually, laptops were rolled out generally by IT I was, naturally, one of the last people to get one.
Just a wild shot in the dark but i bet this could also have been fixed by taking the switches off 10T half duplex.
(based on my own college I.T. employment)
Not "Fixing", Exactly
In uni we had a PDP-11/45 running 6th Edition Unix in the CS department. It was classed as "unsupported" by the IT "department" (mainframe people). The machine lived in an office on the second floor of the maths department building, and was in heavy use all day, 8AM-5PM. The maths dept chairman refused to allow us access to the building outside those hours, though we asked very nicely to do so, that we might make system-wide backups.
Workaround: whoever in the cabal (four of us) was in the room just before 5PM lockup would unlatch the window before leaving the room and locking the single door to the room. Later that night, we would return, climb the exterior stairs to the 2nd-floor level, and select a brave soul by lots.
That brave soul would climb up over the handrail, jump the three feet to a ledge ringing the outside of the maths building, croodle along it to the window, open the unlatched window, and crawl inside and make the multiple-media (RK05 cartridges) backups, the room lit only from the -45's front panel lights and a streetlight shining through the window. Backups complete and verified, the brave soul would close and lock the window, and exit the room via the only door, locking it behind him, then exiting the building via the push-bar double doors at the side of the building.
While this was happening, the other three of us had climbed to various overlook points, equipped with air-horns, and watched for security. Security did come by, but only to check that the exterior doors were locked.
Re: Not "Fixing", Exactly
Ethan Hunt style IT support...
Re: Not "Fixing", Exactly
Pity they weren't writing the backups to CD ..... then along with the Dooo da looo music you could have have the image of a burning fuse CD!
Re: Not "Fixing", Exactly
Are you sure you're not just paraphrasing parts of the script of [1]Brazil ?!?
[1] https://www.imdb.com/title/tt0088846/
Re: Not "Fixing", Exactly
https://xkcd.com/705/
Re: Not "Fixing", Exactly
All wrong!
You should have taken the machine offline at 9am and run the backup then.
At least Ronnie backed up the NAS.
I once had a "(very) cheap and cheerful" LaCie NAS device connected to the local network (contravening at least a couple of central IT commandments) by a staff member purely to use as a clearing house for WH&S (OH&S) inventories. I only heard* about it because the owner (literally as she putchases it personally from a local whitegoods retailer‡) was healthily paranoid about backups generally and even of the few shares containing this rather ephemeral data which I duely arranged.
Many moons later I notice the scheduled back up of the device failed, investigating found the device had failed. The owner had long before moved the workflow to a organisation wide application so no sweat. The device was totally dead so I was given the carcass for disposal but being curious I later performed a postmortem. Bugger all in them a 2.5" SATA disk and a SOC. Extracted the disk for later examination.
About a week later a researcher who was notoriously lacking in any computing skills admitted he was storing his files on a share he found on the network which had recently vanished. Could I find it?
After a lengthy interrogation I discovered he had over 1Gb stored on the defunct NAS. It was originally just a backup of his PC but thought to have two copies was a waste.
As consequence of his unauthorised use of the device his share wasn't backed up leaving him up the proverbial creek.
Given that it was likely only the SOC that lost its magic smoke I thought it worthwhile to examine the drive.
The patron saint of fools and idiots must be a busy soul but this researcher must have enjoyed his particular favour, On examining the drive with Linux† PC I discovered that the SOC didn't run some obscure embedded OS or use a proprietary file system but a MIPS Linux kernel with an XFS data volume which of course could be mounted on the Linux PC - some sods are born lucky!
In Ronnie's case one has to wonder why anyone, professor or not, is running computational workloads on a file server.
* of course it popped up on my network monitoring well before this. ‡ Hardly Normal specifically. † Redhat EL3 or a new EL4 at that time.
"Sometimes it is easier to ask for forgiveness than to get permission,"
The technical term is fait accompli which translates as "the only way to get things done".