Techie left 'For support, contact me' sign on a server. Twenty years later, someone did
- Reference: 1732260726
- News link: https://www.theregister.co.uk/2024/11/22/on_call/
- Source link:
This week, meet a reader who asked to be Regomized as "Ron" and told us that he spent much of the 1980s and 1990s working at a PC manufacturer famed for building machines to order in the Great State of Texas.
Ron worked on the Unix team at the PC-maker and told us his role was a jack of all trades who could fix the network one day, arrange backups the next, and write a driver if asked.
[1]
The users he served included salespeople who, when they managed to secure an order, used their company-provided PCs to log on to an application running on a Tandem mainframe.
[2]
[3]
That app would parse each order and create a unique five-letter code that informed the provisioning system what hardware and software were to be installed in each new machine. Another application that also ran under Unix would then create and install the appropriate OS image for each new PC.
As the box-maker made boxes at significant scale, that Unix system drove over 100 PC assembly stations.
[4]
Ron had made that possible by altering its kernel so it presented 100 virtual network interface cards, each with its own IP address.
The system worked for years – even after Ron moved to another employer.
And then one day, around 15 years after Ron left the building, a former friend and colleague called.
[5]
"Our Singapore production line is down, software isn't being installed, nobody knows what to do."
Ron remembered the hostname of a server in Singapore, shared it with his mate, and learned that his info helped support staff to identify the machine and fix its burned out power supply.
Ron told On Call he was a little amused that nobody on site knew what was going on. He was also not impressed to receive no thanks for this assistance. But he thought little of it – until five years later when he received a LinkedIn message from a stranger.
[6]That hardware will be more reliable if you stop stabbing it all day
[7]Tech support world record? 8.5 seconds from seeing to fixing
[8]Hide the keyboard – it's the only way to keep this software running
[9]Yes, your network is down – you annoyed us so much we crashed it
"They told me a server had been found in a closet in the PC-maker's home city with a huge note on it: 'DO NOT EVER TURN THIS SERVER OFF – CALL RON x12345'."
Nobody knew what the server was, or did, and this time Ron's memory failed him – he could not recall what the box did, or its password.
"I laughed my head off, told them I had no idea, suggested they learn how to boot a Unix box into single-user mode and wished them good luck."
Ron never heard from his former team again. But he notes the PC-maker is still in business (see – this story isn't about Compaq) so either the box was fixed, wasn't needed any longer, or was somehow replaced.
Have you been asked to fix tech you installed in the dim, dark, past? If so [10]click here to send On Call an email so that we can jolt readers' memories with the tale of your feats. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z0BkUxeb0I4Tip_FruBx3wAAAA4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z0BkUxeb0I4Tip_FruBx3wAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z0BkUxeb0I4Tip_FruBx3wAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z0BkUxeb0I4Tip_FruBx3wAAAA4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z0BkUxeb0I4Tip_FruBx3wAAAA4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/11/15/on_call/
[7] https://www.theregister.com/2024/11/08/on_call/
[8] https://www.theregister.com/2024/11/01/on_call/
[9] https://www.theregister.com/2024/10/18/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Passwords
I had something similar in a school.
I worked as a kind of roving technician/consultant for schools and ended up with dozens of them, working one each day or half-day every week.
Eventually, after many years, one loved me so much that they outbid all the other schools to have me full time (it is no exaggeration to say that they had to argue with a London Borough HR/Finance department in order to create a brand new payscale for the desired salary because no such payscale at that level existed at the time for support staff, and there were major ructions over it - until I witnessed the head basically say loudly and clearly down a phone line something along the lines of "I'm the head, you're just the Borough team in place to support me. I'm choosing to hire him, I don't see what that has to do with you and your systems being unable to cope with that, just resolve this"... I was put under NDA and apparently in that Borough was the only support staff on a special, unadvertised payscale that nobody else had... Anyway...).
I worked for them for a while but then the school changed and they employed lots of very daft teaching staff who didn't understand IT (and one who was given the title of IT Coordinator - which is basically the person on teaching staff who interfaces with the actual technical IT manager, etc. - who then tried to tell me my job while knowing absolutely nothing) and things began to get silly.
So I walked. It was a shame, but there you go. One of the governors was an IT guy in the city and we agreed it would be best if I handed over to him - at least in terms of knowledge - and then they could pass on anything to the next guy they found. So that's what I did, I made a CD-R with all the documentation of the network, and handed copies over officially to the head and to this governor.
A few months later, I was in another job. I got a phone call from that school. It was the IT Coordinator guy. He *demanded* the main network administration password. Never gonna happen. It lets you see salaries and HR notes and the like and there was no need for it (this was back when things were divided into "curriculum" and "admin" networks and he basically wanted the master passwords to everything). He was adamant and kept pushing and got quite angry on the phone.
I asked why he needed it.
Flashback 6 months prior when he'd just started. He demanded - without prior consultation, authorisation or notification - that the school must buy a bunch of portable MP3 recorders. Absolutely vital. Critical. More important than everything else on the entire network that was running the school. I refused to authorise it (not really my place to authorise academic IT anyway, but it wasn't coming out of my budget for sure, so he'd have to convince the head, not me). He got really quite shirty about it.
A month later, he comes to me grinning with a cardboard box full of brand-new "MP3" recorders. Smug as hell. I mean, I was expecting that, mate. It's not smug to have convinced the school to do exactly what I told you you would need to do. He demanded I make them work on the network computers. Yeah, I don't think you understand how they work because they have nothing to do with the computer, really, they just record the audio and then act like a storage device. But, yeah, sure. I take the box.
He then asked me for the administrator password. We don't give that out to teaching staff, no. He got shirty again. I asked why he needed it. Apparently he had bought, without authorisation again, a piece of software for the kids to bring in the audio and edit and save it. Okay. I mean, I don't know why you think you need the administrator password (I'll install it, thanks, it's my job) or why you had to go behind people's backs to buy that stuff (and I wouldn't have bought that software and was explaining why when he just stopped me mid-track and told me to "just do it" and walked off).
So I installed the software across the network (cue this massive all-staff email from him boasting about his new project and how everyone should start using it, etc.), and I tested one of the recorders. It did exactly what it was designed to do. So I authorised them and gave them back to him.
The first lesson with them, he came storming in. Turns out the "MP3" recorders only recorded in WMA copy-protected format. I mean, it literally said that on the box and the website description. And the software only accepted MP3 files. I mean, it's the kind of thing an IT manager would check before allowing you to purchase them and authorising such a purchase, but I was cut out of the loop on that one. He demanded that I fix it. Nope. He insisted that I give him the administrator password. Nope. Somehow he believed that would magically fix these things that ONLY recorded WMA and the software that ONLY opened MP3. Ensue a massive several-week-long argument between me and him about it.
In the end, I went as far as I was prepared to go (being the only IT guy on very good hourly rates, I was both extremely busy with actually important stuff, and not willing to waste my time on trivia like that, and the head was generally in agreement). I made a network folder. Any WMA saved in there would trigger some software I wrote, which would convert the audio to MP3 and a minute later it would magically appear in another folder on the network as an MP3. Literally Save it here from the recorder, then by the time you open the software, you can open it from over there in the right format.
Not. Good. Enough. I was told quite clearly, several times, loudly, by this guy.
I was months into having had enough of him (and he was one reason I left), so my support for this project ended there and I wasn't interested in the fuss he continued to make.
So now fast-forward to when I've left that school, am working elsewhere, and I'm being disturbed at my new employer by a long ranty phone call demanding the network administrator password. To get his MP3/WMA software magically talking to each other without having to save it in between (he just wanted the MP3 software to "see" the recorder and pick up the WMA's that it couldn't open, automatically). I told him no. He got extremely irate. I explained (again) that it wouldn't do anything, but also that I was absolutely not going to give him any password - I didn't work there any longer, it wasn't my responsibility and no way was I going to compromise the system.
He then insisted that the head at that school had ordered that I give him that password and that nobody knew it so the school must have it, and he'd been chosen to be the custodian of it. Highly doubtful, I knew that head and he's the one who'd championed my salary - he'd just call me if that was the case. I called him out on the lie. I said he'd have to ask the head for authorisation and I'd have to hear it from him, because I don't believe he'd authorised it. He screamed at me down the phone for some time and demanded how I "would ever possibly know" what the head had ordered or not.
"Well... for one, he already has that password. It was in my handover that he signed off on. For two, the governor also has a copy of that password, just to ensure that someone had it somewhere else. And three... the reason that you don't know this is that it was explicitly discussed on handover and I was told to never give *you* any access or password ever."
The governor, being an IT guy, had understood the "problem" with the MP3/WMA thing immediately and I'd warned him on handover. He was the one who said "Under no circumstances give him that password", and the head was present and agreed. We handed over knowing that it was never going to work, and that he was never going to be party to the privileged details because he just didn't know what he was talking about.
So the big long phone call which he'd made without the school's knowledge, to demand the network administrator's password, to get his embarrassing pet project that had never truly worked the way he intended to work properly, and claiming that the head had authorised it? I can only imagine that got him into big trouble. But he didn't get the password from me.
I checked the school website and their staff list about a year or so later - he was gone.
Re: Passwords
I feel that you gave him way too much phone time. He says the head demands that you give the password ? Nope. He has the password. Go ask him. Have a nice day.
And then I'd hang up. Forever.
Re: Passwords
how I "would ever possibly know" what the head had ordered or not.
"Very easily. I'll ring him to get confirmation."
Cables
About 2 years after leaving a company I got a call from one of the electricians,
Do you remember when you put in the gps time server?
Why
Someone was doing some work ripping out old cables and it no longer works
It is somewhere under the floor outside the comms room and goes through a window frame to get outside
Which one?
Don’t ask me I left 2 years ago I can’t remember the building layout
Almost as bad as discovering I was still the ripe admin (with a valid account) for the companies public ip range - knowing them I probably still am 5 years later.
Re: Cables
> Almost as bad as discovering I was still the ripe admin (with a valid account) for the companies public ip range - knowing them I probably still am 5 years later.
In all seriousness:
If you think this is likely (and from what I've seen of the world, it probably is), do yourself a favour: log on to your RIPE account and enable 2FA for it. With RPKI ROV becoming more and more prevalent in the world (which is a good thing!), it's now possible to cause serious damage from a RIPE account (which is a bad thing, well, depending on which terms you left your ex-employer on). If someone were to do so with an old account of yours, you don't want to get fingered for it.
Re: Cables
Can’t do that, the account was keyed to my old company email address which I don’t have access to so I couldn’t reset the password and cannot remember it after all this time.
I did inform them when I found out so hopefully they should have done something about it by now…
Reminds me a bit about the time a lab I used to work for replaced the image processing software developed by me and a colleague by shiny new stuff from a major vendor. They weren't getting brilliant results, mainly because the eye-wateringly expensive new kit (i) didn't have fluorescence calibration tools at all, and (ii) used a global, automatic threshold for segmentation of bacteria, rather than a locally adaptive automatic thresholding method I had developed. The practical upshot was that bacteria on the edge of the field of view looked smaller and seemed fainter than in the centre. If you moved the bacteria from the edge to the centre, they magically became brighter and larger. Not good.
They asked me whether I could quickly (like in an hour or so) implement my calibration and thresholding methods on the new kit. I answered it would take far more time (which they didn't want to pay for), and gave them my scientific papers on both topics, my C code from the old system, plus its documentation (I had left copies in the lab when I left, but they had "lost" it), and wished them luck. As they had not seen fit to hire any new software developers after I had left, they never got my methods to run on their new kit.
Reminds me of a place I worked at where I proposed a new way of controlling regenerative braking of a motor system, only to be told "We've been doing this our way for years, and we know yours won't work".
I eventually fell out with them over something more serious so, I gave my notice - which included a warning that I would take them to a tribunal for constructive dismissal.
Strangely, they still made me work my month's notice, which I used to implement the braking algorithm I had proposed. I was told they finally adopted it a number of years after I left "so that it could be someone else's idea"...
"which I used to implement the braking algorithm I had proposed."
I definitely wouldn't have done that for two reasons one of which being that they didn't desrve it.
Dumb Management
It's the same where i work you are replaceable and the company will survive without you. I was moved internally to save money and they still wanted me to support my old team when I moved just a week later unfortunately for them amnesia set in.
Over 10 years after i installed an HP MSA1000 storage array as part of a Sharepoint/SQL installation for a major university I got a call asking what the password was for the MSA Admin interface.
I suggested that they read the documentation I'd left with the installation as I'd left precise detail in there
The password was still set as default with a big note in the docs saying "please change all default passwords to secure ones once the consultant has finished the project"
Basically in 10 years they'd not upgraded firmware, changed passwords as instructed etc.
Well...
Better having it on a server than having your name and phone number written on the wall of a bathroom stall...
Documentation pays (sort of)
I left a major defence company after 20 years where I had been looking after a major product line. The code came to about 6,000,000 lines of which the majority was written by a 3rd party, but multiple restricted extensions could be added for certain builds. Yes, the build was fairly complex to avoid the tainting of the build server with ITAR or other restricted code. But, it was all built from a virtual machine image holding the entire development environment that was available on the normal development network. Scripts named BuildProject_A.bat, BuildProject_B.bat, etc. were stored on the secure server that initiated the special builds, and that was where the references between batch files and customers were also held.
Consequently the build could be performed from the virtual machine for normal projects, but could be initiated for many restricted projects from the secure server. This was simple in operation, but had to be meticulously documented, in two versions - on the normal development server and on the secure server. Before I left, I made sure that I watched 3 people follow both sets of instructions to build the images.
3 years after I left, an obscure bug was found in the OS that supported the product line. All three people I had 'initiated' into the build process were still with the company and rebuilt the standard project image without problems and it installed safely across the world.
The Software Manager (one of the three, and who had worked on the product line for 6 months, before greasing the right pole to get promoted) emailed me in a blind panic as all the 'special' customers were wanting a fix and they could find no indication of how it could be done - as I said, the separation was complete to avoid tainting or customer information getting out. I reminded him of the other set of instructions on the secure server and heard nothing more.
Probably saved them millions in lost reputation, but I didn't even get a thank you email back. I'm sure I would have received at least a meal out had he still been an engineer, but as he had turned to the dark side of manglement...
Anon, because this product line is in applications with multiple decade long term build and support agreements - with the immortal line "Off planet, on-site visits, are not supported".
Re: Documentation pays (sort of)
Not IT, but certainly an alignment with the heading. Back in the early 1980's I was working for a company that was making kit for the N Sea and one package whilst part of the company's core lines, was a one-off due to the very specific requirements of the contract; one such peculiarity was the paint type and colour that meant no component was able to be drawn from stock. Suffice it to say, they also wanted a LOT of documentation - 15 sets of some very large folders. My final responsibilities on the project were to compile the master copy and then make 15 photocopies of it (one of which remained on my shelves. Their 15 were packaged up, dispatched and signed-for by the project management company acting for the end-user.
It was 7 or 8 years later that I was actually working for the end-user that I received a call from one of their documentation controllers to ask if I could contact the supplier (my previous employer, though not a fact they yet knew) to request a copy of the documentation. My immediate response was a clear "No!" they were taken aback as I that wasn't my usual response to such requests. I then explained that I spent over a week of time I can never get back, standing at a photocopier making the 15 sets of documentation that were sent to, and signed-for by, the project managers. "Ask them," I suggested. It turned out that everything had been handed over and was somewhere in the document control system. I later learned that there was so much duplication that most copies had been shredded - I could have cried. The final irony to that project was that a few years later I visited the offshore installation; nothing to do with that equipment but I took the opportunity to see it in situ (nostalgia as it had been the first such project I'd seen through from start to finish) - it was all good and running (although the special paint finish that had cost so much time and money to achieve was history - as soon as it had been installed it was repainted to match another colour scheme).
8 years for a piece of software I'd written, 10 years for a password to a Linux proxy server (different times, different companies). I provided a fix for the software problem over the phone, but I could not remember the proxy password. I did talk the guy through recovering it though. Both calls were from people I enjoyed working with, so I was OK with helping them out
lvmirror script, HPUX
Many moons ago, I was the backup admin of a big DC at an international company.
I had written some nice scripts to perform online backup for locally attached storage, using the HPUX mirroring commands.
Of course, I had left my name of them.
10 years after, a lady from the other side of the world called me and wanted support for those. After a quick chat, she was just wanting me to do her work for a new system, since she didn't have a clue how storage worked on HPUX. Now idea how she got those scripts BTW.
LOL
The printer has stopped working
On the very first project I worked on as a full-time employee I ended up doing almost all of the printing functionality. (And the third too, but that is irrelevant.) Years later, while I was still within the same company, but the original project had gone into long-term support provided by a small core team, I got a call asking if I could go back and look into why all printing was failing in the test environment before they made the same change to production. Hurrah for good change control processes. Since that office was only across the road I went over at lunch time and started poking around.
"How long will it take to fix?"
"Probably about 10 minutes after I find out what the problem really is."
"How long will that take?"
"Good question, but it will be shorter if I can work uninterrupted."
The original solution used a custom printer driver on the unix servers, requiring that the standard LPR daemon was replaced. They had just upgraded the servers to the next major release of the OS, which re-enabled its own lpd, so the custom driver was failing with an address/port already in use error. It took surprisingly little time to find the problem as the error was clear in the the system log, but they had not looked. Printing was seen as a black art, so it was better to get somebody who knew about such things in if at all possible. It was of course pure coincidence that while on that original project my print jobs always jumped to the head of the print queue. (Not to the level of interrupting the current job, but that was possible.)
I went to do a telephony job recently and was impressed by the quality and attention to detail of the installation. It just struck me as a job done well. It took me about an hour before I realised that it was me who had done that install, over 15 years ago, for the previous tenant.
Unlike me - One time I was trying to debug some obscure code and thought "What idiot wrote this?".
Then I found my name in the header...
Once, nearly 20 years ago, I company sold my soul to a customer who wanted me to write some derived satellite telemetry parameters (these are data values calculated on the fly from the telemetry data downloaded from the satellite and provide additional supplementary information). Some of these parameters were pretty complicated and required an model of a part of the satellite's internal wiring between components which I developed from the satellite's design schematics. The derived parameters were written in C, and after 3 months everything was written, tested and accepted.
About three years later I was called by my contact in the company in question - can I come in for a month and make some changes to the derived parameter code, they had discovered (shortly after the satellite was launched) that someone had not followed the design schematics, and had wired things up slightly differently (the satellite still worked, just not in the expected way). I had to pass this up the chain to management since, at that time, I was fully allocated to another high-priority project for a different customer who would not be impressed if I disappeared for a month.
Management thought about things and was on the verge of refusing the out-of-the-blue request for my time when my contact phoned back - they had actually looked at the code I had written and had realised that it included extensive inline documentation including full instructions on how to adjust the internal wiring model (basically a set of data tables that drove generic algorithms); they had followed my instructions, applied the changes and now everything was working fine. I was profusely thanked for doing such a good job in the first case and promised a beer when I next saw him.
I got that beer about 3 months later!
Passwords
A former employer contacted me some time after I'd left asking if I knew the administrator passwords for some NT boxes used for SolidWorks 3D CAD.
As they'd made me redundant I wasn't that inclined to help anyway but I pointed out that the computers had been sourced and set up* with no consultation with, or input from, my IT department so why did they think I would know passwords for them?
*My only involvement was rescuing the supplier's van after they'd thought it a good wheeze to drive over soaking wet grass so they could "post" the kit through a window rather than carry it 20 yards or so through the entrance. My old and battered Range Rover was well up to the task of pulling them out but I don't think our building management were very happy about the ruts left by the van!