News: 1734679993

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Techie fluked a fix and found himself the abusive boss's best friend

(2024/12/20)


On Call Digital technology remains frighteningly finickity, which is why good tech support people are always in demand – and also the reason The Register never tires of telling your support stories each Friday in On Call, the column your generosity makes possible.

This week, meet a reader we'll Regomize as "Boris" who years ago worked for a business providing services to what he described as "a large international automotive company" that ran its production planning application on an old school mainframe – proper supervillain lair kit, with big tape drives whirring away all day.

The IT director at this client had a temper.

[1]

"He was known and feared as someone who ate systems support people for breakfast."

[2]

[3]

Boris was therefore far from thrilled when he was called in to address a problem his colleagues had been unable to address.

"The planning application would sometimes suddenly hang at random points without any obvious reason," Boris told On Call. "This was very upsetting as delays in the availability of manufacturing schedules interfered with plant operations, which cost serious money."

[4]

Hardware experts were put on trans-Atlantic flights so they could pore over the mainframe's innards. Software engineers who had hand-coded the machine's OS were sent to find faults.

None could determine the cause of the hangs. Indeed, all reported the machine was working as intended. All systems nominal.

Those investigations consumed months – and did not make the client happier.

[5]

Indeed, the irate IT director began making serious noises about seeking compensation and junking the mainframe.

In desperation, Boris was asked to examine the situation.

Boris wasn't thrilled about that, as his skill set – engineering and scientific matters – was not obviously applicable to the situation. And he knew nothing about scheduling assembly lines.

He nonetheless visited the client's office, and was quickly "shouted at and threatened by the IT director."

Boris managed to retain sufficient composure to ask for the application's source code.

"Fortuitously it was in Fortran – one of the programming languages I was very familiar with," Boris told On Call. It also contained an obvious error that he spotted after about ten minutes.

"The code assumed that all the tapes were at their start point. Whether or not the program would run successfully depended on the state of the tapes left by any previously executed application. Sometimes it would run, and sometimes not."

The fix seemed simple: a Rewind All; statement in the code – one at the start and one at the end – would surely ensure the tape was always at the start point when the application ran.

[6]Backup failed, but the boss didn't slam IT – because his son was to blame

[7]Tech support chap showed boss how to use a browser for a year – he still didn't get it

[8]Tech support world record? 8.5 seconds from seeing to fixing

[9]Hide the keyboard – it's the only way to keep this software running

Boris recompiled the software, ran it, and relaxed as the problem went away.

Which is where his troubles began – because the abusive IT director took a shine to him.

"Forever after I was his 'go to' person for advice on almost everything from hardware selection decisions to application development and I was treated with reverence and the appropriate level of respect by all."

But Boris knew this couldn't last – because his Fortran fix was fortuitous. He therefore lived in fear of being found out and ending up on the wrong side of the abusive IT director's wrath.

"Fortunately, I was moved overseas on a different project before my limitations could be tested," he told On Call.

Phew!

Have you ever found a fix despite not being an expert in the troubled tech you were asked to tend? If so, [10]click here to send On Call an email so we can feature your story after the festive season.

On Call wishes readers all the best for their end-of-year celebrations, and thanks you all for the weekly gift of your stories. ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z2VOYnKFsntpXb-3spy7CAAAAM8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z2VOYnKFsntpXb-3spy7CAAAAM8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z2VOYnKFsntpXb-3spy7CAAAAM8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z2VOYnKFsntpXb-3spy7CAAAAM8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z2VOYnKFsntpXb-3spy7CAAAAM8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2024/12/13/on_call/

[7] https://www.theregister.com/2024/11/29/on_call/

[8] https://www.theregister.com/2024/11/08/on_call/

[9] https://www.theregister.com/2024/11/01/on_call/

[10] mailto:oncall@theregister.com

[11] https://whitepapers.theregister.com/



And This Is Why We Have Code Reviews

simonlb

To make sure things like this are picked up.

Meanwhile, Boris fixed the issue but then acquired a slightly poisoned chalice by way of thanks.

Re: And This Is Why We Have Code Reviews

Neil Barnes

The pellet with the poison is in the vessel with the pestle, the chalice from the palace has the brew that is true, right?

Right, but there's been a change. They ... broke the chalice from the palace.

They ... broke the chalice from the palace?

... and replaced it with a flagon.

A flagon?

... with a figure of a dragon.

A flagon with a dragon.

RIGHT.

But, did you put the pellet with the poison in the vessel with the pestle?

Noooo, the pellet with the poison is in the flagon with the dragon, the vessel with the pestle has the brew that is true.

The pellet with the poison's in the flagon with the dragon, the vessel with the pestle has the brew that is true.

Re: And This Is Why We Have Code Reviews

KittenHuffer

I was thinking to myself that I'd been in plenty of code reviews that sounded a lot like that ....... then I realised your comment was to do with the 'poisoned chalice' mentioned in the OP!

---------> It's a Friday. That's my excuse and I'm sticking to it!

Re: And This Is Why We Have Code Reviews

Bebu sa Ware

For those too young to be able to remember Danny Kaye [1]"The pellet with the poison is..."

[1] https://youtube.com/watch?v=WzmnSyqv37A

Bluffing

Vincent Ballard

> Have you ever found a fix despite not being an expert in the troubled tech you were asked to tend?

I've maintained code in languages I don't know, and in one case I didn't even know what the language was, but generally it's possible to reverse engineer the syntax by looking at the rest of the program. It only becomes a major problem when you need to know the type system.

Re: Bluffing

Pascal Monett

Yup, happened to me too.

Client was ranting about a script that wasn't doing its job anymore. Instead of asking the coder (who had apparently left), I was the one it fell on to try and find a solution.

Thankfully, my 30 years of coding experience in another language (and ample what-the-frak-is-this experience) allowed me to pinpoint the solution, and leave the client happy.

I have never done coding so by the seat of my pants as that.

Re: Bluffing

F. Frederick Skitty

I once made a fix to a legacy system written in FORTRAN 77 (with the inevitable vendor specific extensions), as with our sole FORTRAN guru was off work with long term illness and I was the only other person who had ever used that language. My total experience was writing a few library functions to emulate string functions in the C standard library and then using them to write a command line argument parser as a front end to batch processes. Thankfully the fix was obvious and our guru eventually returned with his health restored.

Re: Bluffing

Admiral Grace Hopper

Same here.

"So what language was it in the end?"

"No idea. It had curly brackets though".

Re: Bluffing

Bebu sa Ware

«major problem when you need to know the type system.»

Not something Snobol IV would present. ;)

Close enough

KittenHuffer

I was once asked to look at a model (I'm not a modeller) that ran on modelling software (that I had never seen) to try to help with the issue where the run times were going up exponentially as the size of the model was increased.

I looked at the model, and scratched my head. Then dug into the modelling software, more heady scratchy.

I ended up doing a setting by setting read up of each of the config settings. That's when I discovered that the initial memory was something like 50KB, and when more was needed it was increased by something like 10KB. On the off-chance that the software was spending all its time increasing the size of the memory pool I jacked those numbers up to values more in line with the resources available ....... and suddenly hours of model time shrank to mere minutes.

This, of course, led to much rejoicing in the halls, tickertape parades on Main Street, and the dancing of Ewoks!

Re: Close enough

ColinPa

Ive been in a similar situation. This problem had been going on for months. I was asked to go on site and be a fresh pair of eyes.

I got the joblogs, and one of the first messages was "storage pool allocations are too small". Apparently I was the first person to have noticed this.

We made the storage pool bigger, and tried it on the tests system, which suddenly went much faster. Although the storage pools were not my area of expertise, I read the doc, and looked at the stats. The stats gave recommended storage pool.

We made the values for production to match what we tested, and restarted production.

Next day the operations people rang up the system progs to say "The application was using about 10% of the CPU it used to - was there a problem?"

I stayed on site for a couple of days, and spent my time poking around other applications. Several applications had the same problem.

The outcome was they deferred a machine upgrade, because now they had spare CPU!

The Power of Fear!

Lil Endian

The early 90s, in a small national hardware maintenance company. I'm writing the in-house engineer scheduling, fault logging, customer BOM (etc) system (basically a bespoke ERP system).

A field engineer RTBs a server, Olivetti summink-or-other. It's at base for a couple of days, and he can't get it sorted. The head engineer has a stab, and gets just about as far. So the MD throws his penny's worth into the coin jar - still no go.

"Lil, can you have a look at this?", I'm asked one morning. I arrive in the engineering shop, "What do I expect if I power up?", I ask. "Nothing, it's totally dead now." I'm told - which had snowballed from the initial, far more trivial issue initialising call out. Things had gone from bad to worse. So I tried a power up. The box POSTs, NetWare etc all good. Stunned faces all around! Power cycle, run-in testing over a few days, it never fell over again. We never had a clue WTF that was all about, but hey, if putting the fear of Christ into hardware is enough to get a fix, I'm good! (I wish obscure bugs were as easy!)

[Icon: there's a scientific explanation for everything - I've just got no idea what that is!]

Re: The Power of Fear!

Caver_Dave

Very similar to the "go onsite to the 'always failing' system, and ask for a demonstration".

The demonstrator follows the instructions as written and it does not fail.

I think we all know where the problem lies here?

Yes, it should not allow the failure if they are not following the instructions, but hey-ho!

Re: The Power of Fear!

Lil Endian

In fairness to the engineers I mentioned, I think they did actually try the power button! :DDD

[To explain the "fear" part, I've always considered myself "fair but firm" - it wasn't beyond me to retire misbehaving kit. I even once dented an IBM Model M, but I had to take a sabbatical after that!]

Re: The Power of Fear!

robinsonb5

> I even once dented an IBM Model M, but I had to take a sabbatical after that!

I bet that hand still hurts when it rains.

Re: The Power of Fear!

breakfast

My wife will sometimes ask me to sit by her when something's not working in the hope that my programmer's aura will fix it. It works more than it should.

Re: The Power of Fear!

RockBurner

I'm the opposite. If there's a flaw within a script/program/config or software, me merely walking by will cause it to find that flaw...

Re: The Power of Fear!

Lil Endian

That's an awesome skill, worth your weight in gold as a tester.

As an example, during the development of the "ERP" system I mentioned, my mate was undertaking bug hunting for me. I'd give him a release that I'd think was "fairly stable" - he'd trot off, then return 5 minutes later (literally) with a sheet of A4 with a dozen bugs on it. "You might wanna look at these to start with." -- to start with, the sod angel! Seriously, I don't even know how he could navigate through the various modules in the system that quickly, let alone pulling off various test! It's not possible to fully debug your own code, but man, sometimes it doesn't even seem to be worth trying!

Re: The Power of Fear!

Admiral Grace Hopper

One of the best testers I ever worked with had this skill. We called him, "Entropy's Little Buddy". There was nothing he could not break.

Cobol...

GlenP

About my only exposure to Cobol was when my then boss asked me to take a look at some code she was struggling to debug.

I don't think I'd ever even seen the language before but I was trained and proficient in BASIC and Pascal*, and had a self-taught working knowledge of FORTRAN, so simply going through the code logically was easy enough. It only took a few minutes to find the obvious error, I'm not blaming my boss, it was one of those where fresh eyes were needed.

*Unlike other establishments at the time Newcastle Uni focussed on programming skills in a single language, UBC Pascal, rather than teaching simple skills in multiple languages. It may have reduced immediate employment opportunities but overall it's stood me in good stead for nearly 40 years.

Re: Cobol...

Sam not the Viking

I started on Algol/Basic then moved onto Fortran. These stood me in good stead for a long time, I think it's the discipline in logic that guide and help in fault-finding in any language or system (how could it be otherwise?).

I was very disappointed to learn that my new Engineering Graduates had learned no programming language at all at Uni. It shows.

Re: Cobol...

KittenHuffer

The other technique I figured out when bug hunting during development was to say to yourself - If I wanted the results that I'm getting how would I code it. You then end up with a second set of code in your head that you can mentally compare to the code that you originally designed, and 9 times out of 10 there will be a line (or area) that you need to change to flick the functionality over to what you're getting rather than what you want.

So instead of asking how do I stop that behaviour, asking how would I get that behaviour has been the quickest way for me to fix bugs.

I haven't done it consciously for a long time, cos I think it became such a habit that my brain now does it in the background, which means that when a bug pops up most of the time I'll immediately just scroll to a certain place expecting to see a certain bit of code that is certainly wrong! That's for certain!

Re: Cobol...

Anonymous Coward

"I was very disappointed to learn that my new Engineering Graduates had learned no programming language at all at Uni. It shows."

I must ask our new mech eng. graduate whether he learned a language...

I learned C at uni on my mech eng course, but I had to use Fortran immediately after I started work 26 years ago, and since then I've had to use awk (yes, really), Matlab, visual basic, tcl scripting (yuk!) and Python. I have only ever had to use C to alter a small piece of code from an obscure in-house FE pre-processor, to get it to compile on a GNU compiler when it was originally written and compiled on a Sun workstation. For the record, I work in mechanical engineering, I'm still occasionally using Fortran for real engineering purposes and my role is *not* IT related....

Fresh eyes

Pascal Monett

I had a case where a script I had written on client site just refused to work.

I know how to write code. I've been writing code for practically all my working life. This code should have worked. But it didn't.

Try as I might, I banged my head against a brick wall for more than an hour. The I called for help. The project manager (who knew code) came to take a look. He didn't find any fault. That was fine and dandy, but it didn't solve the problem.

He called a colleague. After a full half an hour of code review and discussion, someone (not me, but can't remember who) mentioned : "hey, shouldn't those be points in the mail address ?".

Problem solved, with three sets of eyeballs after an hour of debugging.

Sometimes there are things that you just can't see for yourself.

PEOT? What's that?

42656e4d203239

Had an analyst who loved reading data from 12" tapes and failing to check for PEOT, just kept on reading till the tape was all wound off the spool into the innards of the MictroVAX II (Yeh - I forget the drive model but it was the one which mechanically slurped the tape off the spool into the depths of the drive for the takeup spool)

That's when I started to learn FORTRAN and VMS system calls to fix her code for her.... mostly cos I was fed up with respooling the tapes!

We got on a bit better after that, but she was a really odd one... and this is me calling someone odd!

Re: PEOT? What's that?

David Harper 1

I did that too, when I was a Ph.D. student using a VAX 11-750 as a visitor at a famous but now sadly defunct government research institute. I got a royal bollocking from the system administrator who had to fish the end of the tape out of the reservoir of the tape drive and painstakingly thread it back onto the reel. I only made that mistake once. Happy memories.

"Fortunately, I was moved overseas on a different project before my limitations could be tested"

Howard Sway

How much luck did this guy have? This was a much bigger stroke of fortune than spotting the initial problem - otherwise he would have been stuck in the position as the go-to guy who could solve all the problems, having to do all the work whilst watching sundry mediocre managers take all the glory and all the bonuses for everything working efficiently. It might seem appealing to be seen as the "resident genius" , but it can be exploited by the incompetent and cynical, and therefore being the go-to guy should mostly be considered harmful.

Small changes matter

MiguelC

I talked about this recently,, but it'll sit well in this thread, so here it goes again.

I once stopped by a PFY trying to fix a user's Access application that had stopped generating email reports. He was at wit's end, having tried for hours everything he thought might possibly solve the problem. I nonchalantly asked the user if IT had done any upgrade that day. "Hm yes, they installed new printer drivers".

Ah.

After changing the default printer to PDF, the emails started once again being generated, because of course Access needs the printer driver to generate reports, even if they're not being actually printed.

Re: Small changes matter

yetanotheraoc

Perhaps should be "all changes matter".

Apple II

Andy Non

At one place I worked, the accounts dept ran some software on an Apple II computer for reconciling invoices received against orders. Apparently the software was getting slower and slower to run as the amount of data increased over a year or two. It got to the stage that the monthly run was taking 2 or 3 days to process!

I was asked to take a look at the code. Essentially there were two long lists of data and it had been coded to read each entry in the first list, then search the entire second list, record by record from disk, looking for a match. As both lists had far more than 1,000 records it implied over one million records were being read and compared in all. I rewrote the code from scratch and did some pre-processing first, sorting both lists into alpha-numeric order using a Shell-Metzner sort algorithm I'd seen in a Commodore Pet programming manual. Then all that was required was to step through both lists at the same time comparing records in a sort of crab-wise motion. Only one pass of each list was required. This got processing down from several days to around 20 minutes! After that I was given lots more programming tasks.

Re: Apple II

yetanotheraoc

Problem sorted, then.

Re: Apple II

Lil Endian

That is truly terrible! Makes me want to throw up in a bucket sort of receptacle.

[You get my vote for "Most Droll Pun of the Week"!]

If you make any money, the government shoves you in the creek once a year
with it in your pockets, and all that don't get wet you can keep.
-- The Best of Will Rogers