News: 1670223665

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Programming error created billion-dollar mistake that made the coder ... a hero?

(2022/12/05)


who, me? Ah, dear readers, welcome once again to Who, Me? in which Reg readers confess the times their reach exceeded their grasp, technology-wise-speaking.

This time meet "Francis" who was a bright young thing working on contract as a programmer to a largish organisation that was in the process of upgrading certain systems. Despite his youth, Francis had built something of a reputation for himself as someone who could "pull a rabbit out of a hat" when needed. As such, he found himself quite central to the upgrade project.

One of the core parts of the upgrade – the "crown jewel" as Francis put it – was migrating the accounts to the new system. This involved a script that ran the past eight years worth of nightly transaction files on the new system until they caught up to the present.

[1]

The script had already gone through a number of iterations and time was getting tight towards the end of the year, so further delays were to be avoided.

[2]

[3]

Francis admits he wasn't intimately familiar with the scripting language used, but he had enough knowledge to be dangerous. Examining the script for bugs, he says, "it was obvious that some idiot had put an add where there should have been an add-update ."

[4]How not to test a new system: push a button and wait to see what happens

[5]Job 1: Get the boss on the network. Job 2: Figure out why Job 1 broke the network for everyone else

[6]Just follow the instructions … no wait, not that instruction to lock everyone out of everything

[7]Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working?

Naturally, he fixed that to avoid further issues, and the migration continued. All seemed to be running smoothly, and the script ran on, night after night, week after week, over the Christmas and New Year period.

Until a puzzled accountant wandered in with a most confusing report. According to the printout he was reading, the company had quite suddenly accrued multiple billions of dollars for which there was no accounting.

Francis examined the printout, and the problem became obvious.

[8]

"Some idiot," he writes, "had put an add-update where there obviously should have been an add ."

Of course these were the days before strict version control, so it was quite impossible for anyone to track down who the idiot responsible could have been. Just one of those unknowable mysteries, you know?

Francis corrected the script, then explained the situation (in very broad and non-specific strokes) to the CIO who, far from being angry, seemed relieved to have an excuse to slow down the migration. He felt the whole upgrade project had been moving too fast but hadn't been able to convince the higher-ups to tap the brakes. This "disaster" was exactly what he needed.

[9]

Yet again, Francis had pulled a rabbit out of a hat, and his reputation emerged enhanced. He did, however, come out of it just a tad more humble.

How about you? Ever had a moment of ineptitude accidentally turn into eptitude? Or learned an important lesson about having your confidence exceed your competence? Tell us all about it in an [10]email to Who, Me? ®

Get our [11]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y43PVRRQZehrtZ-XJ3f7YgAAAEQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y43PVRRQZehrtZ-XJ3f7YgAAAEQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y43PVRRQZehrtZ-XJ3f7YgAAAEQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://www.theregister.com/2022/11/28/who_me/

[5] https://www.theregister.com/2022/11/21/who_me/

[6] https://www.theregister.com/2022/11/14/who_me/

[7] https://www.theregister.com/2022/11/07/who_me/

[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y43PVRRQZehrtZ-XJ3f7YgAAAEQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y43PVRRQZehrtZ-XJ3f7YgAAAEQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[10] mailto:whome@theregister.com

[11] https://whitepapers.theregister.com/



Explosive demonstration

Anonymous Coward

At a small on site lab, we had a centrifuge that was fitted with a head rated for half the maximum RPM the base unit could deliver. I pointed out that it should probably be hard locked to prevent settings beyond the head rating and even pulled up a manual showing how it could be done. This was left with the lead maintenance technician before I took a much needed holiday.

When I returned, there was a shiny new centrifuge in its place. Apparently, the technician had verified that the centrifuge could indeed be set higher but had then gone off to track down that pesky manual. In the mean time, a sample had been run and the head unit gave way. Fortunately, the shell contained the chunks but it apparently made quite a bang.

Re: Explosive demonstration

Anonymous South African Coward

In my younger days my employer used a big-ass centrifuge to dry out acetyl salicylic acid in mass.

It had a diameter of 4m, all stainless steel, and was quite heavy.

I would not like to be near that thing should its governor decide to pack up and allow it to overspeed out of control....

Worst code I ever saw...

Joe W

... was the one I wrote half a year ago. More or less. It happens less often as I got better. I think it is mostly ok now, except for the bits of code that I know I should have written differently and not under pressure, and I definitely should revise the code rather sooner than later.

Re: Worst code I ever saw...

Neil Barnes

Cheer up - even at my advanced age I can still take two days tracking down what I thought was a serial communications synchronisation issue, but turned out to be that I'd forgotten a '\r' on the end of a string...

Re: Worst code I ever saw...

DrXym

I used a little USB board that could be hooked up with a PC floppy drive to recover some Amiga formatted disks. These held source code I had written a very long time ago including an unpublished game and a couple of published games that ended up on magazine cover disks.

When I looked at the source I was actually surprised that the code wasn't completely abysmal - it was pretty legible for the most part and I could follow it even though it was invoking Amiga APIs. But I was shocked by how few comments there were in it compared to how many I write now. One of the games had a lot of vector geometry going on in it and it was almost impossible to figure out what was going on without any comments.

Re: Worst code I ever saw...

Martin an gof

almost impossible to figure out what was going on without any comments

One of mine is currently "doing Python" in GCSE Computer Science. I've been impressed with how well it's been picked up, though of course I still get the occasional "hey dad, what have I done wrong here?".

But they haven't been told how (or why) to comment code. Everything written so far is almost completely comment-free. Granted, most of it is self-explanatory at the moment, but a few well-placed comments inserted now would get them into good habits for later methinks.

I have a similar gripe about keyboard skills. Nobody seems to teach "touch typing", despite the fact that most (all?) school leavers will be going in to jobs where typing is a major part of their daily lives. Recipe for RSI, carpal tunnel problems and the like.

M.

Re: Worst code I ever saw...

Dave K

That was the problem I had when learning to code. My dad (a university lecturer) taught me first in Pascal, then in C and always expressed the need for comments. As the code I was working on originally was so simple and self-explanatory, plus each exercise was pretty-much standalone, it seemed pointless to me adding comments that explained code which was simple and obvious, and which I had never needed to refer back to so far.

Of course over time, the code steadily becomes more complex as I learnt new commands/functions and more advanced algorithms and techniques. Then we get onto the topic of reusing code when tackling a challenge that is similar to some code I'd written a month or two earlier. Being faced with a fairly complex algorithm and zero comments suddenly made me understand why comments are so useful.

These days I probably end up over-using them if I'm honest, but it doesn't half help when I rip-off some of my old code from previous years.

Re: Worst code I ever saw...

amanfromMars 1

Then we get onto the topic of reusing code when tackling a challenge that is similar to some code I'd written a month or two earlier. Being faced with a fairly complex algorithm and zero comments suddenly made me understand why comments are so useful. ... Dave K

Would anyone else/many more agree that such is very true here on El Reg which makes the exercising of fairly complex algorithms here invaluable, and indeed much appreciated. It is a service par excellence.

Re: Worst code I ever saw...

heyrick

I learned the hard way. Coming from BASIC on a 32K micro, there were no comments as comments took space and execution time.

I started writing C like that.

Until I came back to revise a project several years later and eventually figured that rewriting chunks rather than try to figure out what was actually going on would be a quicker solution.

Gone are the short 1-2 letter variable names, and hello lots of commenting. It'll help down the line, especially if something is being done in an unusual way and you leave a note to explain why.

Re: Worst code I ever saw...

Norman Nescio

When I was doing some particularly involved kernel mode code on a VAX, I ended up with approximately a 80x24 screenful of comments per line of code. I also had a preface explaining what the code was meant to do, why a particular approach had been taken, and why other 'obvious' approaches had not been chosen, with links to extensive documentation elsewhere. When you are doing counter-intuitive stuff, you have to work hard to prevent other people making 'obvious' simplifications to improve what 'some idiot' (including previous versions of myself) has written.

Re: Worst code I ever saw...

Zippy´s Sausage Factory

There's a very dangerous trait to consider comments a "code smell" and to remove them because "good code should be self-commenting".

I don't hold with that. Good code has good comments, and the comments act as signposts. They're your guides, especially in those fraught times when Something Has Gone Horribly Wrong and you're rooting desperately in code while They Who Must Be Obeyed are standing behind you, tutting, and calculating how much this is costing the company every second it's not working.

Icon because I think we all need it after the PTSD trigger I've just dropped.

Re: Worst code I ever saw...

Hans Neeson-Bumpsadese

When I was coding I often laid out the structure of a function/whatever just in comments, then went back and wrote the code to do what the comments said should be happening. That practice came from writing stuff that interfaced to a number of slightly weird libraries which it was easy to mis-remember parameter order/purpose - coding usually meant frequent reference to API documentation. Unless I laid out the structure in comments first, then by the time I was a few lines in I'd started to lose track of what my actual logic was.

Re: Worst code I ever saw...

Flightmode

Anything that has nested if-statements gets comments put in from the start, otherwise there's no way I can keep track of those curly braces or indents.

Re: Worst code I ever saw...

Anonymous Coward

Many is the time I've had a compilation error that's pointed to mis-formed loops, even though code looks OK on visual inspection. Select the offending code, count instances of "{", get a big number. Count instances of "}", get big number minus 1. Put head in hands and weep while mumbling "oh, FFS - not again..."

Re: Worst code I ever saw...

Zippy´s Sausage Factory

Why do programming languages always choose between begin/end and {/}. On some keyboards curly braces are more painful than begin/end and I'd like the option to use either please.

Re: Worst code I ever saw...

Jou (Mxyzptlk)

As soon as an {} section is more than my (large) screen can display I add a comment at the } to make it clear to which { it belongs to.

if (a -eq b) {

# 80 lines of stuff

} # if (a -eq b)

Re: Worst code I ever saw...

GlenP

I'll admit to having been lax in commenting code for a while - only I'm going to read it so there's no need, is there?

Well, yes there is! I've got to understand it if I have to revise it and as I wend my way towards retirement UI have to bear in mind someone else *will* have to support it in the future. Cue a rush of commenting as I migrate code for a new plant.

A fine tradition ...

Pete 2

>"Some idiot," he writes, "had put an add-update where there obviously should have been an add."

... of fixing the problems you cause. And collecting brownie points into the bargain.

While some organisations will instantly sack the individual deemed responsible for such ballsups, there is a better way!

It is said that experience is the lessons learned from mistakes made. So what better way to gain that experience than by making mistakes. Combine with the other cliche of that which does not kill you, makes you stronger and a convincing argument can be made to retain all but the worst problem-makers: the people who always seem to be in the middle of every corporate disaster.

And in this case, had "Francis" been given the heave-ho, what are the chances that the replacement, would not have also spotted the same idiotic add and changed it to the obvious add-update?

Re: A fine tradition ...

Anonymous Coward

You want to keep the ones who learn from their mistakes, but get rid of the ones who don't. If you can find ones who learn from other people's mistakes, hang on to them as long as possible.

enough knowledge to be dangerous!

chivo243

And quick thinking enough to blame the previous programmer. "It was like that when I got here!" I wonder what position Francis holds now?

Welcome Back!

Lil Endian

This loop looks fun - do it again!

....This loop looks fun - do it again!

........This loop looks fun - do it again!

............This loop looks fun - do it again!

*** Out of memory ***

Nah! Never happens!

Re: Welcome Back!

KittenHuffer

To understand recursion you must first understand recursion!

----------> Mine's the one with the jacket in the pocket of the jacket in the pocket .......

Re: Welcome Back!

Lil Endian

Thanks for the feedback for the feedback for the feedback...

Re: Welcome Back!

elsergiovolador

You need to put a stop condition otherwise you get stack overflow eventually!

To understand recursion you must first understand recursion, until you understand recursion!

Anonymous South African Coward

Always a good idea to have a "brake" clause in a contract when doing upgrades, and force things to slow down, or pause for a couple of hours or days.

Because people will make mistakes when under pressure.

But beancounters don't like this as it'll make projects run over budget.

But mistakes make projects run far more over budget plus the extra costs involved should any specialized knowledge be required to fix things should it be required.

Testify

Anonymous Coward

Currently shoveling large piles of stuff on a project which was procured with the exact opposite - a rigidly fixed delivery schedule that just keeps piling on more trouble while we try and recover from the flaws which should have been sorted on day minus one. Just another day in the public sector.

Self inflicted public humiliation

trevorde

A lot of my code is now open source on github. The horror...

Re: Self inflicted public humiliation

jake

Don't worry, you'll live. And also eventually get over the horror.

Wait until you get hate-email from somebody lambasting code you wrote 40 years prior that hasn't been used for at least thirty years to your knowledge, and only runs on a processor that has been out of support for about as long.

Crisis - What crisis?

Anonymous Coward

"Never waste a crisis" - Attributed to Bill Clinton, though I suspect that he learned it from others!

The code also assumes that it's difficult to misspell "a" or "b". :-)
-- Larry Wall in <199710221731.KAA24396@wall.org>