You had one job... Just two lines of code, and now the customer's Inventory Master File has bitten the biscuit
- Reference: 1597043711
- News link: https://www.theregister.co.uk/2020/08/10/who_me/
- Source link:
Today's catastrophic coding comes from a reader we will call "Ed", for that is most definitely not his name, and takes us to Canada in the late 1970s.
Ed worked in Technical Support and was at the sharp, pointy end of client calls. The systems he supported were written in Business BASIC and ran on Data General Nova minicomputers.
The Nova first appeared at the end of the 1960s, and gained popularity during the 1970s before fading as the age of the microcomputer swept away the competition. Data General itself endured until the end of the 1990s before being snapped up by EMC.
"All the computers," recalled Ed, "had acoustic couplers so that we could connect remotely and work on their systems."
The team dealt with all manner of calls from customers, usually just fixing programming errors such as field overflows or unexpected user input.
On the day in question, Ed was asked by a customer to perform a relatively simple task. They were about to do their annual inventory and could he do something nifty to zero out the on-hand stock quantities for all their products?
The customer would then go through the list and enter the actual, eyeballed amounts. "This was in the days before barcoding and .CSV imports," Ed explained.
Are you sitting comfortably? Then we'll begin. Hang on, the PDP 11/70 has dropped offline [2]READ MORE
The task was simple. Ed tracked down the Inventory Master File on the customer's system and found the field that contained the on-hand quantity. He had a program that would do a file dump of any Master File. Normally it would simply read a record, dump the contents, increment the record number and repeat until it reached EOF.
All he needed to do was add a couple of lines: one to set the on-hand quantity to zero and another to write the record back to the file. What could be easier?
A simple two lines.
"That just took a few minutes," said Ed, recalling the confidence with which he bubbled, "after which I called the client and told him it was done.
"I closed the call, marked it as billable and went on with my day."
The client was back on the phone an hour later.
"Hi Ed, it's Bob. We're having a weird problem. When our Order Entry clerk is entering Orders, every product description comes up as 'Brittany Biscuits'. What's going on?"
"With horror," Ed told us, "I realized what I had done...
"When adding the two lines, I had placed the write record statement after the record number increment line, instead of before.
"So now the program read record 1, reset the on-hand count to zero, incremented the record number and wrote the record as record number 2! Ooops.
"Every record in the Inventory Master was now a copy of the first record in the file. Every product description and price were the same.... Brittany Biscuits [it wasn't – the name and business has been changed to protect the guilty].
"The associated index files were unaffected which is why the Order Entry program continued to work.
Unfortunately for biscuit enthusiast Bob, there was no simple fix. The only backup was from the night before, meaning that the customer lost all the work that had been done that morning. Still, at least Ed had remembered to tick the billable box for his efforts in shredding the customer's data.
With a good deal more caution, Ed then re-did his earlier code tweak, but this time got it right.
"From that day on, I have been very careful when making any changes that write data back to the database."
Do you blithely churn out potentially data-destroying code without a care in the world, or is your stomach tied in more knots than a Scouts rope project as you hit 'Execute'? Share your story with an email to [3]Who, Me? ®
Get our [4]Tech Resources
[1] https://www.theregister.com/Tag/who-me
[2] https://www.theregister.com/2020/07/27/who_me/
[3] mailto:whome@theregister.com
[4] https://whitepapers.theregister.com/
Re: Adding a comment sometimes caused compile failure
So one of the few times "Well, it worked on my machine" was valid...
Re: Adding a comment sometimes caused compile failure
Few times? Back in the day, that was quite common. As you would expect when each machine was essentially a hand-built, one-off prototype.
Re: Adding a comment sometimes caused compile failure
There's a well known quote that says:
"There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors."
Re: Adding a comment sometimes caused compile failure
off-by-one errors:
When Scottish independence was being even more hotly debated than it is now, I wanted to find out how many Scottish members there were in the House of Lords*.
I went to the Government web site, asked for a list of peers, and was rewarded with the response:
"Found, 781 of 780"
*The House of Lords is an anachronistic legislative chamber peculiar to the (currently) United Kingdom and the British Commonwealth which contains some wise and venerable people selected for their great knowledge and ability, but mostly politicians who did so well in public life that they had to be removed from the House of Commons (The elected body) to where they could cause less damage, plus some actual hereditary peers (who get to stay there by being voted in by other hereditary peers), and of course senior public servants like old civil servants (whose devotion to a political master was so great it needed / deserved reward), former top police officers and chiefs of the general staff, and Norman Tebbit, Jeffrey Archer, and Andrew Lloyd-Webber.
Re: Adding a comment sometimes caused compile failure
AKA the Plus One Bug.
Re: Adding a comment sometimes caused compile failure
Reminds me of the IBM 360 in my uni back in the 80's that kept crashing. Frustratingly coinciding with every time I tried to compile my COBOL program. Turned out there was a a critical full stop missing from my program and a bug in the compiler that couldn't handle this particular coding scenario and crashed the computer. Oops.
Re: Adding a comment sometimes caused compile failure
Faults like that could be beneficial to students as well.
My first year at university coincided with the introduction of a nice,shiny new Harris H500 super-minicomputer that the CompSci dept used for its undergraduates. It almost worked fine, just some minor little problems with the OS that meant that it Harris completely re-wrote it and installed the new OS (with a name change) over the following summer break, but I digress. Another nearby university also had purchased a Harris super-minicomputer, but this was a H1000 - the big boy of the range. However it kept on crashing just before course work had to be submitted, and it took the operators some while to work out why. It turned out that students had discovered that submitting a JCL command that added three variables and stored the result in a fourth variable (i.e. "A = B + C + D") crashed the OS.
Sadly it was only the big machine that had this fault; our smaller system didn't :-(
Re: Adding a comment sometimes caused compile failure
An unexpected period or missed period can ruin your day, as most women know.
Re: Adding a comment sometimes caused compile failure
I remember one of colleagues submitting a compile to batch and sitting back to read the paper while it happened.
The he went "Oh Shit!" and dived for his keyboard. He'd forgotten to make one important tweak and figured he could get the source updated before the compile happened to save himself half an hour (yes, it took that long).
What I reckon happened next is that with impeccable timing he got the source updated smack between the initial compile to deck and the optimisation phase so, when it referred back to the source, things didn't match.
Only time I ever saw an entire IBM System/38 crash like that, those things were bulletproof. He insisted that it was a coincidence and I know correlation is not causation, but...
Re: Adding a comment sometimes caused compile failure
I remember working on a McDonnel Douglas mini at Freddie Laker's airline. They had 64KB pages and code and data had to fit into a page. Every change to a program had to be carefully calculated as to whether it would still fit in the 64KB page with its data records.
SQL interpreters should have an option to disable any adhoc sql update or delete statement without a where clause.
where 1=1
Warning 911666: Programmer tried to circumvent other programmer's stupidity filter. Conditional statement will evaluate to False for our security and your convenience.
where 1=1
Of course, but a person has to be thinking about a where clause to write a where clause.
To often people are thinking only about the columns that they want to change and not about the the other millions of rows.
@werdsmith
I always write a select statement first, then wrap the update in a transaction. if the rows affected != the select row count something went wrong!
Interestingly, my boss uses WHERE/IF 1=0 AND... when he has to disable a conditional statement, so that it never runs. Instead of removing it or commenting it out, it is just set to always fail "in case we ever need it again".
The example here definitely wasn't SQL.
However yes, such an option would make things slightly safer when writing ad-hoc SQL data changing statements. I am in the habit of typing out the where clause first, then the lines that do the update or delete . I practice paranoia for good reasons...
The example here definitely wasn't SQL.
And the STFO award is won.
But yes, the better way to do it is to create a select to make sure the correct information comes back and then convert it to a change. But may devs know better than this and they know you have a backup so they don't care.
Did it still do the dump?
Just wondering because if all he did was add the lines to set the value to zero and write back the record, there should be a copy of the pre-update data in the dump file. Possibly not a lot of help, but sometimes you are blessed and have a "last updated" date/time on the record which can be used to find any updates made since the backup...
Of course, this is back when every byte counted so adding "helpful" attributes which were rarely used was considered wasteful.
10 format c: /y
20 goto 10
>Save as "autoexec.bat"
*reboot*
I'll get my coat on my way to the pub...
Re: 10 format c: /y
Haven't written an autoexec.bat file in this millennium but if my aged brain is correct then you won't get very far on your quest to the pub. DOS batch files didn't use line numbers. All you are likely to get is two "Command not found" errors for "10" and "20".
Re: 10 format c: /y
They did have labels and a GOTO command though, so it'd still be possible to code that loop.
Trivial Biscuits
Brittany Biscuits... mmmm! I was hoping that Brittany Biscuits means something lewd in canadian, but the story wasn’t that juicy after all.
Anyway, it shows once again that there is no such thing as trivial change!
Re: Trivial Biscuits
Should have used Larsen's Biscuits. Probably would have brought a smile to the face of many a bitter admin this Monday morning ...
When DB stands for "Damn and Blast"
I worked for a company who had a similar tale where we ran a competition for a client. The Developer who wrote the competition accidentally forgot to remove a line of test code when the site was deployed live. The result was that at the end of the competition there were some 50,000-odd entries - all with the exact same name and email address. The test code had basically copied the first line of the DB into every entry when a submission was received.
Now - under normal circumstances, you could have covered this issue up because you could have just said - OK then, that first guy/gal wins the prize.
Problem was - there were 5 prizes...
I believe they extended the competition deadline by a couple of days, so if you entered after the deadline, you stood a much better chance of winning.
Mainframe systems often used hierarchical rather than relational databases. Each database had it's own API. SQL was not commonplace.
The data was generally stored in flat files, and the DBMS, was so slow that it was totally impractical to use it if you needed to process every record in a large table.
For speed and efficiency, it was common practice to reverse-engineer the structure of the database files and edit them directly from your application rather than use the DBMS. Not just in the 1970's either. I was working on banking systems that did that right up to Y2K.
Ahh, Back in the days when the only way to recover dead space from the database was to write the data out to a mag tape then read/write it back into the database again....
Damn, I feel old.
And if the mainframe you were using did support a relational database, the licensing costs where often so high that it was cheaper, even including the womanhours spent writing it, to implement a relational database on top of the hierarchical database that you'd already paid for. That was a fun project.
Defensive Coding
Back when I was a coal face programmer, coming from a VB background, I got caught out by this C error 1 time too many:
if (a = 1) {
// Bug: code here always executed irrespective of 'a'
}
Of course the first line should be: if (a == 1) {
Before compilers cottoned on this common mistake and issued a warning, I started reversing the variable and constant, so:
if (1 == a) {
because if I accidentally used a single = the compiler would catch it.
Oddly I never convinced any other code to follow suit. They just didn't like it.
Re: Defensive Coding
Just last night I was getting equality and assignment operators the wrong way round. That's after decades of using them.
Re: Defensive Coding
Saturday night? You probably needed either one more, or one fewer beer ... Either way, it was clearly an off by one error.
Re: Defensive Coding
I understand the reasoning behind reversing the operands, I just don't like it. It doesn't scan well in my head.
In any code base I touch these days (usually C++) the "assignment in conditional expression" warning is promoted to an error.
And just don't get me started on the if( type variable = expression; variable ) syntax of C++ these days. I'm feeling nauseous just thinking about it.
Re: Defensive Coding
"-Wall -Wextra -Werror" and then test it under valgrind ;)
(or these days, "-fsanitize=address" does about the same but faster)
Re: Defensive Coding
Many years ago I had to use a vendor provided custom language for application development... I've manged to erase the name from my mind, but it ran on VAX VMS systems...
The interpreter didn't cache values, it was all immediate by reference therefore there was a distict difference between the two statements:
A = A + 1
and
A = 1 + A
The first would operate as expected, therefore starting with A assigned the value of 5, after the statement A would have the value of 6.
The second, on the other hand, would assign 1 to the A and then add A to itself, always producing the result 2 no matter the starting value.
In reality what the first statement did was to assign the value of A to A and then add 1 to this afterwards. Slightly inefficient but it worked.
That was not an amusing "bug" to track down and find. Erm, "feature", that's it, definitely a feature and absolutely not a bug. We weren't sure what to call it then, it caused a bug in the application but was it a bug in the interpreter? It wasn't operating as expected in line with other environments.
Re: Defensive Coding
Definitely a compiler/interpreter bug. Which of course means the application has bugs too.
More RAM - program fails
I had a case where adding RAM to a computer made the compiled code fail, due to extra compiler optimization that was done when more RAM was available.
Re: More RAM - program fails
Ouch, that's harsh!
I thought it was bad enough when Windows volumes went past what the number of bytes that could be represented in a signed 32 bit integer. As a result many applications failed to install on larger drives if the amount of disk space remaining wrapped round in the negative values of a signed 32 bit number. What was particularly annoying is that this affected multiples of the is value therefore having 5GB free was OK, but having 3GB was not because that was returned as a negative number of bytes. Something like that anyway... it was a long time ago...
How's August working out for you?
August? I thought we were still in March! Is it not the 163rd March?
Re: How's August working out for you?
No. It's still September.
Monday, Sep 9841 1993 to be precise.
Re: How's August working out for you?
Didn't you hear that Eternal September ended on January 25, 2005 (when AOL stopped Usenet access)?
https://slashdot.org/story/05/01/25/1628233/aol-kills-usenet-access
AOL
it metastasized.
I once ...
... worked on a project where we had a database set up by two people (nameless to protect the guilty, as ever).
Anyway, on my first use fo the database I accidentally entered some incorrect data in two fields. I couldn't find the 'undo edit' button, or the 'exit without saving button' to return to the original data, so sent the two an e-mail, along the lines of:
"Hi, I was using the database and accidentally entered incorrect data in these two fields. Couldn't find a way to return to the original data, can you re-set, please?"
But answer came there none.
Then the database went 'down' for a couple of days, and came back up again.
It seems that these two had set up data base access for everyone on a 100 person project, so that anyone could edit any field with the actual edit happening and being saved in real time, and no back-ups.
Backups
I can remember a time when the hardware engineer pulled the wrong drive and trashed the raid 5 array. NT 3.51
The restores did not work.
This was one of my first roles in desktop\server support, and I spent the night rebuilding the server from scratch as the actual server guy went home. (The server guy was not our normal server guy who was on holiday). The data volumes were ok, it was just the OS volume that would fail to boot fully after the restore.
I then spent some more time looking at the backups, and the actual problem was that the admins weren't removing the home drive shares when removing user accounts.
This cause the netlogon process to take too long to start, which then caused the follow on services to fail to start.
The server guy wasn't asked back again.
Adding a comment sometimes caused compile failure
This is going back 30+ years when 1MB was a large machine.
Someone added a comment to some code, compiled it successfully, and submitted it to be integrated. This compile failed with a syntax error. Because it "broke the build" there was an investigation.
The team leader spoke to the junior programmer and stressed the need to compile before integrating. "I did compile it" said the trog*. Team leader said "ok we'll compile it again and show it fails" "oh it works - hey Charlie... here's an interesting problem..." and so on up the chain"
It turns out that when the original program was compiled, the source would fit in memory. When the comment was added, it was too large for memory, and so used "the spill file". This had a bug where it missed a byte when reading from the spill file - and put a garbage byte in the line.
* trog - troglodyte: in pre-historic times, someone who lived in a cave.