New year, new bug – rivalry between devs led to a deep-code disaster
- Reference: 1704702613
- News link: https://www.theregister.co.uk/2024/01/08/who_me/
- Source link:
This particular Monday meet a programmer we'll Regomize as "Jack" who had a rival we'll call "Irving." The two of them were programmers at a small concern that made business software for the then-popular Amiga platform. That in itself should give you some idea of the time frame in which our tragedy takes place.
At the core of their rivalry was their flagship product – a productivity application. Irving had been the principal developer of version 1 of the program, but had moved to wider responsibilities in the organization (such as it was – there were no "big" Amiga developers). Jack, therefore, did the lion's share of the work on version 2.
[1]
That work, incidentally, included rewriting a lot of Irving's old C code in faster Assembly language. Like the Ship of Theseus, not much of what Irving had created remained in the new version.
[2]
[3]
Nonetheless, Irving felt that, as the originator of the product, his name should go first in the About box for version 2. Jack, for his part, felt that it was OK to give some credit to Irving, but version 2 had been his baby and he wanted top billing.
To cut a longish story shortish, Irving was buds with the guys who founded the business, so he won.
[4]PLACEHOLDER ONLY Someone please write witty headline here
[5]Enterprising techie took the bumpy road to replacing vintage hardware
[6]Sysadmin's favorite collection of infallible utilities failed … foully
[7]IT sent the intern to sort out the nasty VP who was too important to bother with backups
Version 2 came out toward the end of the year to much acclaim – with Irving's name first in the credits.
Then came January, and staff at the software mill were greeted upon their return from holidays with irate messages from unhappy users. The program, which had been working fine a few weeks before, simply would not work.
[8]
Strangely enough, it worked for some users, but for many others it was offering nothing but [9]Guru Meditation (if you know, you know).
Remember of course that this was a productivity application, so there were actual businesses that were relying on it. Those customers were not amused, so Jack and Irving's bosses were equally irate.
When the bug was discovered, it turned out to be an instruction with an odd memory address. Users with newer machines – like the Amiga 2000 – were not having problems because the 68020 processor could take an error like that in it's stride. But anyone with an older 68000-based Amiga inevitably crashed.
[10]
Just what was this poorly formed instruction, and why was it not picked up in the process of debugging?
Well, you've probably guessed that Jack, in a petty moment, had hidden a little routine deep in the code – in Assembly, so the C compiler didn't notice it – that would swap the names around in the About box. And in order to avoid anyone seeing it before or even on launch day, he'd set it to execute on the first day of the new year.
As a developer, Jack obviously had an Amiga 2000, so it ran just fine when he tested it. Oops.
Sadly, Jack's tenure at the small development concern he'd sabotaged did not survive the incident. But an important lesson was learned: always test your code on the lowest-spec machine your users are going to have.
Has hubris ever brought your grand schemes undone? Have you ever found yourself in a battle of wits with an Irving? Or a Jack? Whether you won or lost, tell us about it in [11]an email to Who, Me? and we'll share it on a future Monday. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZZvV1ysy6rWQvqHIi9pgZgAAAYc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZZvV1ysy6rWQvqHIi9pgZgAAAYc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZZvV1ysy6rWQvqHIi9pgZgAAAYc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2023/12/18/who_me/
[5] https://www.theregister.com/2023/12/11/who_me/
[6] https://www.theregister.com/2023/12/04/who_me/
[7] https://www.theregister.com/2023/11/27/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZZvV1ysy6rWQvqHIi9pgZgAAAYc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.youtube.com/watch?v=d_oCPW47N-0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZZvV1ysy6rWQvqHIi9pgZgAAAYc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] mailto:whome@theregister.com
[12] https://whitepapers.theregister.com/
Sadly, Jack's tenure at the small development concern he'd sabotaged did not survive the incident.
So he jacked his job in?
Is that icon you, getting your jacket?
Jack and Irving
Jack and Irving - I like what you did with the names this week. Well, actually, as an Amigan (I still use it under emulation), i don't like what you did with the names - I'm starting to get cold sweats already and I'll probably have nightmares tonight :-(
The real lesson...
Don't (re)write C etc code in assembly. While you are are smarter than the compiler the compiler is a plodder (like Irving:) and usually doesn't make alignment and other foolish mistakes. Get a better compiler or pass the suboptimal assembler code emitted by the existing compiler through your own optimisation stage.
Compilers back then did sfa* optimisation so even adding a peep-hole optimiser was a win.
*classical technical term rendered by Terry Pratchett as adamus con flabello+ dulci [Jingo]
+sir Terry was a gentleman.
Re: The real lesson...
There are times that the difference in performance matters.
In my graphic computation project in uni (30 years ago....God, I'm old!), I used some embedded assembler code in my Visual C++ project to load a polygon vector file. While most of my colleagues' code took several minutes to load the largest files, mine did it in a couple of seconds - it loaded the first file so quickly the teacher first thought my code just didn't work - he only realised it had done it's thing when I told him to press the menu button to show the 3D object and all worked :)
Re: The real lesson...
The Linux kernel devs are currently replacing all the assembly with C because with modern compilers C is actually faster in most cases.
Assembly only really has a place is things like audio and video codecs which are less code and more mathematical wizardry.
Re: The real lesson...
There are arguments for saying that even audio/video codecs can be written in C with no loss of performance - modern optimisation techniques have got so good that assembler language is only really useful in a few edge cases (e.g. embedded board boot-up code).
Re: The real lesson...
I remember a university assignment [redacted] years ago to implement the same sorting algorithm in C and in assembler, to compare the relative speeds. The C code ran in a few seconds. The assembler seemed to me to be already done as soon as I had pressed 'Enter' on the command line invoking it.
(of course it's not to be taken lightly, and there are indeed many things that can go awfully wrong coding in assembly.)
Out in the fields
Not only the lowest common denominator of hardware, but also in the range of common usage scenarios.
We had something recently here which when rolled out royally screwed up everyone working in the field.
Everything had been "tested" of course before roll-out, but always from the comfort and safety of the company LAN when sat in a cosy office.
But when run on-site with the customer breathing down your neck and access only via VPN into the mothership network, the less than optimal speed and other under-the-hood differences between the connection methods were enough to completely screw up the remote users.
And given all this was actually aimed primarily at field usage, there were some interesting questions asked at the post mortem as to why it hadn't actually been field-tried before release...
Strange story about field testing
I used to work with a multinational team that tested telephone networks in the time of GSM rollout.
They had to drive around the country in a van with multiple handsets mounted on a rack and monitoring the signal reception parameters.
They had to test in Greece, but the Greek speaker was off on some kind of illness leave.
They sent the Chinese speaker as he could at least order what food he liked from the Chinese take-aways for the 2 weeks of his stay.
Never worked on an Amiga, but I thought the "Guru Meditation" error messages had a lot more style than the bland "segmentation fault" messages that drive so many of our students to distraction.