Rookie programmer's code goes up in flames ... kind of
- Reference: 1665387008
- News link: https://www.theregister.co.uk/2022/10/10/who_me/
- Source link:
Our correspondent, who we'll Regomize as "Bernie", had just landed a job in the IT department of a well-known chain of grocery stores. Each store had its own central computer, and code updates would be deployed from head office to remote stores, tested, then deployed across the chain.
Head office featured a "wall of shame" upon which programming errors were immortalized. Like one programmer whose bad code led to frozen goods being delivered to a warehouse without adequate refrigeration.
[1]
Bernie did not want to end up on the wall of shame.
[2]PC component scavenging queue jumper pulled into line with a screensaver
[3]Bad UI killed the radio star
[4]Chemical plant taken offline by the best one of all: C 8 H 10 N 4 O 2
[5]Salesperson's tech dream delivered by ill-equipped consultant who charged for the inevitable fix
So Bernie was careful. Bernie always tested code before deploying. Bernie took no risks.
Until …
[6]
[7]
One day, he was entrusted with an update to production code. This had to be deployed live.
Bernie deployed the code to one guinea-pig store, and tested it there. It worked. Sigh of relief. Intending to deploy to the wider network the next day, Bernie went home.
[8]
The next day, Bernie found unhappy faces all around. Red flags were popping up all over the system since the prior night's update, and questions were being asked: What did you update? Did you test the code? Was it compiled correctly?
Then, the heart-stopping news: management announced that the guinea-pig store to which Bernie had deployed the code burned to the ground last night. Ashes and rubble.
Oh boy, our hero worried, this is going to make the wall of shame for sure. This is worse than melted ice cream even.
[9]
As it turns out, the fire at the store had nothing to do with the computer and computers pretty much never actually explode from bad programming outside of science fiction and bad TV shows anyway. Bernie was off the hook.
Nonetheless, Bernie was very very careful next time they had to update code in production.
What about you? Ever actually destroyed a building (or thought you did) with technology decisions? Tell us all about it with an email to [10]Who, Me? . ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y0PtONsc2WP6@NlQglqrFgAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://www.theregister.com/2022/09/26/who_me/
[3] https://www.theregister.com/2022/09/19/who_me/
[4] https://www.theregister.com/2022/09/12/who_me/
[5] https://www.theregister.com/2022/09/05/who_me/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y0PtONsc2WP6@NlQglqrFgAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y0PtONsc2WP6@NlQglqrFgAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y0PtONsc2WP6@NlQglqrFgAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y0PtONsc2WP6@NlQglqrFgAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Was the update deployed? Or not?? If so, When?
Presumably they knew that the guinea-pig store had been given the update.
I'm guessing the red flags were mostly "this entire store dropped off the system" and second-order impacts from that (batches failing instead of skipping the store, maybe).
Re: Was the update deployed? Or not?? If so, When?
They took one look and figured he'd Bernied the store the ground...
Re: Was the update deployed? Or not?? If so, When?
It doesn't really matter.
As we all know, as soon as you've touched a piece of code, whatever happens next is your fault.
Vital detail missing
> the guinea-pig store to which Bernie had deployed the code burned to the ground last night
But what happened to all the guinea pigs?
Re: Vital detail missing
"We have a one-day special on fried guinea pigs"
Re: Vital detail missing
This reminded me of "School's Out" - a type-in adventure game for the 48k ZX Spectrum.
One of the objects was a guinea pig, which you have to feed beer in order for it to get drunk and stagger into a laser beam... and you'd have a hole in the wall and a fried guinea pig.
Re: Vital detail missing
Very popular - and from recent observation, not inexpensive - in Cusco and other Peruvian tourist centres.
Re: Vital detail missing
It's a delicacy all around Peru, not just in tourist traps
Re: Vital detail missing
And where is the playmobil incident picture? Surely playmobil have guinea pig figures?
Re: Vital detail missing
Might be Budgens in Holt (Norfolk) a couple years ago - electrical fire in the roof burned the place to the ground (no flare gun required :P) .
Re: Vital detail missing
"burned the place to the ground"
Thanks, now I have that stuck in my head :)
Re: Vital detail missing
Might be Budgens in Holt (Norfolk) a couple years ago - electrical fire in the roof
Nah, they haven't got "that electrickery" in Norfolk yet...
Halt And Catch Fire
I worked on cheque processors years ago and one of the machines had a device fitted that, if given the wrong value in its command string, would engage all it's solenoids and bust into flames.
It never happened with released code but did a couple of times with beta.
I never really found out why it was beyond the abilities of the writers of the controlware to screen out these problematic commands.
Re: Halt And Catch Fire
"if given the wrong value in its command string, would engage all it's solenoids and bust into flames."
Bring me the heads of all those who signed off on the clearly deficient hardware. No platter necessary, just the heads.
Re: Halt And Catch Fire
> I never really found out why it was beyond the abilities of the writers of the controlware to screen out these problematic commands.
Yeah, they should have chequed for this...
Is this me or not?
In various spheres of endeavor I've come across the logic:
"You did something, therefore your thing caused this completely unrelated problem"
My girlfriend particularly is good at these kinds of leaps.
Re: Is this me or not?
The existence of the Latin phrase "Post hoc, ergo propter hoc" -- "after it, so because of it" -- would suggest that the tendency to blame people for any problem with anything they were the last to touch goes back at least to ancient Roman times.
Re: Is this me or not?
I've had that kind of situation before, as a developer in a professional setting. Also working for a well-known shop chain, as it happens.
I'd updated one bit of code. Maybe five minutes later, something apparently unrelated fell over. My supervisor told me to go take a look at my code and find out how it had caused the issue. To start with, I agreed. It is a reasonable inference to make off the cuff.
An hour later, I reported back and said that I couldn't find any link. I was told to go back and look again.
A further hour later, I reported back with the same. Once more I was told to go back and continue looking, as my supervisor didn't believe in co-incidences. I pushed back, saying that if there was a link I can't find it, someone with better knowledge of the system needs to take a look.
He took a look. My change hadn't caused the issue - someone else had uploaded a change at the same time and had been keeping their head down the entire morning. Fair play to my supervisor, he did apologise.
Re: Is this me or not?
We once had a vendor come out to upgrade the OS on an instrument controller PC, the hard disc chose that moment to die and the technician got full blame from the scientists...
I ended up going over there, explaining that it was bad luck[0] and digging out a spare disc which meant he survived the day...
[0] Of course the disc wouldn't have had much longer to live and OS upgrades are IO heavy
Re: Is this me or not?
If you think its bad now, just wait until you get married.
You'll get blamed for doing something/not doing something before a problem even occurs. Sometimes weeks/months/years later.
The "I told you so" effect.
I've seen updates/changes falsely blamed for many problems
But a building burning down, never.
I suppose the upside of a deploying a bad update that burns down a building is that all the evidence that can prove it was your fault goes up in flames along with the building!
Déjà Vu
Are we on a time loop? I'm sure I've seen this story here before .....
Re: Déjà Vu
Recycling comments?
Bernie
I think the Regoniser did rather well with Bernie, a pint for Mr Powell -->
Was the update deployed? Or not?? If so, When?
Bernie deployed the code to one guinea-pig store, and tested it there. It worked. Sigh of relief. "Intending to deploy to the wider network the next day*," Bernie went home.
"The next day*," Bernie found unhappy faces all around. Red flags were popping up all over the system since the prior night's update, and questions were being asked: What did you update? Did you test the code? Was it compiled correctly?
Am I missing something obvious? It happens.