Coder wrote a bug so bad security guards wanted a word when he arrived at work
- Reference: 1734334331
- News link: https://www.theregister.co.uk/2024/12/16/who_me/
- Source link:
This week's hero we'll Regomize as "Trey" because back in the first decade of this millennium he was working for one of the many startup telcos trying to cash in on 3G. (Sadly, he tells Who, Me? it was not one of the ones that succeeded.)
Trey worked on the platforms and services team, which created and maintained apps for internal users and customers. Among his responsibilities was working with external service providers, such as a payment provider, an identity services outfit, and bulk SMS handler.
[1]
One day, Trey noticed the payments gateway misbehaving, so he wrote a piece of software that sent it a test transaction, checked it had worked, then repeated the process five minutes later.
[2]
[3]
Another experiment saw him write a demo app that automated payments, using SMS as prompts.
The app had its own syntax for commands. In theory, the message “Credit 5” would send that sum to an account, and so on.
[4]
Trey showed the automated payments applications to the head of his department, who was well pleased – so pleased he asked for it to be deployed immediately.
Oh yeah, immediate deployment. That never goes wrong, right?
[5]Panic at the Cisco tech, thanks to ancient IOS syntax helper that outsmarted itself
[6]NetAdmin learns that wooden chocks, unlike swipe cards, open doors when networks can't
[7]Network engineer chose humiliation over a night on the datacenter floor
[8]Undergrad thought he had mastered Unix in weeks. Then he discovered rm -rf
Wrong. It turns out Trey's little demo had exactly three bugs in it that had not been spotted in his limited testing.
The first bug was in the value of the test transactions. The value had to be a whole number, followed by a modifier.
His intention was that the whole number would be 1 and the modifier -2, a combo that would generate a test transaction of $0.01. But the exponent had accidentally been set to 2 – so each transaction was worth $100. Not an insignificant difference.
[9]
The second bug was the lack of a liveness check. If one of the gateways failed, the program wouldn't sleep for five minutes but would simply attempt the transaction again immediately.
The third bug – which Trey did in fact know about but had made a mental note to fix later – was that the choice of credit or debit on the test transactions was supposed to be random, but for some reason always came up credit. He figured it wouldn't be that big of an issue, given the transactions were only supposed to be $0.01 every five minutes, right?
You can easily see where this is headed. As he ran the program overnight, one of the gateways failed. Trey's little proof of concept demo program then began crediting his test account with $100 pretty much constantly for the next few hours.
When he arrived at work the next morning, there were some very serious faces – including a security team – waiting to greet him and find out what sort of fraud he thought he was trying to pull. The account had amassed a considerable fortune by that stage.
Thankfully the head of department, who had authorized the deployment, came to Trey's rescue and explained the situation. Tragically, though, the balance of the test account was reset to zero.
Ever had a programming error make a fortune appear – or disappear – like magic? [10]Tell us all about it in an email to Who, Me? and we may share your adventure on some future Monday morning. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z2AIVtFJjItPH3TcefB-hwAAANM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z2AIVtFJjItPH3TcefB-hwAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z2AIVtFJjItPH3TcefB-hwAAANM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z2AIVtFJjItPH3TcefB-hwAAANM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/12/09/who_me/
[6] https://www.theregister.com/2024/12/02/who_me/
[7] https://www.theregister.com/2024/11/25/who_me/
[8] https://www.theregister.com/2024/11/18/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z2AIVtFJjItPH3TcefB-hwAAANM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Paper trails...
Paper Trey-al surely?
Re: Paper trails...
D'uh.
Defects appearing like magic
When I was involved in development we had a defect tracking system.
One developer used to create empty defects, and then wait. When a tester found a defect, they carefully checked the data base to see if it had been found already, and if not, then raise a defect.
The developer would be given the defect, and copy the details into his own "empty" defect, and suddenly there was a defect saying it had been discovered 2 week before.
The tester would be curious that yesterday the database didn't have a defect - and today it did.
There were suspicions that something was amiss, when there was a defect "raised" two weeks ago about code that was only written last week!
His manager had words with him, and his contract was not renewed.
Re: Defects appearing like magic
but ... why ? What was the benefit of these shenanigans to the developer?
Re: Defects appearing like magic
Possibly to try and pretend that it was something the dev already knew about and was in the process of fixing, as opposed to it being something he'd not noticed when he possibly should have done before sending the code out? (something really obvious if he'd tested it and easy to fix but he wasn't checking the code at all).
A similar issue arose with the early roll-out of some smart meters. The meters would use a mobile data connection to send their readings to the central system. This used the mobile phone network, but meters were often installed in cupboards or locations with poor signal. If the meter failed to connect it would try again. If it still failed, it would wait until the following day and try again. There was a limited time window to send the data, and each failure meant the meter was trying to send more data. As you might expect, the receiving system became saturated and the bills had to be sent out as 'estimated', which was embarrassing for what was promised to be the perfect deployment of modern technology.
I'm sure it now works as intended.
Tounge in cheek?
"I'm sure it now works as intended."
That's a very 'brave' thing to say!
Re: Tounge in cheek?
As long as the estimated was always more than the amount actually owed, then it probably does
Re: Tounge in cheek?
If I was in charge, everyone would have a key meter and pay up-front for the energy they were going to use.
Re: If I was in charge
Damn good job you're not, then!
Oops!
This shows why thorough acceptance tests and code reviews are rather essential. I also would be very hesitant to introduce my own syntax and its special-purpose checker. Why not just demand that amounts are given in accepted forms, and preferably use a tried and tested parser? The former ensures the users enter data in a format they know, the second prevents parsing errors cause the kind of mayhem in this story, like mistaking 0.01 for 100.
We once had someone add a column to our `invoices` table, not knowing that the ancient billing scripts referred to columns by their position instead of their name.
So column[6] no longer referred to the `total` field on the table. This led to us generating and sending out a few thousand invoices with a unix timestamp in the total field. You can imagine the amount of coffee that got spat out in accounts when they came in and saw the pending direct debit payments.
What do you mean accounts saw the DD details... surely it's only customers who see such things.
Accounts would see the unpaid and pending direct debits, certainly the amount and customer account, not necessarily the specific bank account the money is to be taken from.
That's why you always add columns to the end of a table!
Unix timestamps as total values are hilarious though, I can only imagine how badly broken the invoice layouts were by having billion-dollar values.
Paper trails...
...paper trails and more paper trails. This could've been very dicey with a different manager!