The reluctant log trawler: The buck stops with the back-end
- Reference: 1594364586
- News link: https://www.theregister.co.uk/2020/07/10/on_call/
- Source link:
Today's story comes from a reader the Regomiser 9000 has elected to call "Ed" and concerns the fun that comes from bodging around another person's bodgery.
Back in the late 2000s, Ed was working as a developer on IBM's iSeries and, for his sins, was also first line IT support.
His system formed the back-end of a website that allowed customers to buy and sell shares. A request to buy or sell was sent to brokers in real-time to get quotes. Customers then usually had 30 seconds to decline or accept the terms (and have their trade stamped with a unique "Guaranteed Price ID"). If no guaranteed quotes were forthcoming, the customer could also trade at somewhere close to the market price.
It is important stuff and tends to get auditors a bit feisty if there are issues.
Ed was happily tapping away at his keyboard when the first call came in: "Somehow, a trade was sent to the broker for stock A, but with the guaranteed price ID from a quote for stock B." The broker code did not validate for that and so the trade had been processed... "just for the wrong stock."
Could he possibly look at how it could have happened?
Peering at the database tables, Ed was able to see that the customer had quote requests for both stocks. Stock B had returned a guaranteed price ID, Stock A was market price. He could then see the trade for Stock A being sent with Stock B's ID. Not good.
Worse still, there was no obvious way the IDs could have been switched. Unless there was something funny happening with the web front end. Surely not?
He began trawling through the logs to work out the exact path the customer had taken. A thankless task, and one that only he had the knowledge to do (Ed told us he'd tried training others in the dark arts of log inspection, but his students tended to depart the company shortly after).
It wasn't long before he turned up some decidedly whiffy behaviour. Although quotes cannot be saved (just accepted or declined) it looked rather like the customer had multiple browser tabs open at once and was working with several orders at a time.
Surely the web front-end could handle that though? After all, pretty much all the major browser makers had a tabbed interface by then, so such behaviour was not unheard of.
It transpired that it couldn't. Ed was able to recreate the issue with a bit of faffing and multi-tab shenanigans: "If one quote had a data value and the other did not, the web app could end up filling in the blanks for one with data from the other. So the blank guaranteed price ID data item in one quote was replaced by the populated value from another quote," he told us.
Chuffed with his sleuthing, he submitted his findings and thought no more about it. The web team would probably have some work to do at some point, but the issue was not with his precious iSeries.
In our experience of the trading world, such a discovery could set off all manner of panic bombs. So rather than the peaceful weekend he was expecting, Ed received a surprise Saturday night call from the bosses.
"Apparently," he said, still sounding a little aggrieved, "they felt it was such a massive flaw that, despite this being the only occurrence in the system's 10-year life, it needed to be fixed before start of business on Monday."
Even worse (and Ed told us he still recalled the exact words a decade or so on) he was told: "We feel the cause of, and the solution to, this issue is on the iSeries."
The inability of the web app to handle multiple tabs and mangle customer data was not the problem. It apparently lay with the iSeries backend not validating for that particular type of borkage coming down the pipe.
Sunday was then spent on the fix which, Ed admitted, didn't take too long.
"What took the whole day," he grumbled, "was bodging the test broker trading interface to produce quotes on a Sunday when every aspect of its design was set up to make it only work on weekdays."
Still, he pushed on through and got the fix live, although we fear he may have missed that week's Antiques Roadshow. "I was not thanked nearly enough..." he growled.
Ever found yourself doing some hurried hacking with the "unthinkable" happened? Or been called out to bodge your way around someone else's cock-up? Share you story of unexpected weekend working with an email to [2]On Call . ®
Get our [3]Tech Resources
[1] https://www.theregister.com/Tag/on-call
[2] mailto:oncall@theregister.com
[3] https://whitepapers.theregister.com/
Re: From the "if you have to ask" files ...
Surprised you didn't work IR35 into this complaint about your job!!
Re: From the "if you have to ask" files ...
If @Jake had to deal with HMRC he'd be having a bad day. I guess he's got his plate full with the IRS.
Re: From the "if you have to ask" files ...
Yes, that is why we have over-tuned cattle-prods, erm, I mean cable testers, and pinches, erm, I mean wheeled suitcases.
Bodging someone else's cock-up
Obviously there are many 'bodges' in all our pasts, but during my time on-call at a financial institution, more often than not it was coding correctly for someone else's half-assed code that had made it to live and, on several occasions, because there was some data missing from a product configuration causing the system to try and do a divide by zero. Sometimes I've just flagged the problem account and product (and checked for others that might fail in that run) for admin to fix the next day, with a note to get the config screen fixed, other times I've fixed the config screen there and then, middle of the night, to prevent zero values being entered. I've often referred the issue back to the original developer, but almost always I ended up being the one who had to fix it because they were now working on 'something new'...
Fault at both sides
To be fair here, I think there's a measure of blame at both sides. The web interface should handle multiple tabs, but similarly the iSeries back-end should validate what it is receiving - after all, someone nefarious could otherwise try and manipulate the submissions from the web form.
Still, Kudos to Ed for his troubleshooting skills for this one.
Session ID?
Why was there no tracking of the uisessionid?
Sounds a bit odd to me.
Re: Session ID?
Multiple tabs would share a session ID. This is a red herring.
Seems like the web developers need to meet a lift shaft.
Is that you, BOFH?
>Kzzzeeerrrtt<
Nothing to see here, move along, its Pub o'clock.
Many, many, years ago, I was asked to "white hat" a financial transaction system. (Most definitely not my normal job of assembler programmer!) Multiple video cameras pointed at screen, keyboard and me, institutions representative also watching (actually reading crime novels most of the time), specified account to place syphoned money into, pile of video tapes (that gives the age) collected daily by secure courier, etc.
I only managed to transfer funds twice. The first was very obvious social engineering to get passwords and left an audit trail . The second was using a similar (multiple concurrent transactions) technique to this story (but obviously not web based that long ago). I managed to get the money into the correct account AND my special syphon account without any of the checks and balances noticing that I had 'invented' £10M! It was fixed when I was allowed to try again 2 days later.
I got £1000 (two months wages) as a bonus from my companies owner. I now wonder what percentage of what he received that represented, but I was very happy at the time.
I did some white hat testing back in the early 00's.
"You have a SQL Injection vulnerability in your eShop."
"Not important, it works."
"I could insert orders without payment."
"I don't believe you."
"I could disrupt the site."
"Couldn't happen!"
>clickety<>clickety<
"Hey, where has our site gone?"
"Oh, did I just inject 'DROP DATABASE;'?"
(It was on the test system, but still left the devs a little red faced.)
Hurried hacking ...
Well there was the time, around 15 years ago, that the Canadian subsidiary of a rather large engineering company let its domain expire. We had a project with them at the time, so naturally my users had a lot of bounced e-mails. This, of course, was my fault (by definition, anything that goees wrong with e-mail is my fault) and the messages could not wait until the domain registration was fixed.
I added an authoritative zone for the company's .ca domain to our internal DNS, which stopped the questions about when I was going to fix someone else's domain registration.
Posted anonymously on the off-chance that my Reg user name might identify the guilty party (seems unlikely, but best to be safe).
Re: Hurried hacking ...
"Posted anonymously on the off-chance that my Reg user name might identify the guilty party (seems unlikely, but best to be safe)."
just change your reg name every few years, thats what i do
Anon because......................
Multiple tabs doom
I learnt long ago that trying to do two things in parallel in tabs is *dangerous*.
Bit of a windfall, not sure what to do, so of course I wanted to open one fixed-rate fixed-term saving account, and one variable-rate to spread the risk of inflation.
Filling in many tedious details in two windows next to each other. Disaster; neither opened successfully, but at least neither debited my money either. (TBH it could have been so long ago that "apply online" meant "fill in a form online and we send you forms, you sign and return a cheque" - I can't remember)
At least Amazon seems to get that right these days.... I'd still be scared to try it with real finance.
Late 2000s?
Back in the late 2000s ...
Hmm ...
Maybe it was in late 2000?
Still some time to go for the late 2000s.
ie: 2075 -> 2099.
O.
From the "if you have to ask" files ...
"Ever found yourself doing some hurried hacking with the "unthinkable" happened? Or been called out to bodge your way around someone else's cock-up?"
Well, yes. Of course. Weekly. Sometimes daily. It's in the job description.