One bank's brilliant upgrade was another bank's crash
- Reference: 1715578215
- News link: https://www.theregister.co.uk/2024/05/13/who_me/
- Source link:
This week, meet a reader we'll Regomize as "Nick" whose career in finance once had him working for "a large US bank in the City" (for non-UK readers, that means the financial district in London).
One of his functions was keeping an eye on market data, which was sent out regularly by services like Reuters among others. These were early days, so the system was not all high-res and fancy like it is now. Users would type into an X-terminal, and Reuters would send its reports in a standard format: 64 columns by 17 rows.
[1]
Most importantly, all the banks' systems were standardized on that format, so that as data was sent out everyone could read it with no issues of formatting. You would enter a code for a particular company – say, BP for British Petroleum – and up would come the latest info that the markets might need to know. Not a lot of information fit in a page, but it was functional.
[2]
[3]
Of course, as we all know, technology inevitably improves with the relentless march of time. And one day Reuters sent out a memo informing users that it was trying out a new format: 80 columns by 25 entire rows. Can you imagine the vast panoramas of data that were promised?
It was, of course, just a beta test, so don't go trying it unless you know your system can handle the new format. You don't want to go crashing your bank's computers, now.
[4]
Well, Nick was confident his bank's systems were up to date and bleeding edge, so he went and had a go. He entered BP.L – where the L indicated that he wanted the long form document – and his screen automagically reformatted itself to the new size. Up came a massive page with everything he could want to know. And space for more!
[5]Techie's enthusiasm for decluttering fails to spark joy
[6]Software support chap survived breaking his customer
[7]Rarest, strangest, form of Windows saved techie from moment of security madness
[8]Tired techie 'fixed' a server, blamed Microsoft, and got away with it
He showed it to a colleague, who was duly impressed. The colleague suggested he might tell a friend of his, who worked at a different bank. Nick recounted for Who, Me? the half of the conversation he heard:
"Hi there, how are you?"
...
"Hey, have you seen these new large pages from Reuters?"
[9]
...
"Well, try typing in BP.L "
...
"Ah. Well, I'd better let you get on then."
It appeared that Nick's colleague's friend at the other bank had not checked first to ensure that the system could handle the new format pages, and had found out – the very hard way – that no, it could not. The system crashed completely.
And Nick's colleague hadn't thought to warn him.
Because Nick hadn't thought to warn him.
Nick never did find out what the consequences were – if any – of the unscheduled downtime. He wasn't game to call and find out.
But to the extent that he bore any responsibility as the instigator of the ill-fated attempt to load a page, he escaped completely scot-free. As Nick put it: "It was Reuters who crashed their system – not us. All we did was load the gun."
You can argue that one in the comments.
If you've ever set in motion a chain of events that led to technological chaos, then escaped even a hint of blame, this is where you can tell all, anonymously, and finally have your evil genius appreciated. [10]Send an email to Who, Me? and we might share it on some future Monday.
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZkHkwKjzz7xYKXEm3JnTJwAAABg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZkHkwKjzz7xYKXEm3JnTJwAAABg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZkHkwKjzz7xYKXEm3JnTJwAAABg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZkHkwKjzz7xYKXEm3JnTJwAAABg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/05/06/who_me/
[6] https://www.theregister.com/2024/04/29/who_me/
[7] https://www.theregister.com/2024/04/22/who_me/
[8] https://www.theregister.com/2024/04/15/who_me/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZkHkwKjzz7xYKXEm3JnTJwAAABg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Extended Data Format Crashes System
That is how computing has improved over time ; by correcting one fuckup at a time.
And, mind you, in those days compilers were written by actual engineers . . .
Re: Extended Data Format Crashes System
…and small furry creatures from Alpha Centauri were real small furry creatures from Alpha Centauri!
So Nick's friend wasn't banking on the system crashing
It's good that Nick wasn't held to account
Do we know for certain that happened? We have an interest in finding out...
Nope, but I think he probably made a rather large deposit when it did!
I hope Nick got a good share
I'm sure it wont have muchof an impact...
I was at a customer site helping them diagnose a problem ( in production - test was fine). After a day of struggling I had a feeling for the problem, but they had lost the source of the application, so the only way of seeing what was really going on was to take a trace.
We put in a change request to collect trace for 5 seconds, and this was approved.
Next morning when it was "quiet" we turned the trace on and immediately turned trace off
A couple of minutes later the phone rang... "we've just had a performance blip... the response time of transactions went from 10 ms to 20 ms for about 5 seconds - could this be you?"
It was us - I had not realised that a quiet period meant a transaction rate of over 1,000 transactions a second.
We found the problem, and when we discussed it with management they said they had a cliff. If the response time was under 10 milliseconds great - if it went over 10 seconds every thing slowed down catastrophically. This meant they would miss their service level agreements, and then got fined >lots< of money.
We were lucky - there was no big impact, but I learned a lesson
"Users would type into an X-terminal"
"Users would type into an X-terminal"
Given the 64x17 and 80x25 column×lines I suspect X- doesn't mean X11 but some current loop, rs232 or ibm block mode terminal.
The first X Terminal I ever encountered ran off the rs232 connection to a [1]Pyramid mini presumably using something like slip but could also be a plain text DEC VT- compatible terminal with a lot more than 80×25 col×lines.
[1] https://en.m.wikipedia.org/wiki/Pyramid_Technology
Re: "Users would type into an X-terminal"
Maybe xterm rather than X-terminal?
The City: A chap knows a chap...
Reminded me of a memorable Yes Prime Minister scene
The script: [1]https://m.imdb.com/title/tt0751828/characters/nm0894710
[1] https://m.imdb.com/title/tt0751828/characters/nm0894710
Extended Data Format Crashes System
I have to wonder what sort of hardware and OS these banks had, as I ran into a similar thing back in the late 1970s.
My college roomie had me up to his parents' house for the weekend, and on Saturday, he took me to his part-time workplace. The multi-user system used AMD 2900-series bit-slice components to implement a 16-bit machine with no memory protection, each video terminal had its memory physically resident in the system rack, mapped into the 64KB address space, and the OS had no way of protecting users' memory spaces, given the hardware limitations.
I wrote my BASIC program, invoked the compiler, and the system crashed.
He ran into the room and over to my terminal. "What did you do?!!"
"I ran the compiler on my program, just as you showed me," I replied, pointing my finger at the last visible line on my terminal.
He looked at it, then went off and restarted the computer, a 3-minute process.
I tried again, with the same result. He restarted the system again.
I guessed there might be something wierd in my program, so I started a printout. The printer was a wide-carriage Diablo letter-quality beast, loud and slowish.
My roomie and I watched the printout, and I commented on how intelligently the printer handled tab characters.
His forehead furrowed, then he yelled, "You're using wide lines!"
"Yes, well, wide-ish, but wider than what? "
"Wider than eighty columns. If you use wide lines without using the '/W' option on your compiler command line, the compiler overwrites part of the operating system."