Workload written by student made millions, ran on unsupported hardware, with zero maintenance
- Reference: 1697182506
- News link: https://www.theregister.co.uk/2023/10/13/on_call/
- Source link:
This week, meet a reader we will Regomize as "Rik" who shared the story of how, during his undergraduate computing studies, he scored a student placement in the IT support team of a large UK company that traded 24/7 on various financial markets.
This was the sort of environment in which wobbly tech could cost millions in a moment. So Rik was a little surprised when, on top of the usual student chores of fetching coffee and doing drudgework, he was also asked to look at "the platform hosting a real-time graph used to inform the shift traders." The graph wasn't critical, but sometimes revealed market intelligence that proved extremely profitable and impactful.
[1]
And that platform had problems.
[2]
[3]
"I was told that the hardware/infrastructure team had concerns about supportability and there had been no changes for over two decades," Rik recalled.
His next stop was therefore to visit the infrastructure and hardware team, which informed him the graphing code worked across a single Sun SPARCstation 2 and a PC.
[4]
Folks from the engineering team were able to find the relevant Java and C code, and – after examining it and chatting to colleagues – Rik learned its purpose was to monitor a single real-world variable and graph it. He discovered that a previous student on placement wrote a driver for a General Purpose Interface Bus (GPIB) for the sensor that sucked in the variable, and a Java http servlet to read it and share it.
That code was a proof of concept that ran on the student's desktop. Nonetheless, traders had seen this work, profited from it, and begun to rely on it.
"The idea was it would be retired or productized at some point," Rik learned.
[5]Cheapest, oldest, slowest part fixed very modern Mac
[6]Police ignored the laws of datacenter climate control
[7]Techie labelled 'disgusting filth merchant' by disgusting hypocrite
[8]Techie wasn't being paid, until he taught HR a lesson
Which didn't happen. But the hardware it ran on was moved into a proper datacenter.
Because this was not an official workload, it wasn't powered from the datacenter's main power source – its power cables snaked under the floor and emerged next to a regular wall socket.
[9]
The same sockets used by cleaners when they vacuumed the facility.
Nobody knew why the app failed at the same time every week. They just lived with it until Rik made inquiries about the situation.
As his inquiries progressed, he was asked if code for the app existed, and if it was worth persisting with, or could be replaced with an off-the-shelf system.
Rik surveyed the market, and found solutions that could do the job – but were massively overengineered and overpriced.
He ended up modernizing and re-platforming the code – which wasn't vastly difficult because C and Java hadn't changed that much in 20 years. Adding USB support to replace the GPIB was a challenge, as was readying it to run in a VM with proper failover.
But Rik was able to do the job, and the app was finally productized!
And it only took two students, working twenty years apart, to finish the job!
Have you ever been asked to fix unofficial apps, written one yourself, or delivered mission-critical services while still a student? If so, [10]click here to send On Call an email and we'll consider your story for a future instalment.
Don't be shy – we always need more yarns to consider. And remember: you'll always be anonymous. No story too silly, but we do try to avoid smut. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZSkVQWXCU3dcIjFXxROKDwAAANc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZSkVQWXCU3dcIjFXxROKDwAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZSkVQWXCU3dcIjFXxROKDwAAANc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZSkVQWXCU3dcIjFXxROKDwAAANc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2023/05/19/on_call/
[6] https://www.theregister.com/2023/10/06/on_call/
[7] https://www.theregister.com/2023/09/15/on_call/
[8] https://www.theregister.com/2023/06/23/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZSkVQWXCU3dcIjFXxROKDwAAANc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: I'm curious...
It was a Volt-meter monitoring the mains voltage. You can detect the spikes and conclude that traders are on a frenzy.
Re: I'm curious...
"Exactly what crucial financial parameter needs to be read over GPIB?"
Dunno. Probably something coming off the ticker. Could it be something as simple as the DOW, perhaps?
"That's more about reading volts and amps from electronic test gear isn't it?"
The GPIB, originally known as the HP-IB (Hewlett Packard Interface Bus), was invented to tie automated test equipment together. Later, it was bastardized into a peripheral bus for small computer systems. I've seen it used on everything from Mainframes to the Commodore PET.
I'd be really curious as to the connection between the signal of a single physical sensor, and financial trading.
I'm going to hazard a guess, but I would imagine the sensor is a microphone and it measured the noise level of the stock exchange. If it started getting loud with shouts, the sensor would flash "SELL SELL SELL" which the traders would do.
Lucky it was switched off when (and by) the cleaner was running the hoover then or it'd have caused financial meltdown in the US each evening.
Proof Of Concept Business
Many years ago, I did some documentation for a program used by a financial services company. One of their employees, long since gone, had got a copy of Borland C++ Builder and cobbled together a 'proof of concept' which then turned into a £13M per year business line. He had a lot of domain knowledge but no software skills.
There are a lot of *really* nasty things you can do in C++, if you know what you are doing. Thankfully, his only major sin was a function which was 4000 lines long. Not particularly complicated, just long. It took me three weeks to work out what was going on & then document it.
In the end, we produced documentation which no one read but ticked the box for the regulator's audit.
Re: Proof Of Concept Business
his only major sin was a function which was 4000 lines long.
I remember looking at C code from a supplier and finding a 1500-line switch{} statement.
Scary, but I have to admit that the code was really rock solid, we made a lot of money from it over the years.
Re: Proof Of Concept Business
There are a lot of *really* nasty things you can do in C++, if you know what you are doing.
More things, much nastier, if you only "think" you know what you are doing.
A friend discovered that a program he'd written when just starting out was still in use decades later. Helping to run the National Grid.
Access control system - unusual manufacturer (South Africa based company), heavy investment on it on site.
Had the need for a firelist. By this time, every member of staff is using the system to tag-in, tag-out, and it's being (somewhat) misused on occasion to prove time and attendance.
We look at the official fire module. It costs a fortune, takes forever to run, has to be manually triggered, and produces print output. Useless to us, especially in a fire.
So I realise that underlying it is an antique Firebird database (which for those who don't know is a bit like SQLite in that it just logs to ordinary filesystem files that you can query with just ordinary file locking).
So I write some SQL and I write some monitor scripts and it basically watches out for the fire alerts (which do trigger a table on the system), builds the list of everyone on-site, and then I sent it to a thermal receipt printer that churns out the whole list in seconds.
It passes initial testing, solves our problem (which wasn't a CRITICAL problem, but it's certainly very useful to know that Jim actually tagged out of the site and so is unlikely to be burning to death in the building).
As with everything - feature creep sets in.
Within a year, the script is running 24/7, the printout separates people by area and creates perforations on the receipt so that each area can be torn off and given to the person with responsibility for that area to check they have everyone, the output includes a "last seen" time for when people tagged in once in the morning but haven't been seen since and cause confusion over whether they actually are in today or not, it stores the logs plus emails the output to a distribution list, lockdown functionality is added, and there's a second identical redundant system set up at the other end of the site to facilitate quick access to it from there, as well as a backup if one fails. The schedule for replacing the paper is incorporated into consumable replacements, etc. etc. And with some tightening, the first people know of a fire alert is actually the printer being half-way through a receipt printout because it actually outperforms the alerts from the system itself (so audio alarms happen AFTER printing starts!).
The system is now so integrated in processes that it basically is the fire rollcall system, and the suppliers who fit our access control try to buy it off me - because their customers are all asking for "this thing I heard about that this other customer of yours has in place".
Then I leave. All dues to the guy who took over from me, he can keep it running. But he's told them a thousand times that when it stops working, it's dead, simple as that. He has no interest in maintaining or supporting it (and I can quite understand why!). They go back to the company who tell them the price of the official firelist module - still got all the same problems (do you want to wait for a laser printer to warm up to print 20+ pages of A4 while the building is burning? Or would you rather grab a till-receipt from a machine that actually BEATS THE FIRE ALARM in churning it out), and it has tripled in price.
Also, they now need to "convert to the web-based version" which means replacing half the controllers and losing all such access to make your own reports (and quite a few other tweaks we used to do as well). So a firelist from the system is now basically impossible, unless you pay for a module that emails it in a single standard format (a big list of names in an A4 PDF) on its own schedule (cloud, remember) and no customisation whatsoever.
To my knowledge, it's still churning along and a vital part of the system. It was about 2-3 days of collective coding, plus two cheap receipt printers off Amazon. Oh, and there are plans to move it to a Raspberry Pi to keep the old desktops it ran on going. Hope that there are no architecture incompatibilities in my code!
Kudos points just for sending the fire list to a thermal printer :)
Let me guess. There was an old printer just sitting there waiting to be scrapped.
If the printer is near the fire, it's game over anyway.
But I literally couldn't think of any quicker method to get something onto paper... anyone?
Agree a receipt printer is probably the best solution, can be run off batteries and being small don’t take up much desktop/wall space and whilst they can have multiple uses are unlikely to be used for other purposes, so fire list gets queue blocked because that rarely used printer in reception is currently printing someone’s Magnus opus report.
The system is now so integrated in processes that it basically is the fire rollcall system, and the suppliers who fit our access control try to buy it off me - because their customers are all asking for "this thing I heard about that this other customer of yours has in place".
If I was your replacement and your company I'd have a nice chat with that supplier about getting a contract.They can have what you developed (for a nice, ongoing fee) and if they do a good job supporting and developing it further (meeting and/or exceeding your usecase) the use fee for the supplier will be waved and your company would get to use the system, including support from the supplier.
Could you add the "On-Call" label on the home page?
At my former job we had a nagios monitoring system and one guy had written some perl script in 2003 so it could send an SMS alert to the site manger's phone in the event of overheating, as the A/C was flaky and underpowered. Later a 2nd A/C was fitted and all good as either could cope with the heat on its own, except maybe peak summer heat on the old one alone. All quiet on the alert front for many years.
That facility was shut down in 2019 and me and the site manger resurrected the facility at another site a year or two later rather than scrapping it. Eventually fired up the old nagios server and got maybe half of the equipment running to check stuff out. Then a hot day and he got texted!
So script still worked, the bulk SMS company had not buggered their API in 20 years, and somehow, somewhere, there was still enough funds in an account to pay for it!
Later the account was found and updated, and the perl script also bug-fixed to limit any very long messages (as API rejected them) but good to see a job that lasted!
I think my flaky server room temperature monitor failed a while ago - I really should check it and get it working again!
It involved an old laptop with a cheap USB temperature probe, exporting to a file which could then be read and checked by a scheduled job - not exactly elegant but it's saved us a few times and only cost £10.
Makes you proud
A long time ago I was visiting an old client. Over 14 years everything had changed in their office except my software. I was able to say "Apart from the people -- the oldest thing in this room is my program written 14 years ago."
Re: Makes you proud
I'm pretty sure that, if I were to check code I wrote 14 years ago, I'd smack myself in the forehead and mutter "what the blazes were you thinking ?".
The staying power of powerpoint
I made a powerpoint schematic of our system in the first weeks I was working here as I kept running into having to communicate things with clients and not having an easy diagram/visual. Over 10 years later variations of that single slide are still used everywhere both in my own company and that of the client that I was communicating with. Nobody has bothered coming up with something more professional looking because invariably they were less easily readable.
Re: The staying power of powerpoint
I'm writing a documentation wiki and I abandoned all the previous documentation except for reference.
The fibre maps are literal scans of scrappy pencil scribblings over the top of an ancient map (which was made for another purpose).
I took the best vector map I could find, tore it apart with Inkscape, rebuilt it (with the doors where they REALLY are, and things like that), named everything properly, grouped it into individual buildings and produced a bunch of SVG maps - one for each floor, one for each building, whole-site overviews, etc.)
Then I took the whole-site overviews, made it a fixed layer at the back of an SVG with its opacity turned down, and started to overlay CCTV, access control, networking, etc. over the top, one file per system. Every time I find another cupboard that the map says doesn't exist, or another doorway that's just entirely wrong and was bricked up decades ago, I redo the building map, copy it into the overview map, then update the overview map layers in Inkscape on all the others as and when I need to.
Already I've had marketing and the site departments ask for copies of it, because it's the only vaguely-accurate map they have seen. Hell, it's being used to show parking on the visitor sign-in system.
I am now an expert at manipulating SVG with Inkscape, putting them into the documentation, and solving problems with the original mapping and SVG file (P.S. if you want to publish an SVG on a website... remove all clippaths from the XML... you can just delete the tags. Then cleanup the file by adjusting the nodes of lines rather than using clippaths... you can use Inkscape CLIPPING just fine, but purge all clippaths from the SVG... you'll thank me later when it actually renders properly in anything using rSVG, including things like Chrome and most WIki and image-library software).
So much so that I diagrammed out my home solar install in the same fashion, and made something so good that I'm currently looking for a frame to put it in.
Write a real-time market intelligence graph
> Rik .. was also asked to look at "the platform hosting a real-time graph used to inform the shift traders." The graph wasn't critical, but sometimes revealed market intelligence that proved extremely profitable and impactful.
Doesn't surprise me, IT is still considered just as important as the catering /s
As a student?
If you are reading this, you're probably using the hack that I put together in 4.1BSD (now called 4.1aBSD) for part of the TCP/IP stack, to be included in 4.2BSD[0]. It was supposed to be one of those "Just get us through the demo, dammit!" hacks. I got 'er done in a couple days over Christmas/NewYears break in 1981. Virtually every version of TCP/IP since has used it. Not too bad for a quick hack.
[0] Just to cut the usual pack of idiots putting words into my mouth off at the socks, no, I didn't write the whole stack. That's why I said "part of". It is only about 120 lines of C in total.
More personal.
The current code in the one greenhouse that hasn't been updated to AtMega 328 yet is still running software that I started writing about 50 years ago, and finished about 40 years ago[0]. I think I got my money's worth out of the original Z80 & S-100 bus ... that system is now being emulated on a headless Slackware laptop. The code is pretty ugly, especially the very early stuff. Looks like it was written by a kid ... but it works.
[0] Yes. Software that is finished. Really. Running untouched, with no troubles, for around 40 years. It can be done.
I'm curious...
Exactly what crucial financial parameter needs to be read over GPIB? That's more about reading volts and amps from electronic test gear isn't it?