Vendor's secret 'fix' made critical app unusable during business hours
- Reference: 1764919815
- News link: https://www.theregister.co.uk/2025/12/05/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Raoul" who told us about his former life as a commercial Linux support consultant.
"We had a customer call up to ask us to provide a health check on their large server running a critical business application," he told On Call. Raoul was given a week to do the job.
[1]
The customer was a medical facility and the software it ran was a complex Java web app that handled jobs including patient scheduling, booking management, and even payments. It ran in VMs, on a Postgres database, an external storage array, and across three servers – one of which was a warm spare for the database.
[2]
[3]
The app wasn't well.
"During peak load in the morning, the system would grind to a halt and become unresponsive for anything up to half an hour," Raoul explained. Medical specialists, admin staff, and patients were all left waiting for the app to perform, a decidedly unhealthy situation.
[4]
When Raoul arrived to diagnose the problem, he found the on-site techies bickering.
"The virtualization people blamed the storage system, the storage people blamed the application, and the application developers blamed the OS," he said. The customer's relationship with the application's vendor was also toxic as somebody from the medical facility had made unkind remarks on the vendor's support forums. Threats of legal action followed.
The next day, Raoul arrived in time to see the application grind to a halt at around 10:00 AM. He checked the application server – which was fine – but did notice the database server was very busy.
[5]
Raoul kept his head down and on his third day on the job waited until the system locked up again.
"I headed straight to the database server and saw somebody was running a lengthy update task that locked down table rows, meaning all other transactions had to wait," he wrote.
[6]Cabling survived dungeons and fish factories, until a lazy user took the network down
[7]Linux admin hated downtime so much he schlepped a live UPS during office move
[8]Developer battled to write his own documentation, but lost the boss fight
[9]Help desk boss fell for 'Internet Cleaning Day' prank – then swore he got the joke
Raoul dug a little deeper and learned that the vendor of the application had found a bug in their wares, and was patching database errors on the live system, during business hours, without telling their customer.
"I collected the evidence, wrote up my report, and took it to the management," he told On Call. "The app's developers confessed that they had known of the issue for months, had a fix 'almost ready to go,' and everything would be OK."
Raoul soon learned that the situation was far from OK because during his last two days on the job, he decided to do a health check on the Postgres database.
"The production database stored medical data, personal information, and handled payments had no access controls," he told On Call. "It was configured 'ALL ALL ALL', so any user on any system could access any database as any user."
That revelation made Raoul feel ill.
"I nearly fell off my chair, reported that to the management, and they told me the developers said that was the required config, and were not concerned," he wrote.
"I went home with a mental note to never use that vendor," he concluded.
Has anyone hidden the cause of a technical problem from you? Don't hide your story from On Call! [10]Click here to send email to On Call so we can share your story on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aTK7RznNocGx8l5NdhdnGQAAANA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aTK7RznNocGx8l5NdhdnGQAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aTK7RznNocGx8l5NdhdnGQAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aTK7RznNocGx8l5NdhdnGQAAANA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aTK7RznNocGx8l5NdhdnGQAAANA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/11/28/on_call/
[7] https://www.theregister.com/2025/11/21/on_call/
[8] https://www.theregister.com/2025/11/14/on_call/
[9] https://www.theregister.com/2025/11/07/on_call/
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
Re: Lost for words
I've seen numerous apps over the years where the "fix" for a permission denied error was "chmod -R 777 /app/directory". It doesn't overly surprise me, but it's awful, awful practice.
Re: Lost for words
Besides awful practice, it is a sign that they have no clue what they are doing and they should be avoided to touch your systems at all cost.
Only /tmp [and /var/tmp] require[s] 1777 (== 777 with the T-bit set).
Re: Lost for words
Only /tmp [and /var/tmp] require[s] 1777 (== 777 with the T-bit set).
Do applications ever store confidential information in temporary files? I wonder if anyone's ever written something to scan such files?
Re: Lost for words
Many years ago I was doing a migration when abroad. Sites had control of their kit, but there were some global standards for workstations (they ignored the OS upgrade to W8). Servers all had to be in English.
Doing the migration which for the servers involved a lot fo scripting to reconfigure from the sites old standard / settings to our global ones. A well tried and tested script
The workstations which were now in W8 and in a non English language were a bit confusing at first, but commands are commands and established the scripts that ran on the clients to move domains, sort users, remap etc were all good too.
This all started on a Friday night - was a bit slow, but nothing too unusual
Sunday night comes and at this stage we are pissed off for a couple of reasons and then one of the local team who is checking all the users devices and configs says "I cannot get this users home drive to come up". I have a look and sure enough, no mapping to their homedrive. i check the server, its there, all set to go.
I spend some time and am a bit bemused as I cannot see anything that would l cause this and is working for my account (admin) then I ask "Is anyone else affected?". His response "Oh yes, everyone" I am now opened mouthed and ask "When did you notice this?" and their response was "Since the first machines we checked on Friday night..."
I had not choice at that stage to grant them all local admin on this box to get access to their data - just because they had to work on the Monday.
Took a few days back and investigations to find out that the local IT had set some setting on the server (I cannot recall what, but it was obscure) that would cause this. Reset the key and then removed local admin and it was all good.
The easy win would to have just left it - as it is a local box so not as critical as they being admins for that city/country, but it pressed that it needed to be dealt with and could not be left as it was
Re: Lost for words
A totally, gobsmackingly bad "design decision" (phrase used without prejudice) by the application vendors, and monumentally stupid decision by management to accept the vendor's "explanation".
The vendor had better not try this on any BOFH, or they might have an ... accident in the elevator, or a [1]database normalisation warning (the latter seems more appropriate)
[1] https://www.theregister.com/2017/11/24/bofh_2017_episode_16/
Vendors...
the developers said that was the required config
We recently had an issue with a 3rd party integration vendor. They were involved in us uploading open receivables data from our SQL based finance system but their default requirement was a user with full access to the entire database - that wasn't going to happen! They then stated as an absolute fact that access had to be to the tables, not views, again not going to happen as there are multiple companies running on the same database and they were only involved with one of those.
Eventually they corrected themselves and said views would be fine but neither they nor the primary vendor could say what data they actually needed so we killed the project. It was frustrating as I could have sorted the whole thing out in a few hours (create views with the correct data subsets, provide a user with restricted access to those views, install the integration software) which would have taken far less time than the back-and-forth discussions.
Re: Vendors...
This makes me think of the far-too-many vendors whose installation instructions contain a note to the effect "if it doesn't work, turn off your security software and try again". They never seem to learn.
Re: Vendors...
We had an integration from one product to an MS one.
When trying to get the config, the documentaiton says "admin". the customer of course was "no bloody way".
A call was arranged with the product vendor who went through why and showed their refrence - some MS Docs.
Next call was the same people and added MS. They still inisted it needed admin
The customer engineer mumbled and grumbled and a few days later came back with the "right" permissions for us to use and that worked.
Documentation now updated.
There is/was a major school MIS provider who provide a feature within their software which, when used, executes the given command as a plain SQL statement against the underlying main school database as a full administrative user.
I discovered this one day at random by being sat in someone's office while they were troubleshooting a minor pupil-data error in the database and they were instructed by the software support line to enter this menu, and I heard "DELETE * FROM" as it was read over the phone to them and they repeated it back as they typed.
I was horrified that their support staff were instructing otherwise ordinary users to type plain SQL direct into a dialog that was then EXECUTING... including, I found out when I dived over the table to take the phone from, things like dropping tables, changing schemas and removing rows.
And there were no logs of this, no controls on it, the support staff reading out the commands were oblivious to their actual danger (just reading off a script for a given problem), they were then getting users who knew nothing of SQL syntax to type this in over the phone, and there was no attempt to ensure backups had been taken (this was on-prem!) and no attempt to even enclose the statements in a transaction.
The next year we went cloud because... they can sort that out when it all goes horribly wrong, I'm not taking responsibility for it!
Medical systems are a nightmare
I put in an electronic patient record for a hospital group about a decade ago. Vendor selection was "best of a bad lot" but the real problem was getting the decision makers to understand the issues.
"Why can't we use the same password for everyone, we always did before" (for a prescription system with hardwired terminals for 5 users kept in a locked room). And medical devices are worse.
I did eventually come to understand their logic. If we spend the money on increasing security it may prevent a breach sometime in the future. If we spend the money on extra clinical staff hours it will prevent a death in the next few days.
Re: Medical systems are a nightmare
I had an appointment with an anaesthetist a few years ago, whilst talking to him I noticed there was a username and (guessable) password for a computer system on the wall.
Re: Medical systems are a nightmare
I once went on a visit to a customer office to diagnose a problem in the wild. To fill the time I provided onsite support and training for the poor suckers lucky punters using our desktop software. It was the first time saw someone lift and invert their keyboard after being asked. "What is your password?".
Re: Medical systems are a nightmare
My company tried to get us to use a company wide password vault.
One on the early items in the training was how a wonderful feature in the vault allowed you to share passwords.
At which point anyone with sense dropped off the call.
Even the HR droid monitoring the call noticed this drop in attendees and everything has gone quiet on the compulsory use of the tool.
Re: Medical systems are a nightmare
If we spend the money on increasing security it may prevent a breach sometime in the future. If we spend the money on extra clinical staff hours it will prevent a death in the next few days.
1. If a business or organization is so underfunded that that choice -- security vs near-immediate patient death -- is truly binary, they should not be in business/operation.
2. A security breach of a medical organization's computers can lead to many deaths.
Arghhh
Yes, I've seen similar, recently, and I'm still under NDA.
Similar Story with FTP
About 20 years ago, my company outsourced marketing mailshots to a third party and we had to regularly upload the file from us to them for them to process. My colleagues thought nothing of it - they were Windows-based and just did what the procedures said - but as I was Solaris-orientated, I decided to cd to the parent directory. Just for a bit of a laugh, because, well, it won't work, will it?
Oh... I didn't expect to be able to do that. And look at all those company names, some big hitters there. I wonder...
And yes, I was able to access a couple of directories, list the files to view the permissions, but I didn't do anything else except show my boss, and I especially didn't view any files.
In the UK...
...the vendor would be likened to Fujitsu. Because of the Post Office scandal.
Lost for words
The production database stored medical data, personal information, and handled payments had no access controls," he told On Call. "It was configured 'ALL ALL ALL', so any user on any system could access any database as any user.
I had to stop and reflect on that one for a second.
But seeing how the vendor is behaving, it wouldn't surprise me if they suddenly scrambled and implemented access control, in the process of that they break the app, and silently patch their errors without having anyone know.