COBOL-coding volunteers sought as creaking mainframes slow New Jersey's coronavirus response
- Reference: 1586073127
- News link: https://www.theregister.co.uk/2020/04/05/new_jersey_seeks_cobol_volunteers/
- Source link:
COBOL - common business-oriented language - was first introduced in the early 1960s and achieved the then-important trick of offering programmers a language that could work across multiple manufacturers' proprietary computers.
It became a staple of mainframes and, in the last couple of decades, a byword for crusty and dusty legacy code. A fine history of the language can be found [1]here .
In his daily press briefing on April 4th, governor Phil Murphy said: “In our list of volunteers not only do we need health care workers but given the legacy systems we should add a page for cobalt [sic] computer skills, because that's what we're dealing with in these legacies.”
Governor Murphy said his staff is “doing a heck of a job but literally we have systems that are 40-plus years old and there'll be lots of post mortems and one of them on our list will be how the heck did we get here when we literally needed cobalt programmers.”
Here’s the clip of the governor calling for COBOL programmers.
[2]Youtube Video
It appears that New Jersey needs COBOL coders because its benefits system has choked on a surge of requests for unemployment payments.
In the video above, from around the 46:50 mark, commissioner of the New Jersey Department of Labor and Workforce Robert Asaro-Angelo explained that his agency has experienced a 1600 percent increase in its usual volume of requests for assistance.
“There's nothing I want more than to put your hard-earned benefits into your family budget sooner we've made no secret about the inflexibility of our legacy technology and our desperate desire to receive and act on more of your phone calls. We hear your frustration and we are with you we're currently working to bring into our systems more of your calls and emails. This is our number one priority.”
That inflexibility has been noted by local media, which [3]report that half of claims for unemployment benefits haven’t been processed immediately.
At Governor Murphy’s [4]April 2nd briefing he said: “This morning the Department of Labor reported that over the past week more than 206,000 new claims for unemployment were filed, meaning that in just the past two weeks alone more than 362,000 residents have filed for unemployment. “
“And we are also very cognizant that there are delays and backups in the system, and we urge everyone to please have patience and that your claim will be taken care of and you will not lose one penny of your benefits.”
As Murphy’s April 4th remarks suggests, technical debt has come back to bite the State’s Department of Labor and Workforce at the least-convenient possible moment. ®
[1] https://americanhistory.si.edu/cobol/introduction
[2] https://youtu.be/rEDXdMlM-Rk?t=3268
[3] https://www.roi-nj.com/2020/04/04/politics/cant-get-your-claim-through-and-what-about-that-extra-600-labor-commissioner-offers-answers/
[4] https://nj.gov/governor/news/news/562020/approved/20200402d.shtml
COBOL coders came out of retirement to help fix the y2k bug.
That (the work done) was more than twenty years ago. They came out of retirement to do it twenty years ago. A lot of them probably just aren't around any more.
Technical debt always comes due, but it can be paid in installments. Maybe "if it ain't broke don't fix it" isn't such a great idea after all.
Most of us who worked on COBOL for Y2K weren't retired quite yet. Some of us still aren't officially retired.
There is nothing really wrong with the COBOL code we are talking about here. It's been cranking away for years, happily putting in work. All it needs is shifting over to faster hardware with better I/O. Which is available off-the-shelf, for a price. And no, it's not "cloud" crap. For example, I know for a fact that some of IBM's current gear will run COBOL that I wrote in the early 1970s unaltered.
TPTB in New Jersey know this. They just want to avoid spending money. Especially money that'll be going to keeping the legacy code around for another 50 years ... because let's face it, that's exactly what'll happen if it can be made to handle the current crisis.
What is wrong is the ridiculousness of keep using software that's so old it was invented when computers didn't have enough memory to hold the full date of the year and keep replacing the hardware while keeping using the same software.
Practicality the whole servers have been replaced by now, if anything remains of the old servers at all, and yet they keep using COBOL for them.
Hard disks start to fail at about ten years if not earlier, wires and fans don't last forever, there are also problems like the bios ending being so old it just dies even if you keep changing the battery
Buying cheap ends becoming expensive and replacing the whole servers with a version of Linux and up to date (By Debian stable standards) software would end saving the state a huge amount of money in the long run.
Yes, there are newer versions of COBOL but this is the COBOL that was used in the eighties,;even if you want to keep using COBOL you should at least update to the modern version so there will be people left alive that can use it when the Year 2038 problem strikes.
@Blackjack
"there are also problems like the bios ending being so old it just dies even if you keep changing the battery"
I'll just leave that bit there.
Your post makes me sad.
Despair
"What is wrong is the ridiculousness of keep using software that's so old it was invented when computers didn't have enough memory to hold the full date of the year and keep replacing the hardware while keeping using the same software.
Practicality the whole servers have been replaced by now, if anything remains of the old servers at all, and yet they keep using COBOL for them.
Hard disks start to fail at about ten years if not earlier, wires and fans don't last forever, there are also problems like the bios ending being so old it just dies even if you keep changing the battery
Buying cheap ends becoming expensive and replacing the whole servers with a version of Linux and up to date (By Debian stable standards) software would end saving the state a huge amount of money in the long run.
Yes, there are newer versions of COBOL but this is the COBOL that was used in the eighties,;even if you want to keep using COBOL you should at least update to the modern version so there will be people left alive that can use it when the Year 2038 problem strikes."
There are some absurd comments posted hereabouts, but this one is beyond ridiculous. Have you no clue whatsoever about managing computer systems? Even the Linux ones you seem to think are the answers to everyone's prayers?
Yes, hard disks fail. You apparently don't know that it is possible to move data from old disks to new ones, and even if the old ones fail before you do that, you restore from backups. Maybe you've heard of RAID? If not, maybe should learn about it. You'll no doubt be shocked to know that shops which run large COBOL systems actually perform regular scheduled backups in case of worst case disk system failure. And know how to restore from them.
Most systems which run COBOL at scale don't have BIOSes, but even if they did, BIOSes don't fail due to batteries. As for wires wearing out ... maybe the cheap eBay knock-offs you use for your 'phone, but not those used on business systems.
What the version of COBOL originally used has to do with it is entirely beyond me. COBOL programs compiled 40 years ago on IBM mainframes will still run today. Assuming the source code hasn't been lost (this is actually a genuine problem) it can be recompiled with the current compiler, usually with no changes.
@jake
"I know for a fact that some of IBM's current gear will run COBOL that I wrote in the early 1970s unaltered."
Indeed, the z series is backwards compatible with assembler too (I'm still active).
How did they end up in their legacys situation with their cobalts and their assemblers? Outsourcing, downsizing of anything that works, refusal to replace retirements, ignorant people in charge who fail to realise the importance of the IT systems they rely on; they're called mission critical for a reason.
Cobol/assembler on mainframe (whether IMS or CICs or whatever (I program on whatever)) technically has no issues. It's a manpower issue. There just aren't enough people employed who know the business systems well enough to touch them, or even to understand them so they can be adequately specified.
The big g will no doubt fire up its Cornerstone acquisition and convert to java. Imagine, if you will, a sprawling java monolith that can only be understood by looking at the original cobol and that still operates in the same way as the original... once you hit a hardware choke point (even in the cloud), you're f'd...
No so much COBOL as the tools
I would think that the problem isn't so much the language as the environment. Anyone who's familiar with modern procedural langauges shouldn't have much of a problem picking up COBOL. More of an issue will be finding people who are trained on Windows or Unix/Linux and getting them used to working in 1970s/1980s terminal-based environments with line editors, batch processing, obscure toolsets and all on systems with limited storage. Can you imagine a millenial Python/Linux programmer having to get to grips with IBM JCL and overlays configured by link-editors, all in maybe 1MB RAM and possibly with intermediate storage on serial tape? On a 3270 terminal?
Re: No so much COBOL as the tools
I have to put up with millennial programmers who can't even write a simple Windows batch file, and for them RTFM means going looking for a YouTube video.
if they find enough people and start the project now
They might have the problem fixed (whatever it is) by the time a vaccine comes out next year.
This virus seems to have caught a lot of people with their pants down, except probably Germany.
Is that actually what they need?
It sounds like they need capacity and they're unlikely to be able to get that by rewriting code at this point.
Legacy COBOL programs tend to be tied to legacy databases and legacy hardware so it's unlikely blue-sky thinking will involve Azure any more than Cobalt.
They probably need first to consider solutions like replicating the application on any spare hardware they can find and dividing the workload between them in such a way that the data can be consistently reconstructed later (such as by first letter of surname).
Re: Is that actually what they need?
IBM has lots of lovely Mainframe capability in its cloud. If you want to run on the latest hardware and can't stomach the upfront cost (although a lot less than two decades ago) then that's the place I'd turn to in a crisis.
It sounds to me that they need to not only increase capacity but to make some coding changes. The latter is bad news when done in a crisis. 'Act in haste, repent later' sort of thing.
Just think what strains the UK's benefit system is under... PErhas NJ is preferable. Remember in NJ you can't pump your own gas. (or that was the situation the last time I visited)
Re: Is that actually what they need?
It sounds to me that they need to not only increase capacity but to make some coding changes
I just visited the State's [1]job vacancies page to see if there were any clues as to their environment, to be greeted with the announcements:
The application is not compatible with FireFox. Please use one of these browsers; Internet Explorer 11, Edge and Chrome
and
This site does not support browsing from a Mobile or Tablet device.
Perhaps they need to spend less time on actively preventing things working - they might then have time to do the important stuff.
[1] https://www.state.nj.us/csc/seekers/jobs/announcements/
Technical debt bites – but beancounters never get it in time.
We've been saying an upgrade is necessary for literally decades.
So now they want us to work for free? What kind of suckers do they think we are?
If it was an accident, I'd do it for free ... but let's face it, long-term gross incompetence on the part of elected officials is hardly an accident. There is more than enough money in the state coffers to provide a better retirement for a couple dozen retired COBOL programmers who missed out on the dot com boom due to age.
Volunteers? Seriously?
If they don't have money to pay people, then it's obviously not a big problem.
This is a textbook example of "Lack of planning on your part does not constitute an emergency on my part"
You can't make this sh1t up
The problem will most likely be an array in working storage, which are defined like
05 claim OCCURS 1000 TIMES.
Fixed array sizes are not as daft as they seem, but were a mechanism to enable the scheduler to avoid over committing resources while scheduling jobs. For S/370 it was also used for 20-bit, 31-bit (XA), 64-bit (ESA/z) compilers to fit into smaller memory regions
As with any engineering problem, it is not the one-line code change that takes the effort, but the elaboration of dependancies and testing that takes the effort. It might not be feasible withing the constraint of compiler, os, platform. That one-line change can grow into a quarterly upgrade.
The "solution" is probably to run the batch multiple times, or force people to wait longer.. which brings us to the politics: "don't blame me, blame the IT" with the new twist "blame yourself/compatriots for not volunteering"
"cobalt [sic] computer skills"
Nothing new here. My previous employer is still using COBOL based ERP, and fixes are still provided, along with minor development. Luckily no mainframes are used, just x86 servers, and the whole thing is used with the ACUCOBOL runtime binaries.
regarding the cobalt skills...
We had a couple of Cobalt Qubes to play with 20 (?) years ago. It was a reasonably cheap x86 NAS / Mail / Web server for small businesses.
Easy to use since the underlying Linux was supposed to be used from the web UI. (as long as you knew what you were doing)
As was the norm, the SUN hardware was much more cool looking than the contemporary traditional x86 servers.
COBOL is still running
So they have code written 40-50 years ago running fine but now overloaded and probably running on antiquated hardware - I still have an RL02 in the garage if they need some more storage.
So what's the solution? Rewrite the code in Java or Rust? If they do that with today's coding methods do you think that there's any chance that the code will still be running in 40 years? Does anyone think it would even be running in 4 years?
COBOL has a major advantage that it was created for specific business tasks like this, it is significantly self documenting and flexible so it's probably not going to be a big task to maintain the code and update the hardware to support faster access - and still be running the database in 2080 because training new programmers to maintain the code would not be difficult.
Re: COBOL is still running
Rewrite the code in Java
You'd have to be certified to start a new project now using Larry's Trojan Horse full of lawyers.
Solving the wrong problem...
I suspect they do not need cobol programmers.
1)They need application programmers if they need to fix application bugs
2)They need to contact IBM and borrow a faster box. If they have space in the machine room, wheel it in, plug it in and restart. Old systems should still run on the new box. Neweer boxes have even more RAM.
3)The box may be OK, but the database is not. A faster box may help, but not if they are having deadlocks or the applications have built in serializations. These days ever things should be in processors memory or disk cache
4)There is an application bottleneck for problems like "hold this lock while I chat to the end user" a faster box will not help.
I suspect techies know the problems - management do not understand.
Volunteers?
This is y2k part 2.
BOOM.