Sheffield Uni cooks up classic IT disaster in £30m student project: Shifting scope, leadership changes, sunk cost fallacy
- Reference: 1637055013
- News link: https://www.theregister.co.uk/2021/11/16/sheffield_university_classic_it_disaster/
- Source link:
As [1]The Register reported , the original plans for the Russell Group university's Student Lifecycle Project (SLP) – a £30.4m scheme to build a new system for managing student records – has been ditched.
The more you look into something, the more complex it becomes. And then the project leader felt like we needed to get more staff to look into that complexity, so we just snowballed the number of business analysts and spending started to go up so dramatically...
The project stumbled on integration with a Corporate Information System (CIS), which [2]we revealed last week was running on Oracle 11.2.0.4.0 . The software version went out of premium support on 31 January 2015 and extended support on 31 December 2020. Oracle provides "Sustaining Support" with limited updates and support options are available from third-party partners.
When SLP was first planned, it was imagined it would replace the CIS. But as work began developers realised it was connected to so many other legacy systems – with little or no documentation – that this was not going to work. The plan then shifted to using CIS as "middleware" to integrate SLP into the University's central identity management system, building access, timetabling and so on.
"It's plugged into so many different things, you can't really separate it out and then to try to connect the new system to every single legacy system," one insider told us, adding that it "proved uneconomical."
[3]
The plan to replace CIS was abandoned, and instead, the SLP was to use it to help the new system talk to all the legacy applications. But then the project team began to realise the data models between SLP and CIS were fundamentally incompatible. After a change in leadership, the SLP put more effort into understanding the fundamental challenges. That required consultants.
[4]
[5]
"The more you look into something, the more complex it becomes. And then the project leader felt like we needed to get more staff to look into that complexity, so we just snowballed the number of business analysts and spending started to go up so dramatically. Then we got in a lot of contractors and we started to have a lot of churn of those contractors, so they acquired a lot of knowledge and then they would leave," the insider added.
There was another change in project leaders. Meanwhile, the director of IT services was an interim contract, leaving the whole IT department, "directionless," our source claimed.
[6]
Unsurprisingly, the plan for integration also hit problems. "Every time you explore the idea of only sending back a simplified data structure, you run into the problems: namely that there's no documentation to help you understand which systems use which data items," the insider said.
"It's only once they tried to build these integrations, which we always knew would be difficult, did we realise you've got to do the hard stuff first. Sometimes that's easier said than done, only with hindsight do you know exactly what the hard things are. But fundamentally, they didn't deal with some of the hard questions early on," the insider said.
Continuing with complex projects despite setbacks has been characterised as a typical manifestation of [7]the sunk cost fallacy , where time and money already invested in projects is used to justify future spending.
[8]
At Sheffield University, a new director of IT services, Bella Abrams, joined in 2019. Following her arrival, it was realised that one of the main reasons for doing the project in the first place – statutory reporting – had become unnecessary.
The Higher Education Statistics Agency planned to introduce new ways to collect data from universities, part of the means by which the institutions get government funding.
The SLP – and the Tribal SITS software it uses – was necessary because the old system could not support the new data standards, but that was never promoted to management.
"That was a big driver for SITS. But it wasn't really sold to the business in those terms because it's not sexy. It's not process improvement. It's not a better experience for students. It's just to meet statutory requirements and that was an important thing for us," the insider said.
However, the [9]HESA Data Futures project was itself delayed, which gave more time to implement SITS. When Data Futures came back, it was reduced in scope, such that the old CIS system would be able to comply with the requirements.
[10]System at the heart of scaled-back £30m Sheffield University project runs on end-of-life Oracle database
[11]Sheffield University scales back student system after Oracle integration stumbles
[12]Ah, uni days! Drugs, sex, parties... sci-tech startups? Not so much
[13]We are 'heroes,' says police chief whose force frisked a photographer
"That was probably the final nail in the coffin for SITS, when they realised they could do Data Futures out of CIS," the insider said. They argued that Tribal SITS was almost "unimplementable" in a modern project that had high student and staff expectations for features and user experience, although "bare-bones" implementations were straightforward. Kent and Nottingham Universities had also hit delays with their implementations, they said.
"They say SITS is like SAP: configurable, with no code. But the configuration is so complex. You're building all these views and processes and you get all of the problems that you would do with conventional software development, [however] you just don't have any of the standard tooling that makes your life easier while you're doing it," the insider said.
The vendor Tribal has introduced a replacement for SITS called Tribal Edge. The company has been contacted for comment by The Register, as have Kent and Nottingham Universities.
It was only earlier this year the SLP instead decided it would integrate SITS and CIS. In the most recent news, the university said that would not be possible.
In a statement to The Register , the University and College Union, which represents university staff, said: "University management initially pursued the use of SITS, without careful consultation and against concerns raised by staff. Since that time, SLP has undergone several substantial changes of plan, each one driven by potentially foreseeable difficulties with the project, and each one requiring additional financial investment. If University leadership had been led by staff expertise, we might have avoided the difficult position we are currently in."
A Sheffield University spokesperson said: "The investment in the programme has already delivered new systems that previously did not exist at the university and our work to date will be important as we develop our future plans."
But the university has not responded to The Reg 's questions about how it is supporting its ageing Oracle 11.2.0.4.0 database on which its crucial operational middleware and statutory reporting relies. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2021/11/03/sheffield_university_scales_back_student/
[2] https://www.theregister.com/2021/11/11/sheffield_uni_out_of_support_oracle/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YZOPY@M7rNNQkIyafW@mGAAAAMY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YZOPY@M7rNNQkIyafW@mGAAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YZOPY@M7rNNQkIyafW@mGAAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YZOPY@M7rNNQkIyafW@mGAAAAMY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://psycnet.apa.org/record/1985-20101-001
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/databases&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YZOPY@M7rNNQkIyafW@mGAAAAMY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.hesa.ac.uk/innovation/data-futures
[10] https://www.theregister.com/2021/11/11/sheffield_uni_out_of_support_oracle/
[11] https://www.theregister.com/2021/11/03/sheffield_university_scales_back_student/
[12] https://www.theregister.com/2018/03/01/revitalising_university_spinouts/
[13] https://www.theregister.com/2017/05/05/photographer_stop_search_terrorism_act/
[14] https://whitepapers.theregister.com/
I am (sort of...) involved in a bigger project of migrating critical infrastructure away from some very legacy architecture, developed about 50 years ago. Talking to some of the earlier developers you cannot even rely on the source code being correct, as some fixes were done in machine code in the compiled program, it was easier that way. There is a reasonable good understanding of what tries to talk to other systems, and how, but the project has been going on for a while already. The data exchange formats are well documented (mostly), the business requirements are very well written, but migrating this thing is a major effort. The project actually started by having everything new on more modern systems (cuneiform tablets would be "modern"!), forcing the new stuff to talk to the old stuff. Work is also migrated bit by bit from the central system, leaving the old thing in place for now.
Yes, it makes sense to start replacing the all encompassing system bit by bit, there is just no way it can be done in one giant project. Chosing a smaller subsystem as the first candidate while keeping the big, central data kraken in its place makes sense. There is just no conceivable way you will be able to replace that central abomination all in one go without a lot of money, a lot of work, and a lot of downtime. Pick any three of those.
(and an outdated Oracle DB is not the issue here...)
> So if I'm reading that right, they kept the obsolete system because they had no idea how to replace or upgrade the obsolete system.
Worse, they then decided to make it a hub without fully understanding what it does. A textbook example of how skipping documentation and the boring hygiene stuff costs many, many times more when it has to be done retrospectively. (Or in this case, not done.)
I wonder if Sheffield Uni has a business school where they could teach this as a case study?
"But it wasn't really sold to the business in those terms because it's not sexy."
That demonstrates part of the problem with universities. They are supposed to be non profits, but act like businesses.
Not doing stuff because its "not sexy" is a more widespread problem, of course.
The problem with any concept that you can configure something without code is that you always need something just as complicated as code to actually do the heavy lifting in the configuration space.
People keep falling for this concept without realising that anybody who can understand the labyrinthine rules behind the codeless configuration will be fully capable of dealing with code based configuration anyway. Do the managers really think that it's (a) going to be possible for them to futz around in the config without a huge learning curve or (b) anywhere near a good idea for somebody who hasn't climbed that curve to mess around in a configuration file?
30 million bloody quid, easy to piss money away when you can hike up the student fees
What this boils down to
is that the uni spent £30 million to learn that they had to replace everything else before implementing a Student Lifecycle Project, which is just fancy wording for an SQL database of student grades.
Oh well, sometimes lessons are expensive.
I would have said expensive lessons are the best remembered, but somehow I doubt that the uni is going to remember anything from this.
"data models between SLP and CIS were fundamentally incompatible"
here's me thinking that munging data from one form to another was what programmers did.
I expect there was a lot of "if you'd told me up front you wanted to do THAT with the data then I wouldn't have programmed it THIS way in the first place."
Contractors
and we started to have a lot of churn of those contractors, so they acquired a lot of knowledge and then they would leave
That usually happens when the complexity and scope turns out to be not adequate to the money on offer.
The fundamental here is that they tried to modernise a legacy set of systems by implementing a new system. Rather than take a brown field approach cementing existing business processes structured around the legacy systems they should have taken a green field approach and implement a cloud student and ERP system and changed the business processes to fit with best practice blueprints for England HE. Every English University has exactly the same reporting and record keeping needs as they are set by government, you don’t need to analyse them to death or design your own bespoke processes to manage them. Honestly, you could have a three ring binder on how to operate a English University and it would 99% work for all universities.
So if I'm reading that right, they kept the obsolete system because they had no idea how to replace or upgrade the obsolete system. Then of course, nothing would TALK to the obsolete system because it IS obsolete and any support is going to be for CURRENT versions of the product, not something from the 90s.
This wasn't a project. This was some bean counter's glory hole for their ego, trying to stroke their own pride and power by "controlling" this huge disaster. I'm sure those three leaders had great power and fanfare before they were fired or gave up.
Yep. Nothing like watching bean counters try to drive technology. Without fail, in every single corporation or business I've worked for since 1988, the bean-counter driven projects have been abject failures because they're driven by "cost cutting" and vague "advantages" that have nothing to do with making a system actually WORK.