Former Fujitsu engineer apologizes for role in Post Office IT scandal
- Reference: 1719578172
- News link: https://www.theregister.co.uk/2024/06/28/fujitsu_witness_post_office/
- Source link:
Speaking during the inquiry into the Post Office Horizon system, Jenkins apologized for his part in providing incomplete evidence of the errors and failures in the system but said it was not his intention to do so.
Horizon is an EPOS and backend finance system for thousands of Post Office branches around the UK, first implemented by ICL, a UK technology company later bought by Fujitsu. From 1999 until 2015, 736 local branch managers were wrongfully convicted of fraud when errors in the system were to blame.
[1]
Jenkins appeared as an expert witness in 15 subpostmaster cases, including that of Seema Misra, who stood trial in 2010 and was found guilty of theft and false accounting, and was sent to prison while she was eight weeks pregnant. Her conviction was quashed in 2021.
[2]
[3]
Flora Page, the lawyer who represents Misra, among others, pointed out that Jenkins had said in his submission for an appraisal that he had helped succeed in the prosecution and "Horizon was given a clean bill of health."
Page put to Jenkins that he had tailored his evidence to ensure that was the outcome.
[4]
He denied tailoring the evidence, but said: "I'm sorry for what happened to Mrs Misra but I feel that was down to the way that the [Post Office] had actually behaved and wasn't purely down to me. I clearly got trapped into doing things that I shouldn't have done. But that was not intentional on … my part."
Among the known problems with the Horizon system Page brought to Jenkins's attention in the hearing was "terrible code" in the EPOS system, as described by a computer expert. "It's just not the right structure … it indicates to me that they don't understand what those particular structures are," she quoted.
[5]Bill advances to exonerate hundreds in Post Office Horizon scandal
[6]Fancy building a replacement for Post Office's disastrous Horizon system?
[7]Post Office slapped down for late disclosure of documents in Horizon scandal inquiry
[8]Fujitsu set to be preferred bidder in UK digital ID scheme
Jenkins said the EPOS code, which had been developed in 1998, became his responsibility around 2004 and 2005, but he had assumed it had stabilized by then.
"I accepted that there were these problems in the early days, which I hadn't been involved with specifically, but there had been plenty of time then for things to be sorted out for it to be working stably," he said.
He said he did not recall if anyone had told him about problems with the EPOS code.
[9]
Earlier this week, [10]Jenkins apologized for emails in which he said Misra had been "jumping on the bandwagon" in questioning the reliability of the organization's computer system.
This month, [11]the Post Office was accused of "constantly sabotaging" the work of independent investigators examining the Horizon IT system and its problems. During the inquiry, forensic accountant Ian Henderson said the Post Office had unjustifiably withheld documents in the investigation by his company, Second Sight. ®
Get our [12]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zn7eIYglpxlobQGzgPRdeQAAAMM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn7eIYglpxlobQGzgPRdeQAAAMM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn7eIYglpxlobQGzgPRdeQAAAMM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn7eIYglpxlobQGzgPRdeQAAAMM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2024/05/01/post_office_exoneration_bill/
[6] https://www.theregister.com/2024/04/11/uk_post_office_epos_procurement/
[7] https://www.theregister.com/2024/04/09/post_office_horizon_evidence/
[8] https://www.theregister.com/2024/03/26/fujitsu_id_card_scheme/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/publicsector&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn7eIYglpxlobQGzgPRdeQAAAMM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theguardian.com/uk-news/article/2024/jun/27/ex-fujitsu-engineer-gareth-jenkins-apology-post-office-horizon-it-inquiry
[11] https://www.bbc.co.uk/news/articles/cyddlynqlryo
[12] https://whitepapers.theregister.com/
...what was he doing with his time?
Failing to read the documents that the Post Office legal team had sent him about the duty of an expert witness to be impartial, apparently. As he has repeatedly been forced to admit.
I don't think that the last four days have exactly gone well for him.
Re: ...what was he doing with his time?
yep he has had his pants totally pulled down at the inquiry had a right kicking
Re: ...what was he doing with his time?
https://www.postofficescandal.uk/post/gareth-jenkins-day-2-the-god-complex-unravels/
Looks like more than a kicking. See the comments by the inquiry chair.
Re: ...what was he doing with his time?
I am amazed that such an old emails exist. I guess there was no Whatsapp then and ability to actually delete messages.
Re: ...what was he doing with his time?
I'd guess that even if he-d read them they wouldn't have included a critical warning. The expert witness' worst problems come from the side that calls them. The examination that turns into a cross-examination trying to get them to over-commit. The statement that had to be retyped because it had to be on a different form, the original mentioned suspects who are no longer before the court. etc.
Distinguished engineer got trapped into doing things :o
‘Gareth Jenkins, former distinguished engineer at Fujitsu Services Ltd, said he "clearly got trapped into doing things that I shouldn't have done"’
Baloney, Jenkins was complicit in the cover-up. To salvage his own ‘distinguished’ reputation and avoid the expense of fixing the software.
What is it about these kind of NGOs that there is no one individual or department responsible for an issue. Every time I write to a certain local authority, I get a reply from a different individual with a different title, something something officer /s
Re: Distinguished engineer got trapped into doing things :o
Well, I've watched his entire testimony this week so far, with 30 minutes to go.
"Baloney, Jenkins was complicit in the cover-up."
What is clear from his testimony is that he clearly is/was an expert in his domain, that of the Horizon computer systems, but he was certainly not an "expert witness" in the legal sense, and had no training to be one, although he was presented as such by the criminal lawyers in court.
As he's never been a legal expert witness, he should never have been in a criminal court, as he did not know or understand what an expert witness is in legal terms, nor what their responsibilities are.
To be blunt, he was the lawyers' useful idiot.
Re: Distinguished engineer got trapped into doing things :o
I'm beginning to think he was told they needed someone to make a statement on behalf of the company, that it was just a formality and they could draft most/?all of it for him.
Jenkins was complicit in the cover-up
More than that. He seems to have committed perjury. In one of the subpostmaster trials, he testified that it was impossible for anyone to tamper with the Horizon database. But at that time he knew this was possible.
Notice too how all of the evil lying bastards responsible for this scandal have only apologised for causing distress to their victims? Nobody's said sorry for initiating the bogus prosecutions. Or putting innocent people in jail. Or bankrupting some of the victims. Or for bullying subpostmasters into making false confessions. Or interfering in independent audits. Or hiding evidence. Or lying. Or for failing to do their jobs: due diligence checks, getting Horizon's bugs fixed, asking questions when problems became public, etc. One of the PO's directors claimed he didn't know the PO could prosecute people. WTF?!
Where were the tests?
Where were the tests? Surely a system like Horizon would have had thousands and thousands of tests—unit tests, integration tests, directed tests, random tests, etc.—plus coverage analysis to ensure that everything was being tested, and a regression system to run the tests regularly. So Mr Jenkins wouldn't have to 'accept' that everything was working, he could just look at the most recent test results. Or am I being naïve?
Re: Where were the tests?
I've read a lot of documents that have been made public by the court cases and inquiry. As a programmer, I was interested in looking for descriptions of the system architecture, any off the shelf components used and how it was coded. From what I can gather, unit tests were not written for the system as it was originally delivered, and they only began to be written when the original frontend was rewritten somewhere around 2010. The original frontend was Visual Basic, the replacement Java, and it was at this time that they also replaced the messaging middleware system.
According to what I've read, even when fixing bugs in the original system no regression tests were written. I can believe this, since in my own experience of programming commercially since the 1990s, it wasn't until the latter part of the first decade of this century that management would even grudgingly allow time for unit tests to be written. So the programmers may have been familiar with the idea behind test driven development, but retrofitting that to an existing system and ICL/Fujitsu's outdated corporate mindset would have been costly. Having worked on projects where some of the development was outsourced to ICL/Fujitsu I can attest to their culture being entirely driven by doing the bare minimum for the maximum cost.
Finally, if you look at Freedom of Information requests Mr Bates made, he asked about testing and the Post Office's claim the system had been independently verified. This verification turned out to be the kinds of box ticking exercises that allow you to claim conformance with those valueless BSI standards that just result in dusty, unread ring binders full of wishful thinking. When Second Sight were finally brought in to do a real investigation, Fujitsu and the Post Office were obstructive and the investigation ultimately ended when it became apparent the results would not be to the Post Office management's liking.
Re: Where were the tests?
I remember people writing tests - if they managed to convince management - to be described as slackers as tests were not features. Running test suites was perceived as an excuse to do nothing.
Then it flipped somewhat in late 2000, where you couldn't deliver anything without tests, but still people were writing "tests" without understanding of what should be tested and how. So while they existed, mostly they didn't test anything.
This article seems somewhat kinder than others, e.g. [1]here or [2]here , perhaps undeservedly so.
[1] https://www.theguardian.com/uk-news/article/2024/jun/27/ex-fujitsu-engineer-gareth-jenkins-apology-post-office-horizon-it-inquiry
[2] https://www.computerweekly.com/news/366590300/Experts-shocked-by-extraordinary-claim-made-by-Post-Office-IT-expert-witness
Unimpressed
I've seen some of his testimony, and the impression it leaves is that he refuses to take any responsibility for any of the systems and tasks that he was responsible for.
Despite all of the recent evidence, he appears to still believe Horizon only failed in extremely rare circumstances and the majority of the cases against Postmasters were justified. Frankly this is astonishing, given that Fujitsu committed a significant body of staff to continual fire-fighting over the period in question, and staff above and below him are all openly admitting that serious mistakes were made.
That leads us to the conclusion that either he was the most incompetent of managers, completely unaware of either the technical details or the operational reality of the systems he was responsible for, or he has regularly and deliberately perjured himself in order to support prosecutions. Whichever it is, there seems to be a strong case for a prosecution against him personally for the harm and damage he has caused.
Re: Unimpressed
According to the BBC "He is currently under investigation by the Metropolitan Police on suspicion of perjury and perverting the course of justice - no criminal charges have been brought". I recall reading that investigations, into him and others, are on-hold pending the report from the inquiry.
Pending report from the inquiry..
Am I the only one that thinks the inquiry is a means of delaying any prosecution as long as possible, until the smoke has died down.
Re: Pending report from the inquiry..
I take the point, and 'can kicking' is very common in the UK, but just consider the volume of evidence being generated here.
Re: Pending report from the inquiry..
Np you're not. But that doesn't mean your thought is correct.
Re: Unimpressed
Seems rather like Scenario B - Deliberately perjured himself for the company who'd given him his honory 'engineer' title.
His only defence left is that of Paula Vennells, 'I'm actually too stupid to know what was really going on round me'.
Re: Unimpressed
His only defence left is that of Paula Vennells, 'I'm actually too stupid to know what was really going on round me'.
You might be surprised how corporations can collectively believe an untruth. If everybody knows the world is flat, then inevitably most people don't question the fact, and the few that do are inherently misguided and wrong. As a result any doubts about Horizon's veracity were denied first then acknowledged but minimised, "it'll be fixed in the next release", "it's an edge case we won't find in real world use", "testing would have found any errors", "if the code were at fault it would never have been signed off", "it's designed to balance, if it doesn't then it has to be the postmaster's on the take".
Re: Unimpressed: false dichotomy
I've watched his entire testimony, and there's about half an hour to go.
"either he was the most incompetent of managers, completely unaware of either the technical details or the operational reality of the systems he was responsible for, or he has regularly and deliberately perjured himself in order to support prosecutions"
False dichotomy.
"he was the most incompetent of managers"
He wasn't ever a manager. He is and was a techy.
"he has regularly and deliberately perjured himself in order to support prosecutions"
There is zero evidence of that.
****************************
There is plenty of evidence that he's a nerd who understood the software well, but has no legal training, and should never have been presented as an "expert witness".
There's also plenty of evidence that, like the subpostmasters, he's inadvertently found himself to be at the wrong end of institutional and systemic failures of lawyers.
Unfortunately for him, he's also been misrepresented as a key figure who deliberately went into court with malicious intent.
Re: Unimpressed: false dichotomy
How could he possibly be unaware of the support desk being briefed to say "no one else has this problem" and the extent of the remote access to patch up errors.
If any normal professional was called in as an expert witness they would surly check thier legal responsibilities before taking the stand.
And why did the court accept him as "expert" when he had clear conflict of interest and a dodgy title bestowed on him by the provider of the dodgy software.
Re: Unimpressed: false dichotomy
There is plenty of evidence that he's a nerd who understood the software well, but has no legal training, and should never have been presented as an "expert witness".
A nerd who understood the software well is an expert witness.
You neither need nor should be an expert in law. Let's look up the definition, from the industry body shall we?
What is an Expert Witness?
An Expert Witness can be anyone with knowledge or experience of a particular field or discipline beyond that to be expected of a layman. The Expert Witness’s duty is to give to the Court or tribunal an impartial opinion on particular aspects of matters within his expertise which are in dispute.
So anybody employed as an IT professional reading this who has specialist knowledge is qualified to be an expert witness.
The key point here is that the opinion is required to be "impartial".
The appearance is that he was an employee of a software supplier who had induced their staff to lie in court to cover up company mistakes instead of giving independent opinions on the issues.
Re: Unimpressed: false dichotomy
@Howard Long
"There is plenty of evidence that he's a nerd who understood the software well, but has no legal training, and should never have been presented as an "expert witness".
Given that other engineers have described the Horizon system as fatally flawed from an architectural point of view, extremely poorly implemented and with a swathe of bugs, if he genuinely "understood the software well", not only would he have known that it had serious problems but he also would have known that remote access to the system was built-in and used on a regular basis.
As pointed out elsewhere, you do not need to be legally trained to be an expert witness. Your legal obligation is to share your understanding of the matter in hand, be clear on the limitations of your knowledge, and to avoid supposition and conjecture. Complaining that no-one explained to him that just repeating corporate soundbites was not in the job description speaks volumes about his character.
I think his defence on the issue of errors in the software was "I believed they had all been fixed" - that suggests that far from "understanding the software", he had at best a surface level grasp of Horizon (or.. again.. perjured himself deliberately).
I don't think he went to court with malicious intent. I suspect he did not once think about the effect his actions would have on those being prosecuted, and was just happy to have a grand title and a nice paycheque.
I guess they've found who gets thrown under the bus.
At least he deserves it.
Possibly controversial opinion...
I've watched all of his testimony so far this week: there's still about an hour to go.
Unlike other witnesses on the Fujitsu/PO side of this inquiry, he comes over as a reliable witness.
He is a techy, and possibly on the spectrum (we all are to some degree or another). Thus he tends to answer questions more literally than some might.
Furthermore, and key to this, in a legal sense, he is not and never was an "expert witness". Until recently he didn't know that in court an "expert witness" has some legal training, training that he neither had, nor was offered. In particular he did not know about disclosure.
I'll add that Ian Henderson of Second Sight, the forensic accountants, considered him in a favourable light, as being transparent and honest in his work with them. I have seen no evidence in the inquiry that he has been anything other than transparent and honest.
To my mind, this has been a failure of those higher up the chain, in particular lawyers, and if I am to pick one out, Jarnail Singh, who was the only criminal lawyer at POL, who should've known about disclosure requirements in criminal cases, and have liaised as necessary with Fujitsu on this.
While we may not like his answers, like the sub-postmasters, he's someone who's been shafted by those who should've known better.
Re: Possibly controversial opinion...
yep I thought exactly the same. He sees things in black and white, when he knew about errors in Horizon Online and why he didn't mention that when being asked about issues with Horizon he said well they're not connected, and he's right, he couldn't just see the need to mention issues with unconnected systems.
Re: Possibly controversial opinion...
I agree with some of this. He does seem to actually be trying to answer questions and isn't suffering from the intermittently faulty memory of many of the witnesses.
However on the technical side - which I guess is why most of us are here - the issue I have, and this was the same with his colleague Anne Chambers, is the clear assumption that the system was working fine and if they couldn't find a problem in the data after a cursory check then there wasn't a problem and it was somebody else's fault. To find bugs you have to actually go looking for them.
The whole culture of the PO and ICL/Fujitsu seems to have been to see everybody else as the problem when a good hard look in the mirror was needed.
Re: Possibly controversial opinion...
"the issue I have, and this was the same with his colleague Anne Chambers, is the clear assumption that the system was working fine and if they couldn't find a problem in the data after a cursory check then there wasn't a problem and it was somebody else's fault. To find bugs you have to actually go looking for them."
Yes, I'd agree that this is a key disjoint, although from his evidence, he was doing a lot more than "cursory checks", although lacked time to do sufficient investigations as a supposed "expert witness" when issues were requests were often raised by the PO on the day before he was expected to give criminal court testimony.
His (and others') dogmatic claims about the reliability of Horizon's operation in the face of "bugs, errors and defects" does seem to be overstated bearing in mind the real world results. At least he didn't call it "systemically robust"!
Re: Possibly controversial opinion...
Until recently he didn't know that in court an "expert witness" has some legal training, training that he neither had, nor was offered. In particular he did not know about disclosure.
I spent 14 years in a job where being an expert witness was at the core of the work. No legal training was given. OTOH I was constantly thinking, as should any witness, expert or other wise "If I get this wrong someone could be wrongly convicted".
Apart from that the one warning I would give to anyone who is being called as a witness, especially as an expert is "be careful of the side that's calling you; they may well want you to say more than you intend so you must stand your ground against them as well as against counsel on the other side".
Re: Possibly controversial opinion...
"I spent 14 years in a job where being an expert witness was at the core of the work. No legal training was given."
I have news for you: in the legal sense, according to the inquiry's evidence, and barrister questioning, you would never have been an "expert witness" in a criminal court.
Re: Possibly controversial opinion...
"If I get this wrong someone could be wrongly convicted"
Well yes, and I'm pretty sure that you read all the files that any solicitor sent you very carefully. Alas Mr Jenkins did not so it appears...
https://www.postofficescandal.uk/post/gareth-jenkins-day-3-criminal-stupidity-or-helpful-fool/
"Being involved in the criminal prosecution of anyone should focus the mind. There is a lot at stake for the individual concerned. Basic human courtesy demands you take whatever task you have or are given very seriously."
"Today [day 3, 27th June 2024] Gareth Jenkins’ almost Olympian levels of disengagement with his wide-ranging role in helping the Post Office prosecute Subpostmasters was laid bare at the Post Office Horizon IT Inquiry."
Re: Possibly controversial opinion...
One may also ask why the judges and defence lawyers in multiple cases didn't determine whether he was expert or not in the legal sense. But I agree it is POL's responsibility to ensure their expert witness is that and knows that. They should carry the can of putting an incompetent (certainly in the legal sense and possibly in the occupational sense). I'm sure if I was called as an expert I would expect the court to provide a checklist and briefing on what I can be expert about and what not.
BTW doesn't the weight of the error log have some correlation with stability? Oh and I've seen errors as portrayed in the TV programme with my piss-poor attempts with Visual Basic. Also in a certain (non-ICL) leading accountancy VB package at the time.
What was I supposed to do?
"I accepted that there were these problems in the early days, which I hadn't been involved with specifically, but there had been plenty of time then for things to be sorted out for it to be working stably," he said.
*I did nothing g and assumed everything was okay, how was I supposed to know*
Doc we tried nothing and we're all out of ideas.
Whatever punishment he gets should be double, that is open and clear negligence of job duties, fr9m day one with years to rectify it and not even trying to see if there was a problem.
Judge not lest ... Oh, hang on
In all of this inquiry Fujitsu / ICL , the Post Office higher-ups, investigation teams etc have all been criticised, but I wonder at the judges who heard the cases and found people guilty and punished them. Why were they so accepting of the Post Office's claims and Fujitsu's claims? Why did they not ask even the simplest of questions about testing that would have shown Horizon to be basically a 'black box' system with no independent assurance of integrity?
I humbly submit, m'lord, that the judges be summoned by the inquiry and required to account for their conduct and failure to ensure justice was done. (Don't worry, there is less chance of that happening than Liz Truss being re-elected as Prime Minister on 4th July.)
(Big Brother icon, because, well, to whom does the judiciary answer?)
"assumed" & "not recall if anyone had told him"
Jenkins said the EPOS code, which had been developed in 1998, became his responsibility around 2004 and 2005, but he had assumed it had stabilized by then.
"I accepted that there were these problems in the early days, which I hadn't been involved with specifically, but there had been plenty of time then for things to be sorted out for it to be working stably," he said.
He said he did not recall if anyone had told him about problems with the EPOS code.
If it was his responsibility then should he have not made it his business to find out if there were problems so that they could be fixed ?
Apparently not. So what was he doing with his time ?