News: 1610975705

PEBKAC

Competent system design would admit that PEBKAC is likely and should prevent permanent data loss - but this is only part of the problem. Data storage in this environment is complex, complying with legislation created by innocent politicians makes it even worse - particularly when the work is sub-contracted out to a bunch of programmers who's only experience with police data collection and maintenance is just a DUI ticket.

Cock up jobs like this are all too regular. Can you imagine the damage if something like this happened in the airline industry. Oh, hang on...

"Competent system design ... should prevent permanent data loss"

They've been arguing for years that they're so good at preventing data loss that they can't delete records of innocent people despite having been told to do so by the courts.

> "The incident highlights the fact that IT practitioners should be accountable to independent professional standards"

I've no doubt there's going to be an epic amount of recriminations in the wake of this. But I'd be wary of pushing all the blame onto the "IT Practitioners".

There's clearly two issues here.

The first lies in the processes which allowed a breaking change of this magnitude to get onto the live plaform.

The second lies in the backup processes (or lack thereof) which mean that they've been unable to recover the data.

In both cases, a significant factor will have been the amount of time/budget allocated, and the oversight from upper management.

And neither of those are "IT" specific issues.

E.g. I've worked in places where the response to rollout failures was to double down on process. The result was that things took much longer to roll out, and that the "coal face" IT people actually had less time to deal with the technical side of their job, since they were constantly having to complete paperwork and defend the need for said technical activities.

As a result, customer-reported issues remained open for significantly longer. If they were fixed at all - the time/energy cost of fixing some things simply fell below the level where the effort could be justified.

And at best, the number of issues remained pretty much the same. Since the new high-overheads process didn't address the underlying issues; they just slowed down the rate at which changes were rolled out.

There's always going to be a balance of some sort - there has to be. But the blame definitely doesn't all rest at the feet of the IT people.

Wherever the finger points, you can bet it won't be at the PHBs (who, lets face it, are more likely to have had a hand in the mess by pinching the pennies somewhere along the line).

"The incident highlights the fact that IT practitioners should be accountable to independent professional standards"

Notice that it was the BCS that said this. What a surprise(*), as they've always been trying to make IT a chartered profession so that they could charge for qualification courses and exams.

The only problem is that their certification will always be at least 5 years behind the current trends in software development, because of the time it takes to create the certification.

(*) Yes, I've always been bitter, because they did not accept that my degree did not even exempt me from the BCS part 1 entrance exams, so I never joined.

We used to get membership of the BCS provided by our work. As far as I can tell, this consisted of a regular (quarterly?) copy of a very dry and dull magazine, and the ability to go to some meetings and talks that nobody ever bothered with, so our employer stopped paying for it, in favour of more useful things like Pluralsight subscriptions.

At some point there was talk of scientists having to become members of the relevant charted institute so I joined mine whilst my degree counted for admission in case they made the MIBiol exam route compulsory (I'd long since promised myself no more exams ever). The talk came to nothing but what made the membership worthwhile was a section of the magazine. It turned out most/a lot of the members were teachers and there was always a selection of exam howlers.

I am not sure if backups are really a problem here. This process might have actually require there to be no backups.

The police WILL have records that they are not allowed to keep indefinitely and that will need to be permanently removed when certain criteria are met. Perhaps because by law they are only allowed to keep some record for X years. Or because the longer retention of some data is based on the outcome of court proceedings where a not guilty verdict might have a legal impact on retention of some records.

The police SHOULD have a system that culls any data that meets these criteria and that data SHOULD not be retained in backups.

The police MIGHT have a weekly automated job to delete all the records that have met the deletion criteria in the previous seven days. This process MIGHT have some manual requirements (because of a flawed design or because of a deliberate safeguard). If there is a human error involved this is where that might have happened. It may not be a coincidence that it happened at the start of the year. Perhaps someone accidentally changed something to the wrong year and an additional year of data was deleted. In a process that is meant to be legally irretrievable so no backups.

Just speculating but it would explain a thing or two...

I'd speculate that the thing that someone might have accidentally changed might relate to new requirements arising from political decisions to no longer being part of EU-wide policing, and the requirements to remove data relating to EU citizens outside of the UK that arise from that.

This would also explain why the likes of Priti Patel have been so vague about what the actual cause was, because the long-and-short answer is her and her brexity cohorts.

"why the likes of Priti Patel have been so vague about what the actual cause was"

The last time the HO tried to brief their front person on techy matters all she thought she could remember was something about hashtags. I guess they're avoiding going there again.

"The police WILL have records that they are not allowed to keep indefinitely and that will need to be permanently removed when certain criteria are met."

Nothing is ever permanently removed. Beat plods won't be allowed any visibility, sure. But senior police will probably be allowed to see that there was a record of something, and can get an ministerial exception to see the data. Cheltenham will keep everything, for ever. That's their job.

There's a long history of there being records they're not allowed to keep indefinitely, that have been kept indefinitely. The excuse being the difficulty of removing them because although they should have a process for removing them, they don't. It's been reported here a number of times. The admission that they have a weeding process blows that one out of the water.

I did note to my network admin (a couple of years ago, using win 7) that it was a bit unnerving that the rightclick 'delete' was perilously close to the 'rename' and 'create shortcut' function... don't worry that what the bin is for... a big problem is, if it is on a network share, it **does not** go to the bin, but disappears..

Samba can have a virtual recycle bin for just this situation.

can I get 500,000? going once, going twice, sold to plods without a working backup?

If government IT overruns are anything to go by, you can probably whack another 0 on the end of that.

The statement I heard was that the deleted data was of people released without charge. In the event that this statement was actually true, isn't that what is supposed to happen?

That was the original story I heard too, but both the volume and range have expanded since then.

There's been years of campaigning against police retention of data from those who are interviewed then released (or to use a technical term, "innocent"), or from mass 'round up the usual suspects' attacks on demonstrations. (Anyone remember kettling?) The police / government response was not much more sophisticated than "No. Shan't. Remember Ian Huntley?", despite him and that case being a red herring in this context. A lot of people will not be unhappy that those records have now been lost due to an admin SNAFU. Losing actual criminal records and those for investigations in progress is a little more serious.

"A lot of people will not be unhappy that those records have now been lost due to an admin SNAFU."

And a lot of people will be very unhappy on account of being caught out after claiming it couldn't be done.

What if this wasn't an accident? As everyone points out, this should be nearly impossible on a mature system like this, which should have backups and lots of barriers in front of operations such as this.

All you need to know is that an unnamed MP was recently released without charge for a (heinous) alleged crime. Perhaps there are other, unreported, cases as well relating to Tory MPs, or donors.

So this was potentially a planned data loss to remove the records relating to this, covered up with other deleted records, in a needle in a haystack method. They'll gloriously say they recovered 90% of the records in due course from backup or paper trail, but the rest of the paper trail for the critical records will have been disappeared. Did

Yeah, it seems far fetched and conspiracy theory like, and I'm not a fan of these in these times, and it most likely was poor backup practices and system access control. But still.

If I hear that somebody has been released without charge I assume it was because there was no provable case against them and that they are, therefore innocent. That is a basic principle of the English justice system (no "not proven" stuff here), has been for centuries and hopefully, despite the longings of the HO and intelligence services, will continue to be. It applies to everyone. It applies to Tory MPs and donors. It even applies to YOU. And if you think about it a little you'll realise that the reason that it applies to Tory MPs, donors and everyone else is so that it can apply to you. And if you do a little more hard thinking you might realise that that is the most valuable protection you can have under the law.

Meanwhile, those of us who've actually had the job of investigating allegations of criminal behaviour appreciate just how important it is that the subjects of those investigations do go unnamed.

I'd be checking the bank balance of the DBA involved.

Nice of the BCS to blame the developers. This was probably user error compounded by worse management error because there were no backups.

I've always thought the BCS irrelevant to my profession. I'm glad I never joined despite getting the hard sell a few times.

Of course they do. They have an agenda. They've been for decades trying to make IT chartered profession where they can have out of date, expensive certifications.

Must've been over 2 decades ago when I looked and was like how much? For what? Sod that...

With hindsight, yes they are irrelevant.

Absolutely. I had completely forgotten about them until they surfaced in this article.

> Nice of the BCS to blame the developers. This was probably user error compounded by worse management error because there were no backups.

Shirley, the BCS statement is making the point that it wasn't just the developer's fault because there were other people in the chain that should have checked as well but either didn't or also missed it?

...see if the new guy will fall for the old DROP TABLE gag.

My money's on forgetting the WHERE clause in a DELETE statement. I've seen a couple of people manage that one.

I had heard that the PNC has not been compliant with this legislation for years.

Scuttlebutt has it that it was the implementation of rules to become compliant which went wrong, and that this problem goes back to sometime in last year so some data will have rolled off their DR backups. If that was the case, then the rules to make sure that data that should be completely purged, even from the backups may have been incorrect.

Under UK and previously EU law, the police to not have catre blanche to keep information gathered from suspects forever. If someone is arrested for a crime, then the police can take and store information about the person. If they are subsequently released and the charges dropped, or go to trial and are cleared of the crime, then the police are bound by law to delete the fingerprints, DNA and other data that they've collected after a certain period of time.

Anybody convicted of a crime will have their records stored forever (incidentally, this is sometimes quoted as being why police will take fingerprints from people who are stopped for minor motor offenses, even when the data has no bearing on the crime - "it's standard procedure, sir"). Once someone has been convicted, the police have the right to keep any data they've collected whether it was needed or not.

This deletion policy does not please the police. They would really like to build up a complete database of all the people in the country whether they've been found guilty of a crime or not. If I were to put my conspiracy hat on (the one with the tin foil lining), this news story could be a deliberately created attempt to shock the people and government into a policy change to allow them to keep more information for longer.

Within the last decade, I heard a broadcast interview about the police's DNA retention policy with who ever was the chief of ACPO at the time, where he repeatedly called people who had been arrested but not yet tried as "criminals" instead of "suspects" (not even "potential criminals"), even after being pulled up by the interviewer on more than one occasion. I'm sure that some members of the police regard all of the public as criminals who just have not yet been caught yet!

In theory backups should be cleaned as well, but in practice that is usually not practical.The important thing here is putting the deleted data " [1]beyond use ". So it might still be on a tape/... but it cannot be accessed through normal use/mechanisms.

[1] https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5

Is this the same Fujitsu that was the centre of attention in the post office post master scandal?

Fujitsu are the hardware supplier in this instance. The software is maintained by someone else.

(anon because the "someone else" is another division of my employer)

If they are able to recover records from backups then that sort of puts paid to the whole deleting-illegally-retained-personal-data stuff doesn't it? I mean, you sue the police to delete your records when you are cleared of a crime, order is granted but then even deleting the live record just removes the current version, no? Perhaps the background here is an effort to allow backups to be routinely restored in the future?

Loss of loads of data only some of which should have been marked for deletion?

Can it run Crysis? Yes, it would seem so.

The BCS said in a statement: "It is likely that a developer, test analyst and release manager would all have been part of the process leading up to the failure...."

No room for a competent Change Manager in the mix?

Can't help but think what a gift this could be for all those pesky Freedom of Information requests....

Sir Humphrey Appleby:

Well, this is what we normally do in circumstnces like these.

James Hacker:

This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967...

James Hacker:

Was 1967 a particularly bad winter?

Sir Humphrey Appleby:

No, a marvellous winter. We lost no end of embarrassing files.

Everyone mentions "failure in the backups" but doesn't cover other factors. This is a live system, presumably getting updates (new fingerprints, DNA results etc) on a minute by minute basis. Unless you've frozen update access, taken a backup, done your update and noticed any issues before unfreezing access, you CANNOT just go back to the backups without losing ALL changes made since the backup.

This was a "routine housekeeping data update" which could mean it wasn't done by the IT admins, but an end user data administrator. As such, there wouldn't be a change control round it to take backups and lock access etc, so there could be hours worth of data between backup and housekeeping.

Or, simply they didn't realise they'd lost the data until after their change window (which would be a failure in checkout). In either case, restoring the last backup isn't an option.

What's probably taking the time is restoring the data to a test system, unpicking the data they need to restore and figuring out how to add it back in while retaining data integrity in the production system. In most databases (particularly one which has "evolved" over many years with changing requirements), that's not going to be easy. Particularly when any subsequent failure will be on the front pages, adding to stress levels...

That's for sure.

Also citing "Human Error" is not root-causing the real problem.

That makes is sound like the cleaner mistakenly polished the spinning rust.

The PNC doesn't store dates as 2-digit years and then assume it's +1900 for dates above 21 does it?

A housekeeping error can only mean a cleaner wielded a vacuum in the server room.

...wait...

...wait...

..Deck of Cards by Max Bygraves...

Did Robert'); DROP TABLE criminals; get arrested?