News: 1610712546

Clearly not tested correctly, perhaps the Queen of Carnage was in charge?

Or did anyone state that your data isn't backed up until you've done a restore of all data?

"Or did anyone state that your data isn't backed up until you've done a restore of all data?"

Surely it'd take a while to get specific data off a backup though? I mean, your standard backups should also be purged of irrelevant data under GDPR (remember the conversations on this forums about that?) so maybe you'd have the data on tape as part of an archive. In that case you have to restore the whole archive backup onto a separate computer (as your usual computers are currently being used...) and then retrieve the individual files there.

GDPR has specific exemptions for law enforcement.

Guidance from the ICO suggests there’s little risk to retaining the data on backups as long as it is just used for backups and will be expired on a defined schedule https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/individual-rights/right-to-erasure/#ib5

" Guidance from the ICO suggests there’s little risk to retaining the data on backups "

There's a hazard that nobody seems to have allowed for. If records have been deleted as a result of data subjects exercising their right of erasure, and this has been done since the last backup, in event of a restore from that backup the deleted records will re-appear. In order to be fully compliant (is there really any other kind?), an organisation will need a means of identifying those records and purging them from the restored data set.

One wonders if marketing groups who get such requests are periodically restoring old tapes and not reprocessing the deletions - I'm seeing a bunch of such stuff reappear months/weeks/years after having unsubscribed....

It makes a nice excuse to give to the ICO when caught, doesn't it?

My understanding is this is why databases have transactions logs. So there will still be a record of every change made since the backup (including the erroneous ones) which can be selectively applied. Databases are good at this sort of thing...

Databases are good at replaying transaction logs onto backups, it's true. They're not so great at retrieving those transaction logs from a room full of smoke particles if your server has gone up in flames though, which is one of the purposes of backups.

Admittedly, if you want that sort of level of protection, what you need is a mirror of your server, and log shipping to keep it in synch. Again, not so useful if your DR mirror has gone up in flames because it was situated in Buncefield. This does, of course beg the question of why anyone would situate their DR facility next to something as potentially explodey as an oil refinery.

Unless you have inside knowledge that you are not sharing, I am calling bullshit.

The database at the core of the PNC is not the complete system. It is linked to a wide number of ancillary databases, such as DNA database, Fingerprint Database, etc. It is quite possible that the PNC was backed up. However, when the purge job goes ahead it will tell each of these other databases to delete the records as well. So they go ahead, in the sure and certain knowledge that the "Are you sure?" question has already been answered by the PNC operator.

Probably about a second after saying "Yes", the PNC operator has that "oh shit" moment we have all had in our past. Normally we would fake some form of error, shit the system down, restore the last backup and roll forward to about a minute before the mistake. Simples.

This is a real problem in any distributed system. Because now you have to get each of those system to restore and roll forward to the same point in time. And this almost never happens because the backups, transaction logs, even the timebase, is not synchronised across all these system. So you get left with dangling references between systems.

Its a problem for distributed systems. Its a real problem when you have distributed systems run by different organisations, on different hardware, OS, databases, and most probably with different backup strategies.

I agree is a monumental cock up. But a backup would probably not have been the simple solution you suggest.

So why are the police keeping those records anyway?

"So why are the police keeping those records anyway?"

If the police delete all files relating to arrests that result in no further action, then one important casualty would be the ability to hold the police to account for said arrests. Because, of course, there would be no record of them happening.

So some information needs to be kept, at least.

A record of the occurance for a limited time only maybe. But that should not include biometric data, address, phone number or date of birth.

At the very least, you need to keep a record of who has previously been interviewed on an ongoing case, and then released, so that someone doesn't come along later, look at the case and say, "hey, it looks like this guy has been involved, let's bring him in for questioning". Because, you know, that person would then have a pretty strong case for police harassment.

The pendulum swings both ways, and such.

Reading between the lines, it seems the data that was deleted was data that should have been kept, so pertains to ongoing investigations. At the very least, plod is going to have to do some extra work to eliminate people from enquiries. For example, if they have accidentally removed fingerprints of a person who has been burgled, used to eliminate them, then it's going to mean that if they do have prints from the scene-of-crime, then that person could be incorrectly identified as a suspect and have to be eliminated from the investigation again. Given how overstretched the police are from systemic underfunding, that is going to be a ball-ache for someone.

Posting anon, because I work for the company that is responsible for the PNC, although in a completely different division, so technically I should probably keep my opinions to myself.

Pretty sure that the "accidentally" removed the data pertains only to MPs & other officials that have been arrested for not following the lock-down rules...

t was my thought. Either they want to 'accidently' lose the fingerprints lifted from the door handles at Barnard Castle or someone in a camel hair coat and a fedora dropped a monkey or two to make a problem 'go away'.

"used to eliminate them" lol.

get prints, match against db, if victim shows up in the results don't bother chasing them. cops are on a trawl with this elimination nonsense.

I'm not a copper, but I'm pretty sure the normal course of an enquiry goes along the lines of:

1) Find any leads.

2) Follow those leads to find suspects.

3) Eliminate any that obviously didn't do it.

4) hopefully end up with one, and find enough evidence to prove they did it.

If you "accidentally delete" the records of having done steps 2 and 3, and you haven't got to step 4 yet, then you pretty much have to go and do those steps again. If nothing else, it's a waste of police time, and a potential massive inconvenience for those people who had previously been eliminated.

The process you describe applies only to reported crimes (that the police bother to investigate), which have no obvious suspect.

A great many cases *start* with the arrest of someone "suspicious", and the next step is in looking for any crimes that they may have committed. Even if it is found that no crimes had in fact been committed and the person is released with no further action, their fingerprints, DNA, mugshot and other personal details remain on the PNC database, ostensibly for 3 years if the person has no criminal record (but no guarantee that they will in fact be deleted after that time).

Plod has to delete the data after 3 years anyway. What's more, keeping a backup after this time would mean the data wasn't deleted, which would be illegal. To be honest, I'd be more worried if they _were_ able to restore a backup after this cock up...

https://policecautions.uk/2019/01/30/how-long-can-the-police-keep-my-dna-and-fingerprints/

What if I was never Charged or Convicted?

Charged but not convicted

If you were charged but not convicted of an offence, at any age, then your DNA and fingerprints can be retained for three years, plus a two year extension if granted by a District Judge, or indefinitely if you have previously been convicted of a recordable offence which is not “excluded”

Very likely they're violating the law by keeping them. Who is going to enforce the rules if no one is checking.

Another possibility is that 150k records are for the last three years - now that'd be even worse.

The records are for the last three years, or there wouldn’t be such a fuss. The normal purging process obviously went too far. I wonder if it was to do with the new year?

I suspect, but have no evidence, that the normal process had had to be "adjusted" to cope with the changes to data sharing with the EU and that "adjustment" was done in a cack-handed fashion, most likely because the requirements weren't known until the very last moment, and it had to be done at some point between Christmas and the New Year by some poor sod who would have preferred to be drunk.

Of course, how could I have forgotten Brexit?

Wasn't this a plotline from a "The Thick Of It" Episode?

Massive irretrievable data loss.

...Bobby Tables!!

Obviously this kid has grown into a teenager and gone off the rails :-D

Whoopsie!

Yes, she's going to have to spend the weekend working on fixing it ... by finding someone to blame.

We've probably all done something similar but then we've probably just restored the deleted data from a backup prior to the issue (using our DR system). Surely its possible (perhaps not cost effective) to reinstate the data, in fact I'm surprised they don't have a procedure for doing so just for instance like this or cases where it has been deleted with malicious intent. The other option is of course to mark the data for deletion so it is in effect in the wastebasket and can't be seen/used for a period so you can then recover it when you realise you have got it wrong, once the safety period is passed its deleted.

Bit worrying its that easy to lose data from the PNC.

I guess the EU data deletion went a bit to far.

Its not that simple, because of the links to distributed system. Having to restore and roll forward one system is bad enough. I've done it on mainframes and fun is not the word I would use. Add in that this is not a relational system, its ADABAS, and you have a whole new circle of hell to navigate.

Then, its not one database. Its many, with the PNC at the heart. You would have to restore and roll forward all the linked database. Now you REALLY are in the dark and smelly stuff.

"individuals who were arrested and then released with no further action"

OR individuals who were arrested and then de-arrested?

Perhaps they used the de-arrest weeding process by mistake?

Surely this is nothing to do with the vendor, Fujitsu, but the Home Office procedures?

Depends on numerous things, perhaps the Home Office Procedure was correct, and the Vendor/maintainer did the wrong thing, or as its likely something just designed to cleanse EU data since BREXIT, the Home Office may have correctly stated what they wanted to happen and the Vendor/Maintainer got it wrong. In either case the design and implementation of the system seems flawed for something so important that data can not be easily offlined and recovered.

Ah Fujitsu - the same lovely people whose faulty Horizon software resulted in postmasters losing their jobs and being jailed for financial crimes they didn't commit...

So, they had a backup facility in 2005, which burned down, and nothing has been done about it in 15 years ?

Is there anyone with an ounce of IT competence anywhere in UK Government ?

Obviously not.

> So, they had a backup facility in 2005, which burned down, and nothing has been done about it in 15 years ?

What makes you think nothing was done? There's no mention in the article to suggest that a replacement backup site wasn't built.

It is misleading to describe the event as "a Buncefield oil depot fire". A colleague of mine was at home in bed when it happened, several miles away, and his entire house shook with the force of the explosion. According to the BBC it was the largest explosion in the UK since the second world war, and was audible in parts of the Netherlands:

https://www.bbc.co.uk/news/uk-england-beds-bucks-herts-34919922

"On the morning of 11 December 2005, the UK experienced its largest explosion since World War Two. The huge blast at the Buncefield fuel depot in Hemel Hempstead was heard as far away as the Netherlands and shrouded much of south-east England in smoke. "

Didn't somebody awhile back say they could not comply with GDPR to delete information on people innocent of any wrong doing, as too complicated and time consuming and they got a whitewash get out of jail card. Seems they can be delete very easily !!

More lies and deceit from the Police ?

The PNC is old and creaky - I mean the clue is there in the word mainframe . I'm entirely unsurprised that it has proven to be impractical to retro-fit GDPR into something written several decades ago in a niche language.

Blame the government for failure to provide adequate funding for a replacement though. "The party of law and order". Pffft.

the clue is there in the word mainframe

I'd have thought that having all the data in one place, rather than scattered across the cloud, should make it easier to manage, not harder, in principle - but, admittedly, that's not much help if you can't find anyone who still has a clue how it works.

In the old days the police IT department would be running this and maintaining the system. These days they are just discussing support with an outside vendor and keeping the costs down.

If the backup system was destroyed in the Buncefield oil depot fire 1 (Dec.2005) then it's an impressive level of useless management to go 15 years without a replacement.

1 A series of massive explosions starting with several thousand cubic metres of petrol vapour which produced quite a large pressure wave.

There’s no suggestion in the article that the backup system hasn’t been replaced. And keeping the location of a backup system quiet is probably a good idea.

I suspect that they have the backups but they need to be done as a full restore and it's probable that it took a few days to notice and now they have more data added.

So now some poor individual is trying to work out if they do a delta on the data added, restore from backup and add the deltas which would mean having to halt data import at some point. The other option is to find some similar hardware, configure it so they can do a restore, work out the deltas and import it back into the PROD system. As the Irish saying goes, when asking someone in remote Ireland how to get to Dublin, "well, you don't want to be starting from here".

Either option is a world of pain and because of this the "leaders" will be avoiding making a decision, like they do. At some point when everyone has forgotten about it they will decide that it's too much work to do (as so much time has passed not making a decision) and will sack it off, hoping that the Official (covering our mistakes) Secret Act will stop people talking about it.

Just ask the CIA, the FBI or Homeland Security in the US. They still have it all and then some. Unless they were T***p supporters in which case you are SOL. /s

I suppose that if you are legally required to delete data, then you would be legally required to remove it from backups as well. (For example, my Mac has a command "delete from all backups", that you would use for things that you really, really, really don't want, and especially for things that you were legally not supposed to have.

If I had to design that system, I'd design it so users can remove something from live data, and at the same time enter a date in the future where the data has to be removed from backups, for legal reasons. So if you are wrong, you can restore the backup, if you're right it will be gone from the backup when you ask for it. And a very strongly worded message, e-mailed to you and your boss, if the "delete from backup" date is too close in the future.

Our backup system makes a complete system backup every night, and only keeps a weeks set of backups. The system administrator has access to the data and can restore anything if we need it. A separate backup system makes a complete backup every week and only keeps six weeks of backups but logs everything, a third backup system is inaccessible on the network but has access to the daily backup and purges itself once a year. In twenty years, two floods, multiple hurricanes, and a few disk failures we've never lost anything - the backups backups backups mean that nobody worries.

- And the vehicle belongs to you, does it sir?

- And your name is?

Right. Hold on a second.

Ready?

My name is: Derek'); DROP TABLE CriminalRecords; --

- What kind of name is that?