Dallas cops lost 8TB of criminal case data during bungled migration, says the DA... four months later
- Reference: 1629119947
- News link: https://www.theregister.co.uk/2021/08/16/dallas_data_migration_8tb_deletion/
- Source link:
Dallas Police Department confessed to the information blunder last week, revealing in a statement that a data migration exercise carried out at the end of the 2020-21 financial year deleted vast amounts of data from a network drive.
"On August 6, 2021, the Dallas Police Department (DPD) and City of Dallas Information and Technology Services Department (ITS) informed the administration of this Office that in April 2021, the City discovered that multiple terabytes of DPD data had been deleted during a data migration of a DPD network drive," said a [1]statement [PDF] from the Dallas County prosecutor's office.
[2]
The migration, which took place between 31 March and 5 April, actually destroyed 22TB of data. 14TB were recovered, presumably from backups, but "approximately 8 Terabytes remain missing and are believed to be unrecoverable." Affected criminal case files include those created before 28 July 2020, though prosecutors said the precise number "is currently unknown."
[3]150,000 lost UK police records looking more like 400,000 as Home Office continues to blame 'human error'
[4]OVH says some customer data and configs can’t be recovered after fire, some seems to be OK, plenty is safe
[5]GitLab.com melts down after wrong directory deleted, backups fail
[6]The wrong guy: Backup outfit Spanning deleted my personal data, claims Cohesity field CTO
It added:
Effective today, all prosecutors have been instructed to verify with the filing detective that all evidence/files were shared with our office via TechShare before disposing of the case..... Should there be any missing files in a case, the prosecutor will make a written disclosure based upon the information communicated by DPD.
CBS Dallas Fort Worth, a local TV station, [7]reported that murder suspect Jonathan Pitts was due to stand trial on Thursday but has instead been released on bail because his files were deleted in the blunder. The detail was apparently revealed by the prosecutor in a motion filed last week, just a day before the trial had been due to begin. Case files typically contain documents, images, videos, logs of evidence, and more. Evidence (going either way) in Pitts' case may yet be recovered, so the trial is not necessarily off for good. Pitts had pleaded not guilty in the case.
District Attorney John Creuzot claimed that while police were immediately aware of what happened, it took them four months to come clean with his prosecutor's office. Meanwhile the local mayor, Eric Johnson, [8]said he was "blindsided" by the data loss.
[9]
Such blunders aren't unique to Dallas. Earlier this year Britain's Home Office [10]managed to lose 400,000 criminal evidence records from a Fujitsu-provided mainframe backup appliance. Meanwhile, in France, cloud operator OVH suffered an equally catastrophic data loss after a fire in March [11]gutted one of its data centres in Strasbourg . ®
Get our [12]Tech Resources
[1] https://regmedia.co.uk/2021/08/16/dallas_county_memo.pdf
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YRqLtTiGhmPLFCf@37TeUAAAAI8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.theregister.com/2021/01/18/lost_police_records_jump_to_400k/
[4] https://www.theregister.com/2021/03/15/ovh_restoration_roadmap/
[5] https://www.theregister.com/2017/02/01/gitlab_data_loss/
[6] https://www.theregister.com/2021/03/03/cohesity/
[7] https://dfw.cbslocal.com/2021/08/13/murder-suspect-to-be-released-data-lost-dallas-police/
[8] https://www.dallasnews.com/news/courts/2021/08/15/heres-what-to-know-about-the-recently-revealed-deletion-of-dallas-police-evidence-files/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YRqLtTiGhmPLFCf@37TeUAAAAI8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2021/01/18/lost_police_records_jump_to_400k/
[11] https://www.theregister.com/2021/03/15/ovh_restoration_roadmap/
[12] https://whitepapers.theregister.com/
Re: "data migration of a network drive caused [...] deletion"
This is DPD. The initial copy probably failed so they reverted to their default action plan and shot it. "This is what happens when you don't comply"
This is why we still use tape libraries.
I suspect it was someone who thought having copies = having a backup good enough. Problem is, if you are doing a migration, it's a copy-then-delete. If you don't delete the original, it's not a migration, it's adding capacity.
Sounds like DPD IT learned a valuable lesson on why offline backups are necessary. It's not just about protecting from bad guys, it's also about preventing showing up on a future edition of the "Who, Me?" column.
Re: This is why we still use tape libraries.
If you don't delete the original, it's not a migration, it's adding capacity.
Depends on the nature of the migration. I wouldn't expect a DB migration (eg. MySQL -> Postgress) to impose a deletion of source data.
But we can defo surmise that:
1. They didn't follow back up procedures prior to migrating (ie. full backup with verification);
2. They didn't verify the target platform before flattening the source;
3. They were negligant in reporting the clusterfuck within a reasonable timeframe.
Re: This is why we still use tape libraries.
In short, they followed standard operating procedures.
Re: This is why we still use tape libraries.
Agreed :) Where SOP != BestPractice
Re: This is why we still use tape libraries.
or 4: They assumed that 1-3 were good enough.
What I hinted at re: offline backups was that even if you do 1-3, and you test in a test environment, if your backups are not also copied offline, they're still vulnerable to "who, me?"
I've seen at least 4 examples in my IT career where companies thought having an off-site replication was a backup. Then either a hardware bug or a human error resulted in corruption or deletion of both online copies.
To put it through your steps 1-3:
1. Did we replicate the data (dallas3) to the new storage medium (dallas2)? Check.
2. Is the new copy online and did we run 100 tests to verify that it's online, performant, and fit for purpose? Yep, we're 100% certain the new system is perfect.
3...
Well, I _intended_ to sensibly type 'flatten Dallas2' and then wait for a second set of eyes to confirm my command, what actually happened was that my fat fingers bumped 3 and enter at the same time because they're right there together on the keyboard.
Whelp, there goes that data, and I just reported it to you, my boss, right away - so you could sit it for four months.
Re: This is why we still use tape libraries.
A backup is not a mirror site. The back up is performed. The backup is separated from the source then verified on a discrete system, then taken off line. (Put into a fireproof safe off-site and guarded by
The migration is performed and the target is verified against the source.
Only when the new system is shown to be working is the source even considered for zeroing. IMO I'd keep the source for as long as is practical (to the heat death of the universe).
Taking the backup off-line prevents the fat fingered cock-up, as you say "if your backups are not also copied offline" - so keeping the BU on-line is numptyville.
they're still vulnerable to "who, me?" <-- I like that :)
We need backup!
Sorry, that one was easy…
Re: We need backup!
Sorry, the budget we have is allocated to buy surveillance equipment. Priorities demand that we keep the people safe from each other first. That is why we need to listen, peek and poke into all corners physical and digital without you knowing. How we use the data and what we keep or save is not part of our department's budget and therefore not our concern. Please redirect your future inquiries to the department responsible.
Budget constraints can also hinder successful migrations from one system to the other.
Agreed. But surely a 60TB NAS is within budget for DPD. What's that, £4K tops?
Edit: obviously that assumes certain things about the platform, but still...
If you think any NAS device you can purchase for 4K is suitable for mission critical data availability that can literally mean life or death for a suspect, you know, you might have a future at DPD IT!
Edit: I know you weren't making a serious proposal, but I had to use your comment, it was just like you teed that straight line just for me.
Well that's just criminal!
You have the right to remain silent. Anything you say can be used against you in court. You have the right to talk to a lawyer for advice before we ask you any questions. You have the right to have a lawyer with you during questioning. If you cannot afford a lawyer, one will be appointed for you before any questioning if you wish. If you decide to answer questions now without a lawyer present, you have the right to stop answering at any time.
Including
" ... case files in a murder trial"
They need to check the names of some other suspects due for trial. Like a buddy of someone in IT.
It wouldn't be the [1]first time a convenient deletion occured.
[1] https://en.m.wikipedia.org/wiki/Nixon_White_House_tapes
Impressive
On the surface, it sounds like quite an impressive fuck-up.
However, we don't know the total size of the data migrated. We hear of 22TB lost, with 14 recovered. Does that mean that 22TB was the total of the data being migrated (in which case 100% fuck-up - well done!), or was it a small part of the total volume (1% fuck-up would look a lot better)?
"data migration of a network drive caused [...] deletion"
It's a little known fact that copying data rarely destroys it. Maybe someone should have told them the difference betwen copying and moving.