Scottish cops dangle £6m for help understanding 160TB treasure trove of structured and unstructured data
- Reference: 1591780504
- News link: https://www.theregister.co.uk/2020/06/10/scottish_police_authority_6m_tender/
- Source link:
The authority said it stores around 100TB of unstructured data alongside a further 60TB of structured data that it wants to make better use of in order to catch baddies and work more efficiently.
In a [1]tender notice published this week , the authority requests bids from software and services suppliers to create a virtual analytics platform, acquire a metadata management product and buy a data discovery solution for compliance purposes.
The tender notice asks for a "suitable data virtualisation software solution into the Police Scotland in-house ICT environment to allow a single view of all available data to all authorised users."
Once installed in-house, the plan is to migrate the system to the cloud in the fullness of time. The tender requests a "discretionary transition to cloud model during contract term."
"The solution shall be deployed, initially, on-premise, but deployable, in part or in whole, on a private or public hosted cloud environment at a later date," the tender says.
The authority will also address metadata in the contract. It is requesting an "intelligent metadata layer, constructed, inferred, enriched" to provide "deep insight into the service's rapidly growing data sets as well as maximise the use and value of its data."
The notice adds that given the scale of the metadata problem, "automation and collaboration tools are imperative." Metadata should cover governance, risk, and compliance.
"Metadata will provide a means of understanding the breadth and type of information available across the service, its quality, and relevance thereby enabling self-service by providing the right data context for users," it said.
Lastly, the Scottish Police Authority is looking for a compliance solution for GDPR and "other key legislation."
The tender is divided into three lots related to each strand of activity. The initial contract will be let for four years with the option to extend the contract for a further three periods of up to a year each.
The authority may have grand plans for its data strategy but the precedent is not good. [2]According to its 2018/2019 audit report [PDF] , the business case for better IT and data support up to 2026 estimated that the necessary investment was £298m over nine years. But funding for the strategy has not been agreed with the Scottish government.
Meanwhile, the audit notes the authority's IT projects have been dogged by delays. "Limited funding is not the only factor contributing to the delays, as the timescales associated with procurement and the governance of business cases have also had an impact on project plans." ®
Sponsored: [3]Taking the Cloud-Native Approach with Microservices
[1] https://ted.europa.eu/udl?uri=TED:NOTICE:268559-2020:TEXT:EN:HTML&src=0
[2] https://www.audit-scotland.gov.uk/uploads/docs/report/2019/s22_191217_spa.pdf
[3] https://go.theregister.com/tl/1956/-8470/taking-the-cloud-native-approach-with-microservices?td=wptl1956
My advice:
1) sit on it, it might hatch
2) collect more data
a single rack
not exactly a data lake if your requirements is a single rack of storage and CPU even if data doubles this is not high amounts of data infact do they not have camera's in those police cars ?
Re: a single rack
They do, but that video is deleted after 30 days. Or accidentally deleted sooner if someone requests it showing the police doing something wrong. The number of hard disk errors on police computers is a weird and inexpiable anomaly,
Most probably junk
report1.docx
report1_1.docx
report1_11.docx
report1_111.docx
and so on from all the documents that they were trying to save that had the same name. Or they did a minor update to it.
Doesn't this just confirm everybody's suspicions that these government bodies collect mounds of data for the sake of it, without any plan of how to make meaningful sense of it?
Their policy seems to be "Record everything and we'll figure out what to do with it later"
160TB of data, and % that is unlawful, yet Police ignore UK and at the time EU rules and official guidelines?