Cops responding to ShotSpotter's AI alerts rarely find evidence of gun crime, says Chicago watchdog
- Reference: 1629931131
- News link: https://www.theregister.co.uk/2021/08/25/shotspotter_chicago_report/
- Source link:
The California biz uses machine-learning algorithms to determine whether loud bangs caught by microphones deployed across more than 100 US cities are gunshots or not. If a shot is identified, the location of the noise is triangulated and sent to the police as an immediate, real-time alert, and reports are later compiled for prosecutors for use in court cases.
ShotSpotter is under the microscope right now because a 65-year-old man spent almost a year behind bars awaiting trial for murder – and the primary evidence against him was a disputed ShotSpotter report of a gunshot.
[1]
Michael Williams was last year accused of shooting and killing 25-year-old Safarian Herring, and denied any wrongdoing. Prosecutors said ShotSpotter picked up the sound of gunfire right where and when Williams was seen in his car, in Chicago, giving Herring a ride. Williams said Herring was hit in a drive-by shooting.
[2]
[3]
Crucially, Williams' lawyers asked the trial judge to probe the ShotSpotter evidence after it emerged the AI software actually picked up a firework a mile away and this information was later revised by ShotSpotter staff. In response, the prosecution withdrew the ShotSpotter report, and last month asked for the case to be dismissed as it no longer had sufficient evidence. The judge [4]agreed , and Williams was released as a free man.
Don't worry, someone did a probe
The City of Chicago's Office of Inspector General (OIG) decided to dig into the alerts ShotSpotter sends to the Chicago Police Dept (CPD) and the city's Office of Emergency Management. The city had a $33m three-year contract with ShotSpotter that was due to run out in August 2021, and in December last year, that contract was extended ahead of its expiry to mid-2023.
The watchdog said 50,176 alerts generated by ShotSpotter in Chicago between January 2020 and May this year probably were the result of gunfire, and were assigned unique IDs and had officers show up. Of those alerts, 41,830 resulted in some kind of police action, known as a disposition. And of those dispositions, only 4,556 indicated that "evidence of a gun-related criminal offense was found," the auditors said.
Thus, only 9.1 per cent of ShotSpotter alerts led to the police finding evidence of an actual gun crime. This did not go down well with the auditors, given that the ShotSpotter contract has been renewed.
[5]
“The CPD data examined by OIG does not support a conclusion that ShotSpotter is an effective tool in developing evidence of gun-related crime,” the watchdog [6]said this week.
[7]A man spent a year in jail on a murder charge that hinged on disputed AI evidence. Now the case has been dropped
[8]British Army develops AI shotgun drone with machine vision for indoor use
[9]Apple's bright idea for CSAM scanning could start 'persecution on a global basis' – 90+ civil rights groups
[10]Cop drone crashes into flight instructor's airplane
The police, meanwhile, defended their use of the service.
“In order to reduce gun violence, knowing where it occurs is crucial,” a Chicago police spokesperson told The Register . “ShotSpotter has detected hundreds of shootings that would have otherwise gone unreported. ShotSpotter is among a host of tools used by the CPD to keep the public safe and ultimately save lives.
ShotSpotter has detected hundreds of shootings that would have otherwise gone unreported
“Using ShotSpotter, CPD receives real-time alerts of detected gunfire enabling patrol officers to arrive at a precise location of a shooting event quickly. Instead of relying on the historically low rate of 911 calls, law enforcement can respond more quickly to locate and aid victims, identify witnesses, and collect forensic evidence.
"The system gives police the opportunity to reassure communities that law enforcement is there to serve and protect them and helps to build bridges with residents who wish to remain anonymous.”
Chicago’s OIG isn’t convinced. In a 30-page report
[11]PDF
, the watchdog said the technology may lead to over-policing by sending officers into communities in search of serious crimes that never happened.“Our study of ShotSpotter data is not about technological accuracy, it’s about operational value,” Deputy Inspector General for Public Safety Deborah Witzburg stressed.
[12]
“If the Department is to continue to invest in technology which sends CPD members into potentially dangerous situations with little information – and about which there are important community concerns – it should be able to demonstrate the benefit of its use in combatting violent crime.
This technology is changing the way CPD members interact with members of Chicago’s communities
“The data we analyzed plainly doesn’t do that. Meanwhile, the very presence of this technology is changing the way CPD members interact with members of Chicago’s communities.”
ShotSpotter’s software and hardware is proprietary, and hasn’t been openly audited for its accuracy. The company says its algorithms are 97 per cent accurate. “It is important to point out that the CPD continually describes ShotSpotter as an important part of their operations,” a ShotSpotter spokesperson told The Register .
“The OIG report does not negatively reflect on ShotSpotter’s accuracy which has been [13]independently audited at 97 percent based on feedback from more than 120 customers. Nor does the OIG propose that ShotSpotter alerts are not indicative of actual gunfire whether or not physical evidence is recovered.”
The Register has asked Chicago’s OIG for further comment. ®
Get our [14]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YScR95-g3mp08uefu7APyQAAAIY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YScR95-g3mp08uefu7APyQAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YScR95-g3mp08uefu7APyQAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2021/08/22/in_brief_ai/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YScR95-g3mp08uefu7APyQAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://igchicago.org/2021/08/24/oig-finds-that-shotspotter-alerts-rarely-lead-to-evidence-of-a-gun-related-crime-and-that-presence-of-the-technology-changes-police-behavior/
[7] https://www.theregister.com/2021/08/22/in_brief_ai/
[8] https://www.theregister.com/2020/09/29/british_army_shotgun_drone_ai/
[9] https://www.theregister.com/2021/08/19/apple_csam_condemned/
[10] https://www.theregister.com/2021/08/23/canadian_drone_strike/
[11] https://igchicago.org/wp-content/uploads/2021/08/Chicago-Police-Departments-Use-of-ShotSpotter-Technology.pdf
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YScR95-g3mp08uefu7APyQAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://edgeworthanalytics.com/independent-audit-of-the-shotspotter-accuracy/
[14] https://whitepapers.theregister.com/
Re: And we're surprised?
There's also the constitutional element. In Canada, where there is something very similar to the 4th amendment, the police rejected ShotSpotter as being a clear example of unlawful search. I wonder if it would survive a constitutional challenge in the US?
Feels like a hammer meet screw sort of situation.
Would be interesting to know if that 97% accuracy includes staff (and not the software) 'screening' recordings after talking with the police and overriding or reclassifying the results.
"We're 97% accurate when we're allowed to change our results to the correct ones 97% of the time"
Compensation
I'm hoping the guy that lost almost a year of his life sues them and takes a goodly chunk out of that 33 megabucks.
Irreconcilable differences
The manufactures claim of 97% accuracy is clearly facile and inaccurate. It is also glaringly self serving.
The proof of this is the uninterrupted string of negative results in the real world. Much like overhyped facial recognition systems and pre-crime behavioral analysis snake oil, the Shotspotter can't do what it claims. All it does is semi successfully is detect loud noises. Worse, like many other recent cases, there is a pattern of them fitting the data to the intended outcome. This is similar to problems with other forensic evidence(like bite mark "experts") whose methods are both flawed and biased. The police are buying these tools because they can use them as an excuse to continue patterns of abuse and policies that have been increasingly scrutinized and restricted.
The fact the Shotspotter is a black box that is shielded from further scrutiny makes it all the more attractive.
A system with a >90% failure rate isn't trustworthy, and there is a clear pattern of bias in the companies results. They favor results that match what the company thinks that law enforcement wants to hear. Like voice stress analyzers and so many other bogus devices law enforcement waste our money on, this device is a scam, pure and simple.
Don't expect this problem to fix itself either, push your representatives to bar the company from use by state and federal agencies. This company ruins innocent peoples lives as a service, and needs to be shut down.
Re: Irreconcilable differences
The problem will fix itself when criminals notice that playing gunshot sounds into these microphones is an effective denial of service attack against the police.
This was discussed in the shotspotter thread a few days ago, but the company guarantees 80% or your money back. I would really like to see Chicago call them on it. I'm sure they'll try to slime their way out of it and claim that it was 80% accurate but the cops were just bad at getting there in time or some similar nonsense. Doubt it'll work though.
Only 10% effective?
I'm sure they do better just waiting for someone to call the cops saying "I think a heard a gunshot". Even if half the time it doesn't get reported because people are numb to it in certain neighborhoods, sometimes turns out to be fireworks, or they can't pinpoint the direction they still are going to beat 10%!
And we're surprised?
I'm not.
And as for the response from Shotspotter:
“The OIG report does not negatively reflect on ShotSpotter’s accuracy which has been independently audited at 97 percent based on feedback from more than 120 customers."
What that tells me is that 120 customers don't want their purchase questioned.
Not present in the survey, I assume, is the original customer, the Department of Defense, which found the software worthless and terminated the contract.