Surprise, surprise: AI cameras sold to schools in New York struggle with people of color and are full of false positives
(2020/12/07)
- Reference: 1607320874
- News link: https://www.theregister.co.uk/2020/12/07/in_brief_ai/
- Source link:
In brief A Canadian security company apparently lied to officials at New York’s Lockport City School District about the accuracy of its facial recognition cameras, when the technology was installed across schools last year.
Documents, [1]obtained by Vice, show that SN Technologies’ CEO KC Flynn claimed the algorithm, id3, running of its cameras had been vetted by the National Institute of Standards and Technology. It ranked 49th out of 139 in tests for racial bias, Flynn said. Although id3 was tested by NIST, a scientist denied it had tested one that matched Flynn’s description.
Schools believe computer vision systems can detect weapons and prevent shootings. But experts have repeatedly warned that false positives are more likely to discriminate against black students, painting them as suspected criminals when they’re not.
A report also showed that SN Technologies' software was worse at identifying black people than the company let on. It also mistook objects like broom handles for guns. Parents have [2]sued the New York State Education Department (NYSED) for approving facial recognition to be used at Lockport City Schools.
The US President urged the government to build trustworthy AI systems
Donald Trump signed an executive order this week, outlining nine principles that the US government will adhere to when designing and implementing AI technology.
It promised to uphold constitutional rights and laws to protect privacy and civil liberties, make sure the systems in place are accurate, transparent, understandable, and regularly monitored. Agencies deploying the software will be held accountable to ensure the principles are being enforced.
“Artificial intelligence (AI) promises to drive the growth of the United States economy and improve the quality of life of all Americans,” the order said. “Given the broad applicability of AI, nearly every agency and those served by those agencies can benefit from the appropriate use of AI…Agencies are encouraged to continue to use AI, when appropriate, to benefit the American people. The ongoing adoption and acceptance of AI will depend significantly on public trust.”
You can read the full document [3]here .
DeepMind is turning to python-based JAX
PyTorch is the favoured framework in the AI community. It has overtaken Google’s clunky and difficult to use TensorFlow, so the search giant decided to come up with something simpler: JAX.
Like PyTorch, JAX is also based on Python. And this week, DeepMind described how its researchers have been increasingly using it in their work. “We have found that JAX has enabled rapid experimentation with novel algorithms and architectures and it now underpins many of our recent publications,” it said.
It allows researchers to build and test their software more quickly, and has helped them develop all sorts of tools for training models, inspecting code, and creating AI agents in reinforcement learning experiments.
You can read about that more in detail [4]here .
MLCommons, a new benchmarking system for AI infrastructure
The team behind MLPerf, an industry effort that provides standard testing to benchmark machine learning hardware, have launched a new project known as MLCommons.
“Machine Learning is a young field that needs industry-wide shared infrastructure and understanding,” David Kanter, executive director of MLCommons, [5]said in a statement. “With our members, MLCommons is the first organization that focuses on collective engineering to build that infrastructure.”
“We are thrilled to launch the organization today to establish measurements, datasets, and development practices that will be essential for fairness and transparency across the community.”
It published People’s Speech, a giant public dataset containing more than 80,000 hours of speech samples, to test a machine’s ability to accurately transcribe speech to text. Companies selling such a tool over the cloud, for example, can enter the competition to find out which model is most accurate.
Whilst these benchmarking efforts are laudable, they’re only useful and impactful if as many companies take part as much as possible.
Machine learning software has gotten better at identifying faces covered by masks
Face masks are a common sight during the coronavirus pandemic. Covering up the bottom half of your mug, however, makes it difficult for facial recognition software to identify faces.
NIST examined the effects of mask wearing on the technology, earlier this year in July, and found that many vendors struggled with the same problems. The same tests have now been performed again, and this time round things have improved.
“Some newer algorithms from developers performed significantly better than their predecessors. In some cases, error rates decreased by as much as a factor of 10 between their pre- and post-COVID algorithms,” [6]said Mei Ngan, a NIST scientist. “In the best cases, software algorithms are making errors between 2.4 and 5 [per cent] of the time on masked faces, comparable to where the technology was in 2017 on nonmasked photos.”
NIST tested 152 different algorithms, and published the results in a report. Take them with a pinch of salt, however, since the test images used photographs of people with so-called “digital masks” pasted onto their faces rather than them wearing real cloth masks. ®
Get our [7]Tech Resources
[1] https://www.vice.com/en/article/qjpkmx/fac-recognition-company-lied-to-school-district-about-its-racist-tech
[2] https://www.nyclu.org/en/cases/shultz-et-al-v-new-york-state-education-department
[3] https://www.whitehouse.gov/presidential-actions/executive-order-promoting-use-trustworthy-artificial-intelligence-federal-government/
[4] https://deepmind.com/blog/article/using-jax-to-accelerate-our-research
[5] https://www.businesswire.com/news/home/20201203005641/en
[6] https://www.nist.gov/news-events/news/2020/12/face-recognition-software-shows-improvement-recognizing-masked-faces
[7] https://whitepapers.theregister.com/
Documents, [1]obtained by Vice, show that SN Technologies’ CEO KC Flynn claimed the algorithm, id3, running of its cameras had been vetted by the National Institute of Standards and Technology. It ranked 49th out of 139 in tests for racial bias, Flynn said. Although id3 was tested by NIST, a scientist denied it had tested one that matched Flynn’s description.
Schools believe computer vision systems can detect weapons and prevent shootings. But experts have repeatedly warned that false positives are more likely to discriminate against black students, painting them as suspected criminals when they’re not.
A report also showed that SN Technologies' software was worse at identifying black people than the company let on. It also mistook objects like broom handles for guns. Parents have [2]sued the New York State Education Department (NYSED) for approving facial recognition to be used at Lockport City Schools.
The US President urged the government to build trustworthy AI systems
Donald Trump signed an executive order this week, outlining nine principles that the US government will adhere to when designing and implementing AI technology.
It promised to uphold constitutional rights and laws to protect privacy and civil liberties, make sure the systems in place are accurate, transparent, understandable, and regularly monitored. Agencies deploying the software will be held accountable to ensure the principles are being enforced.
“Artificial intelligence (AI) promises to drive the growth of the United States economy and improve the quality of life of all Americans,” the order said. “Given the broad applicability of AI, nearly every agency and those served by those agencies can benefit from the appropriate use of AI…Agencies are encouraged to continue to use AI, when appropriate, to benefit the American people. The ongoing adoption and acceptance of AI will depend significantly on public trust.”
You can read the full document [3]here .
DeepMind is turning to python-based JAX
PyTorch is the favoured framework in the AI community. It has overtaken Google’s clunky and difficult to use TensorFlow, so the search giant decided to come up with something simpler: JAX.
Like PyTorch, JAX is also based on Python. And this week, DeepMind described how its researchers have been increasingly using it in their work. “We have found that JAX has enabled rapid experimentation with novel algorithms and architectures and it now underpins many of our recent publications,” it said.
It allows researchers to build and test their software more quickly, and has helped them develop all sorts of tools for training models, inspecting code, and creating AI agents in reinforcement learning experiments.
You can read about that more in detail [4]here .
MLCommons, a new benchmarking system for AI infrastructure
The team behind MLPerf, an industry effort that provides standard testing to benchmark machine learning hardware, have launched a new project known as MLCommons.
“Machine Learning is a young field that needs industry-wide shared infrastructure and understanding,” David Kanter, executive director of MLCommons, [5]said in a statement. “With our members, MLCommons is the first organization that focuses on collective engineering to build that infrastructure.”
“We are thrilled to launch the organization today to establish measurements, datasets, and development practices that will be essential for fairness and transparency across the community.”
It published People’s Speech, a giant public dataset containing more than 80,000 hours of speech samples, to test a machine’s ability to accurately transcribe speech to text. Companies selling such a tool over the cloud, for example, can enter the competition to find out which model is most accurate.
Whilst these benchmarking efforts are laudable, they’re only useful and impactful if as many companies take part as much as possible.
Machine learning software has gotten better at identifying faces covered by masks
Face masks are a common sight during the coronavirus pandemic. Covering up the bottom half of your mug, however, makes it difficult for facial recognition software to identify faces.
NIST examined the effects of mask wearing on the technology, earlier this year in July, and found that many vendors struggled with the same problems. The same tests have now been performed again, and this time round things have improved.
“Some newer algorithms from developers performed significantly better than their predecessors. In some cases, error rates decreased by as much as a factor of 10 between their pre- and post-COVID algorithms,” [6]said Mei Ngan, a NIST scientist. “In the best cases, software algorithms are making errors between 2.4 and 5 [per cent] of the time on masked faces, comparable to where the technology was in 2017 on nonmasked photos.”
NIST tested 152 different algorithms, and published the results in a report. Take them with a pinch of salt, however, since the test images used photographs of people with so-called “digital masks” pasted onto their faces rather than them wearing real cloth masks. ®
Get our [7]Tech Resources
[1] https://www.vice.com/en/article/qjpkmx/fac-recognition-company-lied-to-school-district-about-its-racist-tech
[2] https://www.nyclu.org/en/cases/shultz-et-al-v-new-york-state-education-department
[3] https://www.whitehouse.gov/presidential-actions/executive-order-promoting-use-trustworthy-artificial-intelligence-federal-government/
[4] https://deepmind.com/blog/article/using-jax-to-accelerate-our-research
[5] https://www.businesswire.com/news/home/20201203005641/en
[6] https://www.nist.gov/news-events/news/2020/12/face-recognition-software-shows-improvement-recognizing-masked-faces
[7] https://whitepapers.theregister.com/
Re: Public confidence?
sgp
Might as well leave out the start over.
Re: Public confidence?
Joe W
"[...] make sure the systems in place are accurate, transparent, understandable [...]"
Hahahahahaha.
Yeah. Right. Transparent. Understandable. AI.
Re: Public confidence?
BinkyTheHorse
The Dunning Kruger effect in this post is off the charts. You do realize machine learning is significantly older than deep learning? Or that most "machine learning algos" have a degree of interpretability that makes neural networks an outlier in this regard? Or that significant work has been put into making deep learning more debuggable and interpretable (the "dog in snow" paper is 4 years old FFS)?
This is essentially like wanting to scrap the entirety of software development after experiencing the multitude of shitty mobile apps. Don't blame the tool for the tools that use it wrong.
Eh?
IGotOut
Facial recognition stops school shootings?
There was me thinking it was mentally unstable kids having easy access to guns.
But hey no, it's they look eerrr like eerrr...ok how does it tell the difference between a pupil at the school going to class and the same pupil at the school ready to kill someone?
Re: Eh?
DwarfPants
must be detecting some sort of frown.
Take a broom handle to school
AMBxx
Easy way to defeat this - let all the kids know that broom handles are detected as guns. Next day, they all turn up with a broom handle. Broken system. Job done.
Or have children changed in the last 20 years?
Re: Take a broom handle to school
Denarius
Err, no. this is MerkinLand. Kids would probably get shot at door, you know, for sake of the children
Public confidence?
There IS none. AI is a load of bollocks in general, deep learning & machine learning algos are a bad joke, & facial recog is just another way for the Powers That Be to arrest you first & claim a reason for having done so after the fact.
The whole thing is a right clusterfuck that should be scrapped, restarted from scratch, & done correctly from the get-go.
Nobody should be allowed to use it until the false posative rate is below 10% at the worst of times & under 1% at the best. Those it flags should then be gone over with a judicial fine toothed comb to verify/reject if the person flagged is the person sought. "You're looking for a white male about 20 years of age, 1.98 meters tall, 80 kilos in weight, with a tattoo of a dog on his cheek. THIS is a 75 year old black man who's all of 1.25 meters tall, 50 kilos, & has no facial tattoos at all. On what planet does this make a match? Get out of my office before I have you arrested for being a fucking idiot."
*Cough*
Bin it, scrap it, start over.