Facial recognition should be used to identify potential matches.

Humans should then be responsible for making decisions.

I understand what you are trying to say but have to disagree as this what exactly what happened here.

An individual was identified by the AI system as a potential known forbidden character, the person responsible that day took the decision to accept the AI detection status and forbade entry. The human element was definitely there, the computer system did not stop the person coming in, a human meatbag did.

The AI System night have a problem distinguishing objects that have specific colors, or lack of contrasts etc but shouldn't that be considered as an algorithmic problem rather than a racial problem ?

He accepted the AI's decision, but he did not control the result.

So he just made himself part of the robot. No intelligence included.

This is a quite common phenomenon, when a device is being used to absolve controller from responsibility.

AI is the new dog

In the past, the police used dogs to justify searching someone, when in reality prejudiced officer would gave dog a command to act as if the suspect raised a suspicion.

Using the same analogy, the operators of such systems can tweak them in a way to underpin any door policy they want, where in case of absence of such device, the policy would unlikely be legal. It was just a glitch... We got hacked... and so on...

When such systems are being used, human should always be solely responsible for the decision and the fact that computer gave this and that suggestion should be irrelevant.

I think Khaptain has nailed it - a flaw in the system (be it inadequate software / hardware / etc) but not racial profiling...

The problem with "inadequate software / hardware / etc" is that it doesn't give someone a reason to lawyer up and make an easy few quid...

It is both an algorithmic problem and a racial problem.

He is not saying what happen here but what should happen here.

Humans should then be responsible for making decisions.

After (re)viewing the evidence (ie images) that the AI matched to the person. This might be hard for a human if the image quality is bad or the person has different make-up or ...

What happens if AI is not being used and a venue owner thinks that s/he recognises someone who caused problems some time back ? The owner may well be confusing an innocent person with someone else - this has likely happened many times.

Humans are also responsible for the development of the technology.

Which just goes to show how much you can rely on human "responsibility".

The problem is one where the AI has been trained to have ingrained racism of the "they all look the same" type due to the training dataset and input data quality.

It also relies on cameras, that have also been shown to exhibit ingrained racist assumptions in the design - the cameras pick up white/pale skin traits quite well, but not darker skin where they are not very sensitive.

So a poor image + poor AI makes a decision, then shows the photos (on a likely poorly calibrated monitor that is also poor at showing dark skin tones) to an employee, who themselves may have their own ingrained racist opinions (or their employer will), and this will fix the previous two issues?

"It also relies on cameras, that have also been shown to exhibit ingrained racist assumptions in the "

design"

That's an extremely serious assumption to make, can you please provide evidence of what you state ?

I'm not too sure what evidence they have, but if you look at racism as simply being "treating differently based on physical appearance", then the software most certainly does.

1. Can racism extended beyond the human element?

2. Is fixing the software based on race, not racism?

I feel if AI facial recognition is to ever have a chance, then it can't depend on cameras that depend on light. Which technology will finally nail it... I'm not sure, but I have a feeling that tech. will be much more costly (I'm envisioning a mass array of lasers throughout the walls, or something crazy like that).

I think we can all understand the point you're trying to make here, but that statement is a bit of a stretch.

There's no racist assumptions in the design, it's just that dark anything reflects significantly less light than pale anything . Elementary physics. Cameras either need a longer exposure to properly image dark things, which will affect image quality in other ways, or need higher gain which can lead to image noise and overexposure of brighter image regions.

There's no easy answer. It's not like this is an already-solved problem whose solution is being deliberately ignored because racism.

Physics isn't racist, it's simply physics.

Yes, but when setting their exposure parameters and so on, they tune them to work best on white people.

They could tune them to work best on black people, and white people would appear as an over-exposed white blob on the image.

And they could use AI to detect the skin tone of the person being looked at and set the exposure parameters accordingly...

There's no racist assumptions in the design, it's just that dark anything reflects significantly less light than pale anything.

That's just an excuse. If the system cannot treat different races with equal quality of results, then by definition it is racist.

That's just an excuse. If the system cannot treat different races with equal quality of results, then by definition it is racist."

That's a seriously tainted point of view to hold.... ? It very sad.

worth suing, metaphorically, because they feel something's seriously fucked up with this society, or literally, calculating the investment to hire a lawyer and potential return, which proves there's something seriously fucked up with this society?

And how much damage to their life, job, housing etc when the other lawyer has access to any previous police records (thanks to some friends on the force) and the ice rink hires a PR comp to make sure this all gets out on Fox news and social medo

Live in fear if you want; given the current Black Lives Matter climate, a lawyer/PR comp who does this is being particularly stupid and deserves to have their life, job, housing etc. damaged.

“The software had her daughter at a 97 percent match. This is what we looked at ... if there was a mistake, we apologize for that."

Again, not actually accepting there was a mistake, that a young girl has been humiliated and denied an enjoyable experience with her friends because of a faulty system. I do hope the parents get appropriate help deciding whether or to to pursue this legally.

Sometime there has to be a legally required standard for facial recognition systems to meet. Claiming that the AI system said it was a 97% match is meaningless without context. What was the AI system trained on? I bet (and this is extremely racist) that if you trained an AI facial recognition system on blond, blue-eyed fair skinned boys plus one baboon, and then showed it a picture of any black person the match would be with the baboon. Conversely, train one on a load of black and asian people and one polar bear then show it a picture of a white person with white hair and it would match with the bear. Unless these systems are trained on an appropriate demographic they should be banned.

OK rant over

Entirely this. Facial Recognition is known for having problems distinguishing between non whites. I don't know why - I don't know if anyone does - but, knowing the system is badly flawed, yet continuing to use it, is inherently discriminatory against anyone who ain't white.

Again, not actually accepting there was a mistake...

Well that's hardly surprising. As the parents are considering suing it would be incredibly unwise of the business to make a statement implicating themselves.

Or, they could consider their reputation as an entertainment venue in that community and just apologise.

Apologies will follow the lawsuit, should they be found guilty.

"Sometime there has to be a legally required standard for facial recognition systems to meet."

No, we approach that in a different way. It's already illegal to act solely on the basis of a flawed (discriminatory) algorithm.

The problem is the users completely misunderstanding what the tech is saying. It isn't a 97pc chance of a match, it's a 97pc chance that a human should look and decide if this is the same person as in whatever stored image they're comparing to.

Yes, it is a major design flaw that the system tells the operator its assessment of the match. It should just display the top three matches from its database, with no further comment, for every visitor and let a human judge if any of them are the same.

The flawed design is so obvious that the creators of the machine should probably be prosecuted for racism.

No it is not.

But that's what you get in a society that accepts flawed results from a technology that heas been proven time and time again to not be reliable.

I think the big question here is - why does a roller-skating rink have facial-recognition AI? Are they actually just a cover for an underground nuclear missile silo? Or maybe, behind the pinball machines, is the entrance to the Pentagon's Michigan CnC bunker? Or perhaps the Navy keeps its most highly-classified results from the rail-gun trials in locker number 76. I mean, it's a roller-skating rink, not a national security site. Seems a bit paranoid to me (and I'm a bit paranoid myself).

Because unlike your corner store, a sign saying "no more than 3 black teens" would be illegal. However 'computer says no' is a company policy and out of my hands.

Ironically to stop employees getting sued if they looked at photos and said you look like her, you're banned

I suspect because they decided that

a) it's cheaper than to hire one or more humans

b) so it's a smart-future-proof move (somebody told them, let me guess who it might have been)

c) "everybody's doing this!", and lookie here, 97%, can not go wrong!

The low tech way to do it is to have pictures of troublemakers, make your staff try to recognise them, and act on that basis.

Here, the computer system is supposed to be doing the first step of 'that looks a bit like this, we should check if they're the same'.

Turning someone away unless you're sure they're the troublemaker in question is very odd. Normally if you aren't certain you'd let someone in and keep a closer than normal eye on them to start with.

Looking at the pictures in the linked story - the girls to look pretty similar, so I could see the same mistake being made without AI

(However, I'm white, and we're all notoriously poorer at differentiating people of different races than our own)

They look completely different to me.

Also white, and a much paler shade of white than the icon.

>Looking at the pictures in the linked story - the girls to look pretty similar

Similar in the sense they are both dark skinned and wearing glasses?

I hope this does go to court, but it needs a decent lawyer and legal team to force the vendor of the facial recognition system to attend and explain in full detail how their system arrived at the 97% match figure, ie. strip away the AI cloak mysticism.

Hmmm

Both lit well enough.

Both wearing glasses, totally different designs.

Victim girl has a thinner face.

10 second look at not very good photo definately not same person.

1) Badly trained AI

2) Idiot staff not checking and seeing two completely different girls.

As similar as the two middle aged shaved head due to bald ex ginger chaps I know.

AI everywhere, Justice nowhere

AI - Artificial Ignorance.

I'm pretty sure there are more people mistakenly turned away every single day by humans than by computers. I get it: This sucks. But it's something that happened since the first bouncer stood outside the first tavern.

So, in order for this all singing all dancing wonderful AI technology (that is going to make all our lives so much better), to work - we need to re-introduce segregation!!! Yeah - go human advancement!

We can at least be grateful that the cameras were not, as yet, attached to mini-gun auto turrets - that will perhaps be a future update!

It's all very well embracing 'big data', but what you end up with is 'data quality' issues at 'big' scale.

I'm not a fan of AI/ML or facial recognition (I'm a people-person and in my 50s)...I should think the the false positives generated outweigh the benefits of the bloody stuff. I suppose it has it's uses, but until it gets a *whole lot better* there has to be human oversight to exert a degree of common sense on it's decisions.

97% Match....just means a 97% match by a bad algorithm on crappy data. I can understand the staff apologizing, but the people who need 'outing' as guilty are the numpties who made the system in the first place. I don't know the details so can't really judge, but the phrase 'not for for purpose' springs to mind.

Me and my mate holidayed with his elderly relative in California in the '80s. She'd moved there from Canada after leaving Scotland. She took us to the largest shopping mall we'd ever seen that had the largest ice rink we'd ever seen in it. There was only one person on it, an angelic seven-ish hispanic lass. She was captivatingly talented and me and my mate were close to tears watching her. At that point the old Scottish woman turned up, glanced at the child and snorted, "Bloody immigrants." No sense of irony or self-awareness.

She kept on trying to take us to her Scottish highland dancing club. We went once and were appalled at their racism. In California 'Scottish' or 'Irish' is shorthand for whites-only. They considered us 'bad Scots', and we didn't consider them Scots at all. Scottishness, it's more than a porridge thing.