Insurance startup backtracks on running videos of claimants through AI lie detector
(2021/05/27)
- Reference: 1622071388
- News link: https://www.theregister.co.uk/2021/05/26/ai_insurance_lemonade/
- Source link:
Updated An insurance biz has retracted boasts of how it uses AI algorithms to study videos of customers for “non-verbal cues” that their claims are fraudulent. The marketing U-turn came after the ethics of this approach was called publicly and loudly into question.
Using machine-learning software to automate decision-making processes to decide whether to accept or deny customers credit or insurance payments is particularly sensitive. Last month, America's consumer watchdog, the FTC, issued a strongly worded [1]statement warning that it was illegal to deploy algorithms end up discriminating against people based on their race, color, religion, national origin, sex, marital status, and age when making financial-related decisions.
Alarm bells were set off when Lemonade, a company based in New York, admitted it built software that scanned videos of customers explaining the situations they found themselves, which were submitted as part of insurance claims, to decide whether those people were essentially lying or committing some other fraudulent.
[2]
[3]
[4]
Lemonade prides itself on providing an easier and simpler way for people to file pet, home, and life insurance claims. Customers speak to a chat bot, submit their claim, and a decision on how much it should pay them can be made in a few minutes.
“When a user files a claim, they record a video on their phone and explain what happened. Our AI carefully analyzes these videos for signs of fraud. It can pick up non-verbal cues that traditional insurers can't, since they don’t use a digital claims process,” Lemonade [5]stated in a series of tweets that have since been deleted.
Netizens criticized Lemonade's technology, accusing it of being potentially biased and reliant on flimsy sentiment and emotion analysis. The backlash on Twitter prompted the company to delete its posts and issue a new statement, where it claimed it just used facial recognition algorithms to make sure the same person wasn’t making multiple claims.
[6]
“There was a sizable discussion on Twitter around a poorly worded tweet of ours (mostly the term ‘non-verbal cues’) which led to confusion as to how we use customer videos to process claims,” the upstart [7]stated on its website. “There were also questions about whether we use approaches like emotion recognition (we don’t), and whether AI is used to automatically decline claims (never!)”
“We do not use, and we’re not trying to build, AI that uses physical or personal features to deny claims,” it reiterated.
Our systems don’t evaluate claims based on background, gender, appearance, skin tone, disability, or any physical characteristic (nor do we evaluate any of these by proxy) (3/4) — Lemonade (@Lemonade_Inc) [8]May 26, 2021
That said, the company’s [9]privacy policy does say it collects, among other details, people's physical characteristics when handling life insurance.
[10]Banks across America test facial recognition cameras 'to spy on staff, customers'
[11]Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves
[12]Another reminder that bias, testing, diversity is needed in machine learning: Twitter's image-crop AI may favor white men, women's chests
[13]Enjoy a tipple or five? You might need this AI system to tell you when it's time for a new liver
And in a S-1 filing to America's financial regulator, the SEC, Lemonade said its system collects roughly 1,700 “data points” from customers.
“We use technology and artificial intelligence to reduce hassle, time, and cost associated with purchasing insurance and the claims submission and fulfillment process,” Lemonade [14]stated in its SEC submission. "We built our entire company on a unified, proprietary, state-of-the-art technology platform. Our customers are able to purchase insurance on our website or through our app, generally in a matter of minutes. Our artificial intelligence system handles substantially all of our customer onboarding and a meaningful portion of our claims."
What those data points describe is unclear. It went on to admit its own technology could have unintended consequences, in customers may be paid too much or too little, leading to biased and discriminatory decisions. On the one hand, this is a boilerplate warning to investors and the financial markets that the biz could go belly up, and thus investments could be lost, though on the other hand, it is pretty specific about how it could go wrong.
[15]
“Our proprietary artificial intelligence algorithms may not operate properly or as we expect them to, which could cause us to write policies we should not write, price those policies inappropriately or overpay claims that are made by our customers. Moreover, our proprietary artificial intelligence algorithms may lead to unintentional bias and discrimination.”
It added:
A determination by federal or state regulators that the data points we collect and the process we use for collecting this data unfairly discriminates against some groups of people could also subject us to fines and other sanctions, including, but not limited to, disciplinary action, revocation and suspension of licenses, and withdrawal of product forms.
Any such event could, in turn, materially and adversely affect our business, financial condition, results of operations and prospects, and make it harder for us to be profitable over time. Although we have implemented policies and procedures into our business operations that we feel are appropriately calibrated to our artificial intelligence and automation-driven operations, these policies and procedures may prove inadequate to manage our use of this nascent technology, resulting in a greater likelihood of inadvertent legal or compliance failures.
The company was launched in 2016, and operates across the US and parts of Europe, including France, Germany, and the Netherlands. It has yet to turn a profit, and spends most of its money on sales and marketing.
“Our future success depends on our ability to continue to develop and implement our proprietary artificial intelligence algorithms, and to maintain the confidentiality of this technology,” it said in its filing.
The Register has asked Lemonade for further comment. ®
Updated to add
We've noticed that even though Lemonade said its software is "never" used to "automatically decline claims," in its the S-1 filing, it boasted that one of its machine-learning systems, as of March 2020, could handle a customer's "entire claim through resolution in approximately a third of cases, paying the claimant or declining the claim without human intervention."
Despite that boast in the SEC submission, a spokesperson for the upstart insisted to The Register : "The term 'non-verbal cues' was a bad choice of words to describe the facial recognition technology we’re using to flag claims submitted by the same person under different identities. These flagged claims then get reviewed by our human investigators. We have never, and will never, let AI auto-reject claims.”
Get our [16]Tech Resources
[1] https://www.ftc.gov/news-events/blogs/business-blog/2021/04/aiming-truth-fairness-equity-your-companys-use-ai
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://archive.ph/i5m4v
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.lemonade.com/blog/lemonades-claim-automation/
[8] https://twitter.com/Lemonade_Inc/status/1397564445648424965?ref_src=twsrc%5Etfw
[9] https://www.lemonade.com/privacy-policy
[10] https://www.theregister.com/2021/04/24/in_brief_ai/
[11] https://www.theregister.com/2020/10/28/gpt3_medical_chatbot_experiment/
[12] https://www.theregister.com/2020/09/21/twitter_image_cropping_ai/
[13] https://www.theregister.com/2019/11/08/ai_liver_transplant/
[14] https://www.sec.gov/Archives/edgar/data/1691421/000104746920003846/a2241899zs-1a.htm#ds40510_legal_matters
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[16] https://whitepapers.theregister.com/
Using machine-learning software to automate decision-making processes to decide whether to accept or deny customers credit or insurance payments is particularly sensitive. Last month, America's consumer watchdog, the FTC, issued a strongly worded [1]statement warning that it was illegal to deploy algorithms end up discriminating against people based on their race, color, religion, national origin, sex, marital status, and age when making financial-related decisions.
Alarm bells were set off when Lemonade, a company based in New York, admitted it built software that scanned videos of customers explaining the situations they found themselves, which were submitted as part of insurance claims, to decide whether those people were essentially lying or committing some other fraudulent.
[2]
[3]
[4]
Lemonade prides itself on providing an easier and simpler way for people to file pet, home, and life insurance claims. Customers speak to a chat bot, submit their claim, and a decision on how much it should pay them can be made in a few minutes.
“When a user files a claim, they record a video on their phone and explain what happened. Our AI carefully analyzes these videos for signs of fraud. It can pick up non-verbal cues that traditional insurers can't, since they don’t use a digital claims process,” Lemonade [5]stated in a series of tweets that have since been deleted.
Netizens criticized Lemonade's technology, accusing it of being potentially biased and reliant on flimsy sentiment and emotion analysis. The backlash on Twitter prompted the company to delete its posts and issue a new statement, where it claimed it just used facial recognition algorithms to make sure the same person wasn’t making multiple claims.
[6]
“There was a sizable discussion on Twitter around a poorly worded tweet of ours (mostly the term ‘non-verbal cues’) which led to confusion as to how we use customer videos to process claims,” the upstart [7]stated on its website. “There were also questions about whether we use approaches like emotion recognition (we don’t), and whether AI is used to automatically decline claims (never!)”
“We do not use, and we’re not trying to build, AI that uses physical or personal features to deny claims,” it reiterated.
Our systems don’t evaluate claims based on background, gender, appearance, skin tone, disability, or any physical characteristic (nor do we evaluate any of these by proxy) (3/4) — Lemonade (@Lemonade_Inc) [8]May 26, 2021
That said, the company’s [9]privacy policy does say it collects, among other details, people's physical characteristics when handling life insurance.
[10]Banks across America test facial recognition cameras 'to spy on staff, customers'
[11]Researchers made an OpenAI GPT-3 medical chatbot as an experiment. It told a mock patient to kill themselves
[12]Another reminder that bias, testing, diversity is needed in machine learning: Twitter's image-crop AI may favor white men, women's chests
[13]Enjoy a tipple or five? You might need this AI system to tell you when it's time for a new liver
And in a S-1 filing to America's financial regulator, the SEC, Lemonade said its system collects roughly 1,700 “data points” from customers.
“We use technology and artificial intelligence to reduce hassle, time, and cost associated with purchasing insurance and the claims submission and fulfillment process,” Lemonade [14]stated in its SEC submission. "We built our entire company on a unified, proprietary, state-of-the-art technology platform. Our customers are able to purchase insurance on our website or through our app, generally in a matter of minutes. Our artificial intelligence system handles substantially all of our customer onboarding and a meaningful portion of our claims."
What those data points describe is unclear. It went on to admit its own technology could have unintended consequences, in customers may be paid too much or too little, leading to biased and discriminatory decisions. On the one hand, this is a boilerplate warning to investors and the financial markets that the biz could go belly up, and thus investments could be lost, though on the other hand, it is pretty specific about how it could go wrong.
[15]
“Our proprietary artificial intelligence algorithms may not operate properly or as we expect them to, which could cause us to write policies we should not write, price those policies inappropriately or overpay claims that are made by our customers. Moreover, our proprietary artificial intelligence algorithms may lead to unintentional bias and discrimination.”
It added:
A determination by federal or state regulators that the data points we collect and the process we use for collecting this data unfairly discriminates against some groups of people could also subject us to fines and other sanctions, including, but not limited to, disciplinary action, revocation and suspension of licenses, and withdrawal of product forms.
Any such event could, in turn, materially and adversely affect our business, financial condition, results of operations and prospects, and make it harder for us to be profitable over time. Although we have implemented policies and procedures into our business operations that we feel are appropriately calibrated to our artificial intelligence and automation-driven operations, these policies and procedures may prove inadequate to manage our use of this nascent technology, resulting in a greater likelihood of inadvertent legal or compliance failures.
The company was launched in 2016, and operates across the US and parts of Europe, including France, Germany, and the Netherlands. It has yet to turn a profit, and spends most of its money on sales and marketing.
“Our future success depends on our ability to continue to develop and implement our proprietary artificial intelligence algorithms, and to maintain the confidentiality of this technology,” it said in its filing.
The Register has asked Lemonade for further comment. ®
Updated to add
We've noticed that even though Lemonade said its software is "never" used to "automatically decline claims," in its the S-1 filing, it boasted that one of its machine-learning systems, as of March 2020, could handle a customer's "entire claim through resolution in approximately a third of cases, paying the claimant or declining the claim without human intervention."
Despite that boast in the SEC submission, a spokesperson for the upstart insisted to The Register : "The term 'non-verbal cues' was a bad choice of words to describe the facial recognition technology we’re using to flag claims submitted by the same person under different identities. These flagged claims then get reviewed by our human investigators. We have never, and will never, let AI auto-reject claims.”
Get our [16]Tech Resources
[1] https://www.ftc.gov/news-events/blogs/business-blog/2021/04/aiming-truth-fairness-equity-your-companys-use-ai
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://archive.ph/i5m4v
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.lemonade.com/blog/lemonades-claim-automation/
[8] https://twitter.com/Lemonade_Inc/status/1397564445648424965?ref_src=twsrc%5Etfw
[9] https://www.lemonade.com/privacy-policy
[10] https://www.theregister.com/2021/04/24/in_brief_ai/
[11] https://www.theregister.com/2020/10/28/gpt3_medical_chatbot_experiment/
[12] https://www.theregister.com/2020/09/21/twitter_image_cropping_ai/
[13] https://www.theregister.com/2019/11/08/ai_liver_transplant/
[14] https://www.sec.gov/Archives/edgar/data/1691421/000104746920003846/a2241899zs-1a.htm#ds40510_legal_matters
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YK8ZXASjA4VM7c3ATKmSfgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[16] https://whitepapers.theregister.com/
Just wrong
Falmari
Using AI and facial recognition is simply wrong for the reasons mentioned many times on The Reg. But even if you accept their statement that they don’t use AI why do they need a video?
If they are using people to do the same thing it is still wrong you can’t deny a claim just because you think the claimant looked a bit shady, hesitated here, touched their nose there.
If there are discrepancies in their statement that shows they are lying sure you can deny the claim. Investigate and get evidence if you think the claim is false. Neither of which need a video. But you can’t deny the claim because he looked like a dodgy geezer.
Why do you need AI to process claims especially if it can “overpay claims that are made by our customers”. You pay out on the cover offered. For example, a car if replace as new it is the current price or if current value then the average second had prices. For personal injury insurance companies have algorithms to calculate the damages based on profession age etc. No need for AI.
They also operate in France, Germany, and the Netherlands, surely collecting this data must fall foul of GDPR?
Apprentice Human
Apart from the USA being the only place that believes lie detectors work, as they have been proven to be rubbish (even in the US)... any 2nd year psychology student should be able to say why this sort of AI will be bad.
To your favourite chat: "What a load of rubbish."
And permissions...
"That said, the company’s privacy policy does say it collects, among other details, people's physical characteristics when handling life insurance.
And in a filing to America's financial regulator, the SEC, Lemonade said its system collects roughly 1,700 “data points” from customers."
And this is why I have all permissions removed from phone apps until I decide when and where they need them at time of usage. As I've often said before, who needs to fear governmental Big Brother when Big Business already does it so much better??!
The miserable creeps.