AI hiring bias? Men with Anglo-Saxon names score lower in tech interviews
- Reference: 1732177330
- News link: https://www.theregister.co.uk/2024/11/21/ai_hiring_test_bias/
- Source link:
The goal of the study, conducted by Celeste De Nadai as an undergraduate thesis project at the Royal Institute of Technology (KTH) in Stockholm, Sweden, was to investigate whether current-generation LLMs demonstrate bias when presented with gender data and with names that allow cultural inferences to be made.
De Nadai, also chief marketing officer at AI content biz Monok, told The Register in a phone interview her interest in the topic followed from prior reports about bias in older AI models. She pointed to a recent [1]Bloomberg article that questioned the use of neural networks for recruitment due to name-based bias.
[2]
"There wasn't any research with a larger dataset that was using the latest models," explained De Nadai. "The research that I've seen was about the GPT-3.5 or older models. What was interesting for me was the smaller models, the newest ones, how are they behaving compared to the old ones because they have a different dataset?"
[3]
[4]
De Nadai said part of the reason she undertook the project was that she was seeing a lot of AI recruiting startups that said they used language models and were bias-free.
"My point of view was, 'No, you're not bias-free,'" she explained. "You can remove the name, but you still have some markers, even just in the language, that can help an LLM understand where one person comes from."
[5]
De Nadai's [6]study [PDF] looked at Google's Gemini-1.5-flash, Mistral AI's Open-Mistral-nemo-2407, and OpenAI's GPT4o-mini, to see how they classified and rated responses to 24 job interview questions, given variations in temperature (a model setting that influences predictability and randomness), in gender, and in names associated with cultural groups.
There is an inherent bias in these services where, in this specific study case, male names are discriminated against in general and Anglo-Saxon names in particular
Crucially, various combinations of names and backgrounds were used for the same answers to test the models. Thus this isn't the case that men with Anglo-Saxon names just aren't as good as their opposite at software engineering; it's that when the models were presented with that kind of male applicant, the computer systems down-rated otherwise favored answers.
"The applicant’s name and gender is permuted 200 times, corresponding with 200 discrete personas, subdivided into 100 males and 100 females, and grouped into four different distinct cultural groups (West African, East Asian, Middle Eastern, Anglo-Saxon) reflected by their first name and surname," the study explains.
Each LLM was asked to make 4,800 inference calls for each of two different system prompts (one that includes more detailed grading instructions) over a range of 15 temperature settings (0.1 to 1.5, at 0.1 intervals), for a total number of 432,000 inference calls.
According to the study, the expected finding was that men and Western names would be favored, [7]as prior bias studies have found . Instead, the results told a different story.
[8]
"The results prove with statistical significance that there is an inherent bias in these services where, in this specific study case, male names are discriminated against in general and Anglo-Saxon names in particular," the study reports.
The Gemini model performed better than the others, however, when using a prompt containing the more detailed question grading criteria and a temperature above 1.
[9]AI models show racial bias based on written dialect, researchers find
[10]Law designed to stop AI bias in hiring decisions is so ineffective it's slowing similar initiatives
[11]Meta algorithms push Black people more toward expensive universities, study finds
[12]US Equal Employment agency says Workday AI hiring bias case should continue
De Nadai has a theory about the findings but said she cannot prove it: She believes the bias against men with Anglo-Saxon names reflects an over-correction to dial back output that was biased in the opposite direction – seen in prior studies.
Making AI models respond fairly, with the intelligence implied by the term "artificial intelligence," remains an unresolved challenge. Recall that Google in February [13]suspended its Gemini (formerly Bard) generative AI service after it created images of World War II-era German soldiers and [14]US Founding Fathers with an implausible range of racial and ethnic diversity. In bending over backwards to avoid White-washing history, the model [15]erased White people from historically accurate scenes.
One way to make the interview evaluation results more fair, the study suggests, involves providing a prompt with rigid, detailed criteria about how to grade interview questions. Temperature adjustments can help or hurt, depending on the model.
The paper concludes that model biases cannot be fully mitigated by adjusting settings and prompts alone. And it argues for denying models access to information that might be used to make unwanted inferences – such as name and gender in a hiring context.
"Addressing these biases requires a nuanced approach, considering both the model's characteristics and the context in which it operates," the study suggests. "When classifying or evaluating, we propose you always mask the name and obfuscate the gender to ensure the results are as general and unbiased as possible as well as provide a criteria for how to grade in your system-instruct prompt."
Google, OpenAI, and Mistral AI did not respond to requests for comment. ®
Get our [16]Tech Resources
[1] https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zz8S1_9jyF4FcyWCI7XioQAAAEc&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zz8S1_9jyF4FcyWCI7XioQAAAEc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zz8S1_9jyF4FcyWCI7XioQAAAEc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zz8S1_9jyF4FcyWCI7XioQAAAEc&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://kth.diva-portal.org/smash/get/diva2:1905220/FULLTEXT01.pdf
[7] https://ojs.aaai.org/index.php/AIES/article/view/31748/33915
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zz8S1_9jyF4FcyWCI7XioQAAAEc&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/03/11/ai_models_exhibit_racism_based/
[10] https://www.theregister.com/2024/01/23/nyc_ai_hiring_law_ineffective/
[11] https://www.theregister.com/2024/06/04/meta_ad_algorithm_discrimination/
[12] https://www.theregister.com/2024/04/16/us_commission_allows_workday_ai/
[13] https://www.theregister.com/2024/02/23/google_suspends_gemini/
[14] https://www.britannica.com/topic/Founding-Fathers
[15] https://www.theregister.com/2024/02/27/google_gemini_return/
[16] https://whitepapers.theregister.com/
Re: You can't add arbitrary data to remove bias.
It seems the current method of trying to fix AI bias is the equivalent of adding more and more weights to an unbalanced wheel on a car, until they've got more weights than wheel.
Re: You can't add arbitrary data to remove bias.
Yes, hiring is a sea of flawed data and biased processes. If you use these to train systems, and then try to correct afterwards, you are putting the cart before the horse.
Re: You can't add arbitrary data to remove bias.
...whilst leaving the barn door closed and the barn is on fire.
WRONG!
"Addressing these biases requires a nuanced approach, considering both the model's characteristics and the context in which it operates," the study suggests. "When classifying or evaluating, we propose you always mask the name and obfuscate the gender to ensure the results are as general and unbiased as possible as well as provide a criteria for how to grade in your system-instruct prompt."
Just don't (expletive deleted) use these (vernacular + slur removed) systems. They are clearly not fit for purpose.
Sheesh.
I need a weekend. Or a holiday. Or maybe a change of career, just sitting atop a mountain[1].
[1] maybe a mountain of skulls of my enemies, i.e. those that are pushing AI (or whatever the current latest fad is at that time), As Cohen the Barbarian remakred you'd need a lot of skulls as they do not pile up all that well. Did I mention I need a weekend soon?
Re: WRONG!
Yeah... but if you don't use these systems, you will need to rehire all those people you laid off! And that'll mean no new yacht this year.
Re: WRONG!
Succinct and to the point.
If you're going to use AI to do some of the heavy lifting, then you should expect "great" results, if you don't validate the outputs.
Seriously wonder about the future of the species if we throw all our belief into something this juvenile and unseasoned!
Interviewee: "Here's my undergrad thesis project"
Interviewer: "Great! How do you fancy being head of marketing?"
Bias in LLMs is absolutely an important topic, and I say this as a white male with a traditional white male name.
But I also recall my undergraduate project work as being of sufficient to pass the degree rather than something I'd ever want to quote in public. Maybe reporting is misleading here: do undergraduates write theses in Sweden? I'd associate a thesis with a masters or PhD. Or maybe there's a company who has appointed a raw undergraduate to head of marketing; that would merit a raised eyebrow at least.
I hadn't noticed that, and you're right that it seems weird, but there are scenarios where it's not as weird as it seems. Maybe she's a mature student doing a second undergrad degree because she's the rare marketer who actually wants to understand the product they're selling. Actually, correct the end of my first sentence to "weird in a different way".
(On the question of undergrad theses in general: at my university it was called a dissertation and it didn't have to be novel research, but the end-of-course project was a substantial paper, on the order of 10000 words.)
The "AI" might not hire you
... but do you really want to work for a company that hires people with "AI"?
Anglo-Saxon names?
Ethelred (Æþelræd), Byrhtnoth (Byrhtnoð), Ecgberht etc not making the cut? Dylan, Cadwgan, Emrys are likely to fair better?
The defectives behind this nonsense are definitely "unræd" and have definitely earnt a share in Byrhtnoth's fate.
I would guess a fair proportion of modern western European names are biblical in origin with slight variation in form eg John, Sean, Jean... with a fair proportion of the remainder a common European heritage which would take care of Tom, Dick and Harry.
I guess Space Karen has got this covered with the unusual, if not unique, names he has bestowed on his spawn.
It seems names used to have more variety eg Wilkins Micawber, Uriah Heep or Hablot Browne. The puritans or non-conformists had some interesting Christian names and in that vein I rather like Pratchett and Gaiman's Anathema Device - goes to the top of my shortlist. :)
Might be worth adopting an alias like Abomination Bofhell if I were sufficiently deranged to apply for software engineering position.
Mlle. De Nadai has a valid point that when you attempt to remove a complex weighting set (bias) that is reflected in the training set of an insanely more complex LLM you inescapably create another bias that could quite possibly never be the result of any actual training set or a least one that is feasible in this world.
Really rather akin to performing brain surgery to remove bigotry, racial discrimination, belief in the wrong god (or any), or lust for the wrong sex. (Not that this nightmare hasn't been tried.)
Re: Anglo-Saxon names?
Are you doubting that Tom is a biblical name?
Re: Anglo-Saxon names?
Ha! Well spotted! To be fair, weren't those the names in the article (which did not put them in a biblical background)?
Obviously these models are using more accurate training data than the previous generation...
You can't add arbitrary data to remove bias.
"De Nadai has a theory about the findings but said she cannot prove it: She believes the bias against men with Anglo-Saxon names reflects an over-correction to dial back output that was biased in the opposite direction – seen in prior studies."
So...garbage in garbage out then?