AI Secretly Helped Write California Bar Exam, Sparking Uproar (arstechnica.com)
- Reference: 0177111559
- News link: https://news.slashdot.org/story/25/04/23/2025217/ai-secretly-helped-write-california-bar-exam-sparking-uproar
- Source link: https://arstechnica.com/tech-policy/2025/04/ai-secretly-helped-write-california-bar-exam-sparking-uproar/
> On Monday, the State Bar of California [1]revealed that it [2]used AI to develop a portion of multiple-choice questions on its February 2025 bar exam , causing outrage among law school faculty and test takers. The admission comes after weeks of complaints about technical problems and irregularities during the exam administration, [3]reports the Los Angeles Times. The State Bar disclosed that its psychometrician (a person skilled in administrating psychological tests), ACS Ventures, created 23 of the 171 scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the remaining 100 questions.
>
> The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before the exam. "The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam," wrote State Bar Executive Director Leah Wilson in a press release. According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable." Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam.
The report notes that the AI disclosure follows technical glitches with the February exam (like login issues, screen lag, and confusing questions), which led to a federal lawsuit against Meazure Learning and calls for a State Bar audit.
[1] https://www.calbar.ca.gov/About-Us/News/News-Releases/CBE-Recommends-Scoring-Adjustments-for-February-Bar-Exam
[2] https://arstechnica.com/tech-policy/2025/04/ai-secretly-helped-write-california-bar-exam-sparking-uproar/
[3] https://www.latimes.com/california/story/2025-04-23/state-bar-of-california-used-ai-for-exam-questions
"No, put your hand in the OPEN side of the box." (Score:2)
With the advent of LLMs, the lazily thoughtless among us are outing themselves as never before. None of these people would survive the gom jabbar test.
Re: (Score:1)
please pray for the pope
Re: "No, put your hand in the OPEN side of the box (Score:2)
Nitpick, but the test is referred to as the "human test" or "test for humanity", while the gom jabbar is the poison. The gom jabbar is not necessarily the only way to run the test. You could just as easily point a lasgun at their head.
Re: (Score:2)
The test for humanity is that you gut out and bear the pain of the nerve-induction box, not just because you don't want to die but because you are plotting your revenge against who is subjecting you to this torture.
Not only did Paul Atreides amaze the Reverend Mother by bearing much more pain than anyone else, the revenge he carried out against the Reverend Mother was way beyond anything she could have even imagined.
As to gom jabbar vs lasgun, first of all lasgun weren't threatening to people in a cult
Tempest in a teapot? (Score:4, Insightful)
> The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam...
Because of the subsequent review by qualified humans, I don't see this as a problem. But the scenario described in TFS is just begging for a 'slippery slope' argument.
I think it's inevitable that in the future, one AI will be tasked with checking the work of another AI. Then, in a storm of stupidity, the policy will become having the AI check its own homework.
After that, of course, the practice will devolve into just trusting the initial AI output, with no verification step. Welcome to the AI apocalypse!
Re: (Score:2)
>> The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam...
> Because of the subsequent review by qualified humans, I don't see this as a problem. But the scenario described in TFS is just begging for a 'slippery slope' argument.
> I think it's inevitable that in the future, one AI will be tasked with checking the work of another AI. Then, in a storm of stupidity, the policy will become having the AI check its own homework.
> After that, of course, the practice will devolve into just trusting the initial AI output, with no verification step. Welcome to the AI apocalypse!
My concern in this particular instance is that the company using the AI was also in charge of reviewing the questions generated by AI. And the end result was "confusing questions" (straight from the summary) which was partially the reason the entire test (along with login issues and screen lag) was called into question. I'm thinking there's bigger systemic issues here than just having AI involved in the process, but having AI involved in the process clearly wasn't helping anything.
Though I know what you're
Would they have counted? (Score:3)
The real question is if these AI-generated questions were going to be counted in the final score, or if they were merely being evaluated for inclusion in future exams.
A colleague of mine is getting his MD, and the periodic tests they take contain a fair fraction of questions that are being evaluated for inclusion on future exams, but do not count toward the present score. Could the same have been happening here, but that important distinction being lost in the clamorous din?
As with any professional qualification exam, there is a certain level of knowledge that must be demonstrated. As long as the questions being used to demonstrate that knowledge are vetted by experts in the field and validated before being officially used, does it really matter who wrote them or how?
So, in CA (Score:1)
The Bars are owned by the State?
Re: So, in CA (Score:2)
Yup, just like MAGA is owned by Russians and the Saudis
Re: (Score:2)
Is this different than any other US state?
First year law student exam (Score:2)
I am more interested in the fact that 48 questions came from a first-year law student exam. Wouldn't missing even one of these raise questions about the exam takers knowledge? A certain level of basic knowledge would seem to be implied by merely having a law degree before even taking the exam.
Re:First year law student exam (Score:4, Insightful)
> I am more interested in the fact that 48 questions came from a first-year law student exam. Wouldn't missing even one of these raise questions about the exam takers knowledge?
Eh, that's a standard that is a bit too high: law students and lawyers aren't superhumans. Would you be able to get 48/48 questions right in an exam that you took 3-4 years after the class in which you learned them? If the answer is yes, do you think ALL of your peers could have?
Re: (Score:2)
Could I get 48/48 on a first year exam?
Possibly. But even if it's a 46/48 or 47/48, it's still a high passing score.
Could my classmates? I went to school with a lot of retards so probably not. And I'm 110% okay with the retards failing the test in any field of study.
Re: First year law student exam (Score:2)
How do you say "you're all gonna be replaced by AI some day" without saying it directly? Use it to make their tests. ;-)