LLMs can write and answer quizzes – but aren't quite ready to disrupt trivia night
(2024/05/29)
- Reference: 1716981251
- News link: https://www.theregister.co.uk/2024/05/29/autoquizzer_llm_quiz_generation/
- Source link:
A developer has put large language models (LLMs) to the test, literally, by creating AutoQuizzer – a tool that creates quizzes from text on web pages.
The application was made by Stefano Fiorucci – whose day job sees him toil as a software engineer for enterprise AI outfit Deepset – and the code is available on [1]GitHub . Fiorucci also hosts a version of AutoQuizzer on [2]Hugging Face .
Using the app is easy: Feed it a URL, click "Generate quiz," and then prepare to test yourself against an LLM's interpretation of the page's content in a multiple-choice quiz created by the model. The system attempts to generate five questions per page.
[3]
In our testing, it only took a second or two to create a quiz, which users can complete themselves or hand back to be answered by the AI system. When the app itself takes the quiz, you have the option to force it into a "closed book exam" mode, in which the model relies just on the page topic, the questions, and any information it was trained on to pick an answer. Alternatively, the AI can be allowed to consider the top three Google search results regarding the topic of the web page. In either mode, the AI code needs a handful of seconds to come up with answers.
[4]
[5]
Fiorucci explained to The Register that creating AutoQuizzer was actually "very simple," since the components to build it were already available. The app uses Deepset's open source framework Haystack to extract text from a specified page, and pass it to Meta's LLaMa-3-8B-Instruct LLM via Groq's free inference API. The LLaMa neural network is [6]prompted to analyze the text and generate a quiz based on that content in JSON format for the web app to display, and either the user or LLaMa itself can answer.
It's possible to use other, more powerful LLMs, Fiorucci noted, but there are specific reasons why he uses LLaMa-3-8B for AutoQuizzer. Perhaps most importantly, the model – being relatively small and fast – can be used for free via Groq's API, making a free-to-use web-based demo possible.
[7]
Other small LLMs didn't pan out. "I tried Phi-3-mini by Microsoft because it has very good performance on benchmarks, despite its small size: it has fewer than 4 billion parameters. Compared to LLaMa-3, it failed to produce valid JSON, and the quiz questions sometimes were too easy or poorly created," Fiorucci said.
Anyone who wants to make their own version of AutoQuizzer with another LLM, such as a more powerful, larger model, can do so; Fiorucci said LLaMa-3 can be swapped out with, say, a member of the GPT family.
Proof of concept – not a trivia night disrupter
In order to stay within the rules of the free Groq API, AutoQuizzer will only send the first 4,000 characters of a web page to the LLM to analyze. Fiorucci told The Register LLaMa-3-8B copes better with sources like Wikipedia articles than it does with news articles. That said, the character limit is more likely to be a problem with Wikipedia pages, which is inconvenient: News stories tend to put the most important information at the start, whereas Wikipedia entries aren't structured as such other than kicking off with a summary.
In The Register 's testing, AutoQuizzer usually served up decent questions and suitable answers. Almost every single question had four basic answer choices – with only one question ever offering an "all of the above" option – and all questions were on topic. It can even generate questions in English from a non-English article, though this is not ideal for the LLM and can introduce mistakes.
[8]Has your machine really learned something? Snap quiz time
[9]AI's most convincing conversations are not what they seem
[10]OpenAI's DALL·E 3 teams up with ChatGPT to turn brainfarts into art
[11]Is this paragraph from Trump or an AI bot? You decide, plus buy your own AI for $399
When we let LLaMa-3-8B answer the quizzes it generated, it usually answered three or four of the five questions correctly when allowed access to Google results – which isn't half bad but is, well, cheating. Also, one might expect an LLM to be able to answer its own questions, given the text-completion nature of these sorts of language models.
We did find some quirks. Some questions had duplicated or very similar answer choices, or answers that didn't completely address the question.
The tool could also miss the point of the content it was asked to consider. This Register article about [12]Microsoft offering relocation opportunities to Chinese employees prompted the AI to ask: "What is the reason for the increased tariffs on electric vehicles in China, according to the article?" The correct answer was: "Due to US President Joe Biden's decision."
[13]
Which is kind-of correct, but unrelated to the text AutoQuizzer was asked to consider.
For Fiorucci, however, the point of AutoQuizzer isn't to test LLMs in a unique way or to provide some kind of practical use case. "AutoQuizzer is part of an effort to show how easily you can make both demos and production software using Haystack," he explained, referring to his employer's framework, natch. "Haystack is a powerful open source framework for building applications based on large language models."
"In its current form, AutoQuizzer is a hobby project," he conceded, noting "it could be turned into a library or CLI application. It might also serve as inspiration for creating similar, more refined tools in the educational or entertainment fields."
Given the quality of its output, LLaMa-3-8B probably isn't the right tool for such an application in education or academia, though perhaps a more powerful or a future model would be more usable. Indeed, some coders might already be working to refine AutoQuizzer – GitHub reveals six forks. ®
Get our [14]Tech Resources
[1] https://github.com/anakin87/autoquizzer
[2] https://huggingface.co/spaces/deepset/autoquizzer
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://github.com/anakin87/autoquizzer/blob/2a15409c6fa55f23e8505e7012fec040fd91f7d6/backend/pipelines.py#L11
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2018/04/13/testing_what_machine_learning_knows/
[9] https://www.theregister.com/2022/06/20/ais_most_convincing_conversations_are/
[10] https://www.theregister.com/2023/09/25/ai_in_brief/
[11] https://www.theregister.com/2019/11/11/ai_roundup_081119/
[12] https://www.theregister.com/2024/05/17/microsoft_china_staff_relocate/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
The application was made by Stefano Fiorucci – whose day job sees him toil as a software engineer for enterprise AI outfit Deepset – and the code is available on [1]GitHub . Fiorucci also hosts a version of AutoQuizzer on [2]Hugging Face .
Using the app is easy: Feed it a URL, click "Generate quiz," and then prepare to test yourself against an LLM's interpretation of the page's content in a multiple-choice quiz created by the model. The system attempts to generate five questions per page.
[3]
In our testing, it only took a second or two to create a quiz, which users can complete themselves or hand back to be answered by the AI system. When the app itself takes the quiz, you have the option to force it into a "closed book exam" mode, in which the model relies just on the page topic, the questions, and any information it was trained on to pick an answer. Alternatively, the AI can be allowed to consider the top three Google search results regarding the topic of the web page. In either mode, the AI code needs a handful of seconds to come up with answers.
[4]
[5]
Fiorucci explained to The Register that creating AutoQuizzer was actually "very simple," since the components to build it were already available. The app uses Deepset's open source framework Haystack to extract text from a specified page, and pass it to Meta's LLaMa-3-8B-Instruct LLM via Groq's free inference API. The LLaMa neural network is [6]prompted to analyze the text and generate a quiz based on that content in JSON format for the web app to display, and either the user or LLaMa itself can answer.
It's possible to use other, more powerful LLMs, Fiorucci noted, but there are specific reasons why he uses LLaMa-3-8B for AutoQuizzer. Perhaps most importantly, the model – being relatively small and fast – can be used for free via Groq's API, making a free-to-use web-based demo possible.
[7]
Other small LLMs didn't pan out. "I tried Phi-3-mini by Microsoft because it has very good performance on benchmarks, despite its small size: it has fewer than 4 billion parameters. Compared to LLaMa-3, it failed to produce valid JSON, and the quiz questions sometimes were too easy or poorly created," Fiorucci said.
Anyone who wants to make their own version of AutoQuizzer with another LLM, such as a more powerful, larger model, can do so; Fiorucci said LLaMa-3 can be swapped out with, say, a member of the GPT family.
Proof of concept – not a trivia night disrupter
In order to stay within the rules of the free Groq API, AutoQuizzer will only send the first 4,000 characters of a web page to the LLM to analyze. Fiorucci told The Register LLaMa-3-8B copes better with sources like Wikipedia articles than it does with news articles. That said, the character limit is more likely to be a problem with Wikipedia pages, which is inconvenient: News stories tend to put the most important information at the start, whereas Wikipedia entries aren't structured as such other than kicking off with a summary.
In The Register 's testing, AutoQuizzer usually served up decent questions and suitable answers. Almost every single question had four basic answer choices – with only one question ever offering an "all of the above" option – and all questions were on topic. It can even generate questions in English from a non-English article, though this is not ideal for the LLM and can introduce mistakes.
[8]Has your machine really learned something? Snap quiz time
[9]AI's most convincing conversations are not what they seem
[10]OpenAI's DALL·E 3 teams up with ChatGPT to turn brainfarts into art
[11]Is this paragraph from Trump or an AI bot? You decide, plus buy your own AI for $399
When we let LLaMa-3-8B answer the quizzes it generated, it usually answered three or four of the five questions correctly when allowed access to Google results – which isn't half bad but is, well, cheating. Also, one might expect an LLM to be able to answer its own questions, given the text-completion nature of these sorts of language models.
We did find some quirks. Some questions had duplicated or very similar answer choices, or answers that didn't completely address the question.
The tool could also miss the point of the content it was asked to consider. This Register article about [12]Microsoft offering relocation opportunities to Chinese employees prompted the AI to ask: "What is the reason for the increased tariffs on electric vehicles in China, according to the article?" The correct answer was: "Due to US President Joe Biden's decision."
[13]
Which is kind-of correct, but unrelated to the text AutoQuizzer was asked to consider.
For Fiorucci, however, the point of AutoQuizzer isn't to test LLMs in a unique way or to provide some kind of practical use case. "AutoQuizzer is part of an effort to show how easily you can make both demos and production software using Haystack," he explained, referring to his employer's framework, natch. "Haystack is a powerful open source framework for building applications based on large language models."
"In its current form, AutoQuizzer is a hobby project," he conceded, noting "it could be turned into a library or CLI application. It might also serve as inspiration for creating similar, more refined tools in the educational or entertainment fields."
Given the quality of its output, LLaMa-3-8B probably isn't the right tool for such an application in education or academia, though perhaps a more powerful or a future model would be more usable. Indeed, some coders might already be working to refine AutoQuizzer – GitHub reveals six forks. ®
Get our [14]Tech Resources
[1] https://github.com/anakin87/autoquizzer
[2] https://huggingface.co/spaces/deepset/autoquizzer
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://github.com/anakin87/autoquizzer/blob/2a15409c6fa55f23e8505e7012fec040fd91f7d6/backend/pipelines.py#L11
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2018/04/13/testing_what_machine_learning_knows/
[9] https://www.theregister.com/2022/06/20/ais_most_convincing_conversations_are/
[10] https://www.theregister.com/2023/09/25/ai_in_brief/
[11] https://www.theregister.com/2019/11/11/ai_roundup_081119/
[12] https://www.theregister.com/2024/05/17/microsoft_china_staff_relocate/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZldRIBcu22yZfvU05E0SlAAAAEU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
Really needs RAG
Charlie Clark
The problem with the public models is that they're circular. Create a quiz with one and it's easy to create a client that can answer all the questions.
With RAG you can use the models to create questions with documents and data that clients don't necessarily have. You can see how copyright holders might use this approach to get on board.
PB90210
I came across a page on The Poke showing ChatGPT is crap at 'Rock, Paper, Scissors'...
ChatGPT> I'll go first. I choose Rock
Human> I choose Paper
(repeat)
Dan 55
I got Bing Copilot stuck in a loop in three questions and I wasn't even trying. How the hell this nonsense is getting billions of dollars of investment I'll never know.
Charlie Clark
And yet, can you remember Groupon or pets.com or Uber? Silicon Valley has become the most effective snake oil salesman the world has ever seen. Well, apart from religions that is!
Anonymous Coward
I stuck in the URL of this article. https://www.theregister.com/2024/05/29/autoquizzer_llm_quiz_generation/
Questions looked good but a lot of the suggested answers were the same across questions. It scored 60% in the closed book and 60% from the web crawl.
Could actually be fun trying to find really obscure sites and testing it.
Coming soon quizzy APIs
H. sapien Floridanus
Captcha: STOP. He who pass the login screen, must answer me these questions three: What is your name?
User: hot2trot1995
Captcha: What is your quest?
User: Cat videos? [aaaaaayy]
Real housewives
No, not porn. The crappy reality show that my partner loves to watch. I got ChatGPT to write 10 quiz questions on it to see if she could answer them as she knows the show so much.
About 4 of them were questions about only one person in the show so the answer was the same 4 times, another 2 were about another person in the show so again, same answer for those 2 questions and about 3 of them the answer was wrong and one of them wasn't really a proper question.