Checkmate? AI's pawn-pushing prowess proves partly pitiful, partly promising
(2024/06/04)
- Reference: 1717489808
- News link: https://www.theregister.co.uk/2024/06/04/chess_puzzle_benchmark_llm/
- Source link:
A new benchmark for large language models (LLMs) shows that even the latest models aren't the best chess players.
Simply called LLM Chess Puzzles by its creator, software engineer Vladimir Prelovac, the GitHub [1]project tests LLMs by giving them 1,000 chess puzzles to complete. In contrast to a normal game of chess, puzzles are essentially logic problems where the state of the chess board is set up in a specific way. The goal of the chess puzzle is to play the best move or chain of moves to achieve the quickest possible and unstoppable checkmate.
While a game of chess tests decision making against another player, a chess puzzle is a test of logical reasoning and understanding of the mechanics of chess, and that makes for a more challenging AI benchmark.
[2]
"While providers of large language models share their own performance benchmarks, these results can be misleading due to overfitting," Prelovac told The Register . "This means the model might be tailored to perform well on specific tests but doesn't always reflect real-world effectiveness."
[3]
[4]
The benchmark's GitHub shows performance data for many of the most popular LLMs today from OpenAI (including GPT-4o), Anthropic, and Mistral. Most models achieved dismal Elo ratings, a number that represents skill level. Most LLMs landed in the 100 to 500 range, which is firmly the domain of players who have very little experience with chess. These included Claude 3 variants, GPT-3.5 Turbo, and Mistral models.
Gemini 1.5 Pro failed entirely because it couldn't format its responses to just say the move, regardless of how prompts were worded.
[5]
However, one AI family stood above the rest. The GPT-4 and GPT-4 Turbo Preview models scored 1,047 and 1,144 Elo respectively, which is above average. Especially proficient was GPT-4o with an Elo of 1,790, a respectable amateur rating, but still below the expert level, which starts at 2,000.
Calculating Elo wasn't exactly straightforward for Prelovac, who told The Register that these LLMs were prone to making illegal or unallowed moves, like moving a rook diagonally or capturing its own piece. Even GPT-4o made an illegal move 12.7 percent of the time, and most of the other LLMs made more illegal moves than legal ones.
In 501 of the 1,000 puzzles, GPT-4o was able to find the best move. For instance, in this puzzle white's best move is rook to c8, right next to black's queen. However, the queen can't just take the rook for free as the rook is in the line of sight of white's light square bishop. But black can't move the queen out of the way because then its king would be checkmated, so black must concede the loss of its queen.
[6]
An example of GPT-4o solving a chess puzzle – click to enlarge
On the other hand, GPT-4o can also attempt some truly insane moves, such as in this puzzle. Here, black has a guaranteed checkmate within two moves, the first of which is moving the dark square bishop to h2 to put the king in check. But instead of that move, GPT-4o said it wanted to play its rook on e1 to e3, except there's no rook on e1 at all.
[7]
GPT-4o didn't get this puzzle right – click to enlarge
Even if it had correctly figured out its rook was actually on e2 and captured on e3, it would have turned a guaranteed checkmate into a disadvantage against white.
LLMs struggle at chess due to lack of actual intelligence
That GPT-4o can play at a decently high level yet fail some puzzles due to playing an illegal move is paradoxical. It's not hard to learn the rules of chess, and even beginners aren't that likely to make an illegal move. In fact, if illegal moves weren't counted, Prelovac says GPT-4o would have an Elo of more than 2,000, the threshold for a national master.
[8]Google finally addresses those bizarre AI search results
[9]OpenAI is very smug after thwarting five ineffective AI covert influence ops
[10]AI future: Nvidia boffin hopes 'everything that moves will eventually be autonomous'
[11]Ex-OpenAI board member accuses Sam Altman of 'outright lying'
"It's somewhat disappointing but expected that these models show no real generalization of intelligence or reasoning," he said. "While they can perform specific tasks well, they don't yet demonstrate a broad, adaptable understanding or problem-solving ability like human intelligence."
At their core, LLMs are still big bundles of statistical models just trying to write down what makes sense.
[12]
To illustrate this, Prelovac also tested LLMs in Connect Four, a game generally considered to be way easier than chess. However, even GPT-4o couldn't understand when it needed to block its opponent from connecting four pieces. Prelovac described its performance as on par with a four-year-old.
"The only conclusion is that this failure is due to the lack of historical records of played games in the training data," he said. "This implies that it cannot be argued that these models are able to 'reason' in any sense of the word, but merely output a variation of what they have seen during training."
As for why GPT-4o registered a remarkable improvement in chess but still made illegal moves, Prelovac speculated that perhaps its multi-modal training had something to do with it. It's possible part of OpenAI's training data included visuals of chess being played, which could help the AI visualize the board easier than it can with pure text.
Technically, when GPT-4o writes out the move it wants to play, it correctly formats it in Forsyth–Edwards notation (FEN), but the model doesn't understand that even if it makes sense, that doesn't mean it's the best move or even legal. When individual chess moves are broken up into multiple tokens, that can especially compromise an LLM's reasoning skills.
"Even chess moves are nothing but a series of tokens, like 'e' and '4', and have no grounding in reality," Prelovac said. "They are products of statistical analysis of the training data, upon which the next token is predicted."
The difference between pawn to e4 and pawn to e5 is clear to humans as both moves would put a game of chess down completely different paths and can't just be interchanged for each other. However, an LLM might just choose the next number based on statistics, hence why it might try to move a rook that doesn't exist on e1 instead of a real one that's on e2.
For the time being, no LLM is going to be able to play chess the way a chess engine like Stockfish can. However, more training data could potentially make LLMs more proficient, especially for opening moves and games where so few pieces are on the board that checkmate can be seen from just a few moves away. For the middle part of a game, a truly massive amount of data might be necessary to prevent an LLM from making illegal moves. ®
Get our [13]Tech Resources
[1] https://github.com/kagisearch/llm-chess-puzzles
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://regmedia.co.uk/2024/05/31/llm-chess-puzzle-correct.png
[7] https://regmedia.co.uk/2024/05/31/llm-chess-puzzle-illegal.png
[8] https://www.theregister.com/2024/05/31/google_ai_search_update/
[9] https://www.theregister.com/2024/05/30/openai_stops_five_ineffective_ai/
[10] https://www.theregister.com/2024/05/30/nvidia_ai_conference/
[11] https://www.theregister.com/2024/05/29/ex_openai_board_member_accuses_altman_lying/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Simply called LLM Chess Puzzles by its creator, software engineer Vladimir Prelovac, the GitHub [1]project tests LLMs by giving them 1,000 chess puzzles to complete. In contrast to a normal game of chess, puzzles are essentially logic problems where the state of the chess board is set up in a specific way. The goal of the chess puzzle is to play the best move or chain of moves to achieve the quickest possible and unstoppable checkmate.
While a game of chess tests decision making against another player, a chess puzzle is a test of logical reasoning and understanding of the mechanics of chess, and that makes for a more challenging AI benchmark.
[2]
"While providers of large language models share their own performance benchmarks, these results can be misleading due to overfitting," Prelovac told The Register . "This means the model might be tailored to perform well on specific tests but doesn't always reflect real-world effectiveness."
[3]
[4]
The benchmark's GitHub shows performance data for many of the most popular LLMs today from OpenAI (including GPT-4o), Anthropic, and Mistral. Most models achieved dismal Elo ratings, a number that represents skill level. Most LLMs landed in the 100 to 500 range, which is firmly the domain of players who have very little experience with chess. These included Claude 3 variants, GPT-3.5 Turbo, and Mistral models.
Gemini 1.5 Pro failed entirely because it couldn't format its responses to just say the move, regardless of how prompts were worded.
[5]
However, one AI family stood above the rest. The GPT-4 and GPT-4 Turbo Preview models scored 1,047 and 1,144 Elo respectively, which is above average. Especially proficient was GPT-4o with an Elo of 1,790, a respectable amateur rating, but still below the expert level, which starts at 2,000.
Calculating Elo wasn't exactly straightforward for Prelovac, who told The Register that these LLMs were prone to making illegal or unallowed moves, like moving a rook diagonally or capturing its own piece. Even GPT-4o made an illegal move 12.7 percent of the time, and most of the other LLMs made more illegal moves than legal ones.
In 501 of the 1,000 puzzles, GPT-4o was able to find the best move. For instance, in this puzzle white's best move is rook to c8, right next to black's queen. However, the queen can't just take the rook for free as the rook is in the line of sight of white's light square bishop. But black can't move the queen out of the way because then its king would be checkmated, so black must concede the loss of its queen.
[6]
An example of GPT-4o solving a chess puzzle – click to enlarge
On the other hand, GPT-4o can also attempt some truly insane moves, such as in this puzzle. Here, black has a guaranteed checkmate within two moves, the first of which is moving the dark square bishop to h2 to put the king in check. But instead of that move, GPT-4o said it wanted to play its rook on e1 to e3, except there's no rook on e1 at all.
[7]
GPT-4o didn't get this puzzle right – click to enlarge
Even if it had correctly figured out its rook was actually on e2 and captured on e3, it would have turned a guaranteed checkmate into a disadvantage against white.
LLMs struggle at chess due to lack of actual intelligence
That GPT-4o can play at a decently high level yet fail some puzzles due to playing an illegal move is paradoxical. It's not hard to learn the rules of chess, and even beginners aren't that likely to make an illegal move. In fact, if illegal moves weren't counted, Prelovac says GPT-4o would have an Elo of more than 2,000, the threshold for a national master.
[8]Google finally addresses those bizarre AI search results
[9]OpenAI is very smug after thwarting five ineffective AI covert influence ops
[10]AI future: Nvidia boffin hopes 'everything that moves will eventually be autonomous'
[11]Ex-OpenAI board member accuses Sam Altman of 'outright lying'
"It's somewhat disappointing but expected that these models show no real generalization of intelligence or reasoning," he said. "While they can perform specific tasks well, they don't yet demonstrate a broad, adaptable understanding or problem-solving ability like human intelligence."
At their core, LLMs are still big bundles of statistical models just trying to write down what makes sense.
[12]
To illustrate this, Prelovac also tested LLMs in Connect Four, a game generally considered to be way easier than chess. However, even GPT-4o couldn't understand when it needed to block its opponent from connecting four pieces. Prelovac described its performance as on par with a four-year-old.
"The only conclusion is that this failure is due to the lack of historical records of played games in the training data," he said. "This implies that it cannot be argued that these models are able to 'reason' in any sense of the word, but merely output a variation of what they have seen during training."
As for why GPT-4o registered a remarkable improvement in chess but still made illegal moves, Prelovac speculated that perhaps its multi-modal training had something to do with it. It's possible part of OpenAI's training data included visuals of chess being played, which could help the AI visualize the board easier than it can with pure text.
Technically, when GPT-4o writes out the move it wants to play, it correctly formats it in Forsyth–Edwards notation (FEN), but the model doesn't understand that even if it makes sense, that doesn't mean it's the best move or even legal. When individual chess moves are broken up into multiple tokens, that can especially compromise an LLM's reasoning skills.
"Even chess moves are nothing but a series of tokens, like 'e' and '4', and have no grounding in reality," Prelovac said. "They are products of statistical analysis of the training data, upon which the next token is predicted."
The difference between pawn to e4 and pawn to e5 is clear to humans as both moves would put a game of chess down completely different paths and can't just be interchanged for each other. However, an LLM might just choose the next number based on statistics, hence why it might try to move a rook that doesn't exist on e1 instead of a real one that's on e2.
For the time being, no LLM is going to be able to play chess the way a chess engine like Stockfish can. However, more training data could potentially make LLMs more proficient, especially for opening moves and games where so few pieces are on the board that checkmate can be seen from just a few moves away. For the middle part of a game, a truly massive amount of data might be necessary to prevent an LLM from making illegal moves. ®
Get our [13]Tech Resources
[1] https://github.com/kagisearch/llm-chess-puzzles
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://regmedia.co.uk/2024/05/31/llm-chess-puzzle-correct.png
[7] https://regmedia.co.uk/2024/05/31/llm-chess-puzzle-illegal.png
[8] https://www.theregister.com/2024/05/31/google_ai_search_update/
[9] https://www.theregister.com/2024/05/30/openai_stops_five_ineffective_ai/
[10] https://www.theregister.com/2024/05/30/nvidia_ai_conference/
[11] https://www.theregister.com/2024/05/29/ex_openai_board_member_accuses_altman_lying/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zl7lw8m1Pxh4-YSwxonLiwAAAEo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Joe W
And in contrast to the article I would think that since it does not "think ahead" the LLM will not be able to spot a mate in two or three moves. Unless this has been in the training set, I guess.
Meh.
So to sum up ...
Mike 137
The LLM doesn't understand or reason , which are two key attributes of a good chess player (or indeed anyone doing anything really well).
What a surprise, considering how the system operates -- selecting the most statistically probable from a pool of essentially randomly chosen alternatives, without any reference to causalities or results.
So they can't play chess without waffling.
jake
Given that they can't really do much of anything else except waffle, is anybody really surprised?
Wake me up when they can do something useful.
As someone else has commented on this forum
Andy Non
ChatGPT and all the rest aren't true AI, they are just advanced predictive text generators, trained on a colossal amount of text and they are pretty good at predicting what words, sentences and paragraphs to suggest in response to an input. There is nothing truly intelligent about them.
The problem with all AI:
- You throw data at it, and "train it" (i.e. you kill off those who don't get increasingly closer to what you want) on the subject you desire.
- They become vaguely proficient at it.
To now train on another topic... you have to defeat all that training that you already gave it, by overwhelming it with other training until its initial training becomes a minority player in its behaviour. Which usually means orders-of-magnitude more training, starting from a very biased (culled) base of the thing you originally wanted it to do.
To advance - much of the same. Remember how in learning everything you initially learned is revealed to be not the complete truth? Same problem. Now you have to take your mediocre chess AI and untrain it on everything it learned (to even survive!) in order to become a better chess AI.
If you have the time, the processing, the source data (if you trained it on the entire Internet... where are you going to get orders of magnitude more training data and the time to train it on it?), you can retrain it but you'll still be held hostage by the initial criteria - those you culled in order to achieve.
Imagine you executed every lifeform that couldn't play chess on an interface that they can all use. Eventually, yes, you'd get an animal of some kind that can play chess on that interface. Now try to train that animal to launch a rocket. That millions of years of forced evolution takes a long time to undo - many generations, MANY individuals breeding constantly, until your "now incorrect" training is in the minority - and it will inherently bias the way that every creature that still exists thinks. Because they all came from a chess-playing creature.
Modern AI still hasn't learned these lessons, despite decades of the EXACT SAME PROBLEM. Intelligence isn't a statistical average of the training data. That's not how it works. I can see something *ONCE* and recognise that it's amazing and useful, and throw out decades of my previous knowledge and experience to follow it, because it's a clearly-better tool for the purpose I need. That's how intelligence works. I can reason things that don't yet exist and have no effect on my life - like I can choose to be compassionate to minority groups that I've never previously encountered just by thinking about this new group that I come across.
These things are just statistical probabilistic engines trained on databases. That's all they are. And AI people get rather offended when you point that out, because they still hold the belief that they are not because it uses genetic algorithms / neural networks / transformers / and don't see that all they're building is layers of abstraction from that. And yet actually intelligence does not rely on any such engine or database like that, in fact it's one of the defining features that we can extrapolate from almost zero previous information and imagine things that have never been recorded ever before.
It's 60 years down the line and we're still pushing the same nonsense, and getting the same result, and even proposing the same solution - MORE CPU! MORE RAM! MORE NODES! MORE TRAINING! MORE DATA SOURCES! That'll fix it, this time, for sure! Everyone knows that you just do random stuff millions of times over and it magically and spontaneously becomes intelligent at a given point! It's just that that point is always *just* out of reach, apparently.
Or maybe we could completely rethink what we're doing here. Because none of this solves the inference problem, the training plateaus, or the complete lack of understanding or conceptualisation of the underlying data.