Google claims Big Sleep 'first' AI to spot freshly committed security bug that fuzzing missed
(2024/11/05)
- Reference: 1730788693
- News link: https://www.theregister.co.uk/2024/11/05/google_ai_vulnerability_hunting/
- Source link:
Google claims one of its AI models is the first of its kind to spot a memory safety vulnerability in the wild – specifically an exploitable stack buffer underflow in SQLite – which was then fixed before the buggy code's official release.
The Chocolate Factory's LLM-based bug-hunting tool, dubbed Big Sleep, is a collaboration between Google's Project Zero and DeepMind. This software is said to be an evolution of earlier Project Naptime, announced in June.
SQLite is an open source database engine, and the stack buffer underflow [1]vulnerability could have allowed an attacker to cause a crash or perhaps even achieve arbitrary code execution. More specifically, the crash or code execution would happen in the SQLite executable (not the library) due to a magic value of -1 accidentally being used at one point as an array index. There is an assert() in the code to catch the use of -1 as an index, but in release builds, this debug-level check would be removed.
[2]
Thus, a miscreant could cause a crash or achieve code execution on a victim's machine by, perhaps, triggering that bad index bug with a maliciously crafted database shared with that user or through some SQL injection. Even the Googlers admit the flaw is non-trivial to exploit, so be aware that the severity of the hole is not really the news here – it's that the web giant believes its AI has scored a first.
[3]
[4]
We're told that fuzzing – feeding random and/or carefully crafted data into software to uncover exploitable bugs – didn't find the issue.
The LLM, however, did. According to Google, this is the first time an AI agent has found a previously unknown exploitable memory-safety flaw in widely used real-world software. After Big Sleep clocked the bug in early October, having been told to go through a bunch of commits to the project's source code, SQLite's developers [5]fixed it on the same day. Thus the flaw was removed before an official release.
[6]
"We think that this work has tremendous defensive potential," the Big Sleep team [7]crowed in a November 1 write-up. "Fuzzing has helped significantly, but we need an approach that can help defenders to find the bugs that are difficult (or impossible) to find by fuzzing, and we're hopeful that AI can narrow this gap."
We should note that in October, Seattle-based Protect AI [8]announced a free, open source tool that it claimed can find zero-day vulnerabilities in Python codebases with an assist from Anthropic's Claude AI model.
This tool is called Vulnhuntr and, according to its developers, it has found more than a dozen zero-day bugs in large, open source Python projects.
[9]
The two tools have different purposes, according to Google. "Our assertion in the blog post is that Big Sleep discovered the first unknown exploitable memory-safety issue in widely used real-world software," a Google spokesperson told The Register , with our emphasis added. "The Python LLM finds different types of bugs that aren't related to memory safety."
[10]Open source LLM tool primed to sniff out Python zero-days
[11]Cast a hex on ChatGPT to trick the AI into writing exploit code
[12]Why the long name? Okta discloses auth bypass bug affecting 52-character usernames
[13]Hack Nintendo's alarm clock to show cat pics? Let's-a-go!
Big Sleep, which is still in the research stage, has thus far used small programs with known vulnerabilities to evaluate its bug-finding prowess. This was its first real-world experiment.
For the test, the team collected several recent commits to the SQLite repository. After manually removing trivial and document-only changes, "we then adjusted the prompt to provide the agent with both the commit message and a diff for the change, and asked the agent to review the current repository (at [14]HEAD ) for related issues that might not have been fixed," the team wrote.
The LLM, based on Gemini 1.5 Pro, ultimately found the bug, which was loosely related to changes in the seed commit [15][1976c3f7] . "This is not uncommon in manual variant analysis, understanding one bug in a codebase often leads a researcher to other problems," the Googlers explained.
In the write-up, the Big Sleep team also detailed the "highlights" of the steps that the agent took to evaluate the code, find the vulnerability, crash the system, and then produce a root-cause analysis.
"However, we want to reiterate that these are highly experimental results," they wrote. "The position of the Big Sleep team is that at present, it's likely that a target-specific fuzzer would be at least as effective (at finding vulnerabilities)." ®
Get our [16]Tech Resources
[1] https://project-zero.issues.chromium.org/issues/372435124
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://sqlite.org/src/info/41d58a014ce89356
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://googleprojectzero.blogspot.com/2024/10/from-naptime-to-big-sleep.html
[8] https://www.theregister.com/2024/10/20/python_zero_day_tool/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/10/20/python_zero_day_tool/
[11] https://www.theregister.com/2024/10/29/chatgpt_hex_encoded_jailbreak/
[12] https://www.theregister.com/2024/11/04/why_the_long_name_okta/
[13] https://www.theregister.com/2024/11/01/hack_nintendos_alarmo/
[14] https://sqlite.org/src/info/2f7eab381e167609
[15] https://sqlite.org/src/info/1976c3f7e1fe77cf
[16] https://whitepapers.theregister.com/
The Chocolate Factory's LLM-based bug-hunting tool, dubbed Big Sleep, is a collaboration between Google's Project Zero and DeepMind. This software is said to be an evolution of earlier Project Naptime, announced in June.
SQLite is an open source database engine, and the stack buffer underflow [1]vulnerability could have allowed an attacker to cause a crash or perhaps even achieve arbitrary code execution. More specifically, the crash or code execution would happen in the SQLite executable (not the library) due to a magic value of -1 accidentally being used at one point as an array index. There is an assert() in the code to catch the use of -1 as an index, but in release builds, this debug-level check would be removed.
[2]
Thus, a miscreant could cause a crash or achieve code execution on a victim's machine by, perhaps, triggering that bad index bug with a maliciously crafted database shared with that user or through some SQL injection. Even the Googlers admit the flaw is non-trivial to exploit, so be aware that the severity of the hole is not really the news here – it's that the web giant believes its AI has scored a first.
[3]
[4]
We're told that fuzzing – feeding random and/or carefully crafted data into software to uncover exploitable bugs – didn't find the issue.
The LLM, however, did. According to Google, this is the first time an AI agent has found a previously unknown exploitable memory-safety flaw in widely used real-world software. After Big Sleep clocked the bug in early October, having been told to go through a bunch of commits to the project's source code, SQLite's developers [5]fixed it on the same day. Thus the flaw was removed before an official release.
[6]
"We think that this work has tremendous defensive potential," the Big Sleep team [7]crowed in a November 1 write-up. "Fuzzing has helped significantly, but we need an approach that can help defenders to find the bugs that are difficult (or impossible) to find by fuzzing, and we're hopeful that AI can narrow this gap."
We should note that in October, Seattle-based Protect AI [8]announced a free, open source tool that it claimed can find zero-day vulnerabilities in Python codebases with an assist from Anthropic's Claude AI model.
This tool is called Vulnhuntr and, according to its developers, it has found more than a dozen zero-day bugs in large, open source Python projects.
[9]
The two tools have different purposes, according to Google. "Our assertion in the blog post is that Big Sleep discovered the first unknown exploitable memory-safety issue in widely used real-world software," a Google spokesperson told The Register , with our emphasis added. "The Python LLM finds different types of bugs that aren't related to memory safety."
[10]Open source LLM tool primed to sniff out Python zero-days
[11]Cast a hex on ChatGPT to trick the AI into writing exploit code
[12]Why the long name? Okta discloses auth bypass bug affecting 52-character usernames
[13]Hack Nintendo's alarm clock to show cat pics? Let's-a-go!
Big Sleep, which is still in the research stage, has thus far used small programs with known vulnerabilities to evaluate its bug-finding prowess. This was its first real-world experiment.
For the test, the team collected several recent commits to the SQLite repository. After manually removing trivial and document-only changes, "we then adjusted the prompt to provide the agent with both the commit message and a diff for the change, and asked the agent to review the current repository (at [14]HEAD ) for related issues that might not have been fixed," the team wrote.
The LLM, based on Gemini 1.5 Pro, ultimately found the bug, which was loosely related to changes in the seed commit [15][1976c3f7] . "This is not uncommon in manual variant analysis, understanding one bug in a codebase often leads a researcher to other problems," the Googlers explained.
In the write-up, the Big Sleep team also detailed the "highlights" of the steps that the agent took to evaluate the code, find the vulnerability, crash the system, and then produce a root-cause analysis.
"However, we want to reiterate that these are highly experimental results," they wrote. "The position of the Big Sleep team is that at present, it's likely that a target-specific fuzzer would be at least as effective (at finding vulnerabilities)." ®
Get our [16]Tech Resources
[1] https://project-zero.issues.chromium.org/issues/372435124
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://sqlite.org/src/info/41d58a014ce89356
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://googleprojectzero.blogspot.com/2024/10/from-naptime-to-big-sleep.html
[8] https://www.theregister.com/2024/10/20/python_zero_day_tool/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zyn614V9VxBt4bCF0GpNXwAAAIM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.theregister.com/2024/10/20/python_zero_day_tool/
[11] https://www.theregister.com/2024/10/29/chatgpt_hex_encoded_jailbreak/
[12] https://www.theregister.com/2024/11/04/why_the_long_name_okta/
[13] https://www.theregister.com/2024/11/01/hack_nintendos_alarmo/
[14] https://sqlite.org/src/info/2f7eab381e167609
[15] https://sqlite.org/src/info/1976c3f7e1fe77cf
[16] https://whitepapers.theregister.com/
Infinite Number of Monkeys
Anonymous Coward
Syndrome.
Re: Infinite Number of Monkeys
Bebu sa Ware
Just read recently that apparently not: [1]monkeys-do-not-have-the-time .
Although clearly you might train in finite time a single (surviving) monkey to transcribe Shakespeare's Hamlet using a typewriter. Think brain implants and lots of pain and a adequate supply of monkeys.
[1] https://www.sciencealert.com/scientists-confirm-monkeys-do-not-have-the-time-to-write-shakespeare
Fuzzing?
Anonymous Coward
I'm struggling to think of how any adequate fuzzing wouldn't have caught this. If the fuzzing was done and didn't catch this one, then I'd worry about all the other similar problems that neither the fuzzing nor the AI caught.
Removing assert() in production
Bebu sa Ware
I recall many, many years ago reading in the work of one of Djisktra, Brinch Hansen or Wirth that removing bounds checks or correctness assertions (eg invariant checks) from production code was akin to training with parachutes but flying into combat without.
The only logical reason for removing assert()s from production code is where it is proven that b-expr in assert ( b-expr ) evaluates to the constant true .
Decent compiler/linker for a language less shabby than most could probably optimise away most assertions.
Using magic values outside the domain (of an array, of a function...) can frequently end in tears - nul terminated strings are probably responsible an ocean or two, (-1) for 0-based arrays a couple of lakes (array indices are often unsigned and if are 32 bit on a 64 bit system (-1) is interpreted as a large positive offset (232-1) while being in the boonies could be a valid location in the process' memory.)
How many false positives though?
It rather makes me wonder if it simply listed every assert()