What if AI produces code not just quickly but also, dunno, securely, DARPA wonders
(2024/04/02)
- Reference: 1712084412
- News link: https://www.theregister.co.uk/2024/04/02/ai_dominates_at_darpa_and/
- Source link:
A DARPA leader has revealed that around 70 percent of the US government agency's programs involve AI in some shape or form, and those projects could have serious ramifications for the future of jobs in software development.
Speaking at a Center for Strategic and International Studies event last week, Dr Matt Turek, deputy director of DARPA's Information Innovation Office (I2O), talked about a wide array of AI projects DARPA is working on and the overwhelming dominance of this technology within the agency currently.
"There is really broad penetration across the agency," Turek [1]said . "From an I2O perspective we're really looking to try and advance, you know, how do we get to highly trustworthy AI – AI that we can bet our lives on – and that not be a foolish thing to do."
[2]
The I2O currently has four [3]research thrusts : Proficient AI; resilient, adaptable and secure systems; advantage in cyber operations; and confidence in the information domain. Only one of those four thrusts directly mentions AI, but that doesn't mean it isn't involved in all of them.
[4]
[5]
"There's a lot of synergies across those thrust areas," Turek stated. "We have efforts that are blending both advancing AI and advancing the state of capability in cyber … I think it's worth saying that AI and autonomy is really being used broadly across the agency now."
ChatGPT creates mostly insecure code, but won't tell you unless you ask [6]READ MORE
While many of the AI projects at DARPA are focused on how the technology can benefit the Department of Defense, that's hardly the only focus area, nor is I2O limiting its research to staying ahead of the US's military adversaries.
"It's not just [the] US government that needs to have these capabilities. The attack surface is broad," Turek said.
Citing the importance of commercial industries like scientific research, critical infrastructure and even online commerce to national security, Turek said I2O wants to "create commercial industry in this space" through its research.
[7]
One of the key ways to do that, according to Turek, is developing artificial intelligence that can not only write code, but do it in a secure and "provably correct" manner. We all know today's LLMs have a habit of [8]inventing bad or insecure code.
"There's really interesting use cases that our commercial industry is pursuing now around using LLMs to help with the code generation process," Turek said. "But what if we could make it so that they produce not just code more quickly, but secure code?"
"That would allow us to scale out, you know, robust, secure software development processes," Turek said, noting it's a critical concept for the Department of Defense, but a concept area, not an actual area of investment – yet.
[9]Simon Willison interview: AI software still needs the human touch
[10]If you use AI to teach you how to code, remember you still need to think for yourself
[11]How DARPA wants to rethink the fundamentals of AI to include trust
[12]Mamas, don't let your babies grow up to be coders, Jensen Huang warns
While AI isn't writing secure code for DARPA or commercial industries yet, the agency is seeking solutions to turn it toward examining existing software for vulnerabilities. That initiative, the AI Cyber Challenge, was [13]discussed last year at Black Hat, and Turek mentioned it last week as well, saying it's looking for vulnerabilities in critical infrastructure software and [14]open source projects .
Developers aren't the only category of tech professionals that DARPA's AI initiatives could endanger, though. During his talk, Turek also mentioned the [15]CASTLE program , an I2O initiative training autonomous AI agents to handle network security. At the outside end of the program, Turek said CASTLE AI agents would ideally be able to prevent the need to rebuild networks during an APT compromise, which he noted often results in the need to "start from scratch and rebuild."
[16]
"CASTLE is really focused on trying to build those sorts of automated defensive agents that, again, can preserve some level of critical network functions," Turek said.
Another program, [17]PROVERS , is seeking to use AI to guide software development toward the development of "proof-friendly" systems.
All of this relies on developing AI that is itself understandable in its processes – something that Turek admits isn't quite there yet.
"Modern statistical machine learning approaches oftentimes are opaque and they're not introspectable," Turek said. "I still feel like there's a lot of work that needs to be done."
So don't worry about an AI taking your software development job yet – we've seen [18]plenty of examples of AI developing lousy code, but that doesn't mean the tech won't be pawned off on developers anyways. It's [19]just a matter of time . ®
Get our [20]Tech Resources
[1] https://www.csis.org/analysis/darpa-perspective-ai-and-autonomy-dod
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.darpa.mil/work-with-us/i2o-thrust-areas
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/04/21/chatgpt_insecure_code/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/12/21/ai_assistants_bad_code/
[9] https://www.theregister.com/2024/01/24/willison_ai_software_development/
[10] https://www.theregister.com/2024/01/27/ai_coding_automatic/
[11] https://www.theregister.com/2023/04/20/darpa_ai_trust/
[12] https://www.theregister.com/2024/02/27/jensen_huang_coders/
[13] https://www.theregister.com/2023/08/09/darpa_aixcc/
[14] https://www.theregister.com/2024/04/01/xz_backdoor_open_source/
[15] https://www.darpa.mil/program/cyber-agents-for-security-testing-and-learning-environments
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.darpa.mil/program/pipelined-reasoning-of-verifiers-enabling-robust-systems
[18] https://www.theregister.com/2022/12/21/ai_assistants_bad_code/
[19] https://www.theregister.com/2024/02/12/opinion_column_on_forcing_ai_features_on_developers/
[20] https://whitepapers.theregister.com/
Speaking at a Center for Strategic and International Studies event last week, Dr Matt Turek, deputy director of DARPA's Information Innovation Office (I2O), talked about a wide array of AI projects DARPA is working on and the overwhelming dominance of this technology within the agency currently.
"There is really broad penetration across the agency," Turek [1]said . "From an I2O perspective we're really looking to try and advance, you know, how do we get to highly trustworthy AI – AI that we can bet our lives on – and that not be a foolish thing to do."
[2]
The I2O currently has four [3]research thrusts : Proficient AI; resilient, adaptable and secure systems; advantage in cyber operations; and confidence in the information domain. Only one of those four thrusts directly mentions AI, but that doesn't mean it isn't involved in all of them.
[4]
[5]
"There's a lot of synergies across those thrust areas," Turek stated. "We have efforts that are blending both advancing AI and advancing the state of capability in cyber … I think it's worth saying that AI and autonomy is really being used broadly across the agency now."
ChatGPT creates mostly insecure code, but won't tell you unless you ask [6]READ MORE
While many of the AI projects at DARPA are focused on how the technology can benefit the Department of Defense, that's hardly the only focus area, nor is I2O limiting its research to staying ahead of the US's military adversaries.
"It's not just [the] US government that needs to have these capabilities. The attack surface is broad," Turek said.
Citing the importance of commercial industries like scientific research, critical infrastructure and even online commerce to national security, Turek said I2O wants to "create commercial industry in this space" through its research.
[7]
One of the key ways to do that, according to Turek, is developing artificial intelligence that can not only write code, but do it in a secure and "provably correct" manner. We all know today's LLMs have a habit of [8]inventing bad or insecure code.
"There's really interesting use cases that our commercial industry is pursuing now around using LLMs to help with the code generation process," Turek said. "But what if we could make it so that they produce not just code more quickly, but secure code?"
"That would allow us to scale out, you know, robust, secure software development processes," Turek said, noting it's a critical concept for the Department of Defense, but a concept area, not an actual area of investment – yet.
[9]Simon Willison interview: AI software still needs the human touch
[10]If you use AI to teach you how to code, remember you still need to think for yourself
[11]How DARPA wants to rethink the fundamentals of AI to include trust
[12]Mamas, don't let your babies grow up to be coders, Jensen Huang warns
While AI isn't writing secure code for DARPA or commercial industries yet, the agency is seeking solutions to turn it toward examining existing software for vulnerabilities. That initiative, the AI Cyber Challenge, was [13]discussed last year at Black Hat, and Turek mentioned it last week as well, saying it's looking for vulnerabilities in critical infrastructure software and [14]open source projects .
Developers aren't the only category of tech professionals that DARPA's AI initiatives could endanger, though. During his talk, Turek also mentioned the [15]CASTLE program , an I2O initiative training autonomous AI agents to handle network security. At the outside end of the program, Turek said CASTLE AI agents would ideally be able to prevent the need to rebuild networks during an APT compromise, which he noted often results in the need to "start from scratch and rebuild."
[16]
"CASTLE is really focused on trying to build those sorts of automated defensive agents that, again, can preserve some level of critical network functions," Turek said.
Another program, [17]PROVERS , is seeking to use AI to guide software development toward the development of "proof-friendly" systems.
All of this relies on developing AI that is itself understandable in its processes – something that Turek admits isn't quite there yet.
"Modern statistical machine learning approaches oftentimes are opaque and they're not introspectable," Turek said. "I still feel like there's a lot of work that needs to be done."
So don't worry about an AI taking your software development job yet – we've seen [18]plenty of examples of AI developing lousy code, but that doesn't mean the tech won't be pawned off on developers anyways. It's [19]just a matter of time . ®
Get our [20]Tech Resources
[1] https://www.csis.org/analysis/darpa-perspective-ai-and-autonomy-dod
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.darpa.mil/work-with-us/i2o-thrust-areas
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/04/21/chatgpt_insecure_code/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/12/21/ai_assistants_bad_code/
[9] https://www.theregister.com/2024/01/24/willison_ai_software_development/
[10] https://www.theregister.com/2024/01/27/ai_coding_automatic/
[11] https://www.theregister.com/2023/04/20/darpa_ai_trust/
[12] https://www.theregister.com/2024/02/27/jensen_huang_coders/
[13] https://www.theregister.com/2023/08/09/darpa_aixcc/
[14] https://www.theregister.com/2024/04/01/xz_backdoor_open_source/
[15] https://www.darpa.mil/program/cyber-agents-for-security-testing-and-learning-environments
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zgx-@b89TLhCIzQD1KhyEgAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.darpa.mil/program/pipelined-reasoning-of-verifiers-enabling-robust-systems
[18] https://www.theregister.com/2022/12/21/ai_assistants_bad_code/
[19] https://www.theregister.com/2024/02/12/opinion_column_on_forcing_ai_features_on_developers/
[20] https://whitepapers.theregister.com/
And probably an improvement over devs looking at 100,000 line packages
elDog
and trying to figure out if a single change will impact thousands of other lines of code, and given this is usually DoD - how it would impact our national defense.
I've spent too many years looking at huge projects with so many dependencies and hundreds of devs that come and go and don't understand the entire code-base - they can't!
I understand that fears of AI failures are real and have been seen. But the same/worse happens with a developer who doesn't have the mental bandwidth to be perfect (that's me.)
What if?
ecofeco
That's a BIG whatif.
The safe way to bet? It won't.
Why? Same reason as today: too much GIGO.
Greetings Professor Falken.
Tron
Shall we play a game?
Probably
"..developing artificial intelligence that can not only write code, but do it in a secure and "provably correct" manner."
I read that as "probably correct". That's what we normally get out of AI. Provably would be an improvement (literally).