Red teams are safe from robots for now, as AI makes better shield than spear
(2025/08/11)
- Reference: 1754929392
- News link: https://www.theregister.co.uk/2025/08/11/ai_security_offense_defense/
- Source link:
Black Hat/DEF CON At the opening of Black Hat, the largest security shindig in the Hacker Summer Camp week ahead of DEF CON and BSides, the opening keynote speaker suggested the current state of AI slightly favors defenders over attackers, but he warned that was not a given for much longer.
"I do believe that AI is the key [in security] because that's one of the few fields where defenders are ahead of the attackers," Mikko Hyppönen, [1]outgoing chief research officer for Finnish security firm WithSecure, told the audience.
"All the cybersecurity companies here will tell you how extensively they use generative AI in their products. Yes, attackers are using AI as well, but they're only beginning. We've only seen fairly simple attacks with AI so far. It will change, but right now, I would claim we are ahead."
[2]
Hyppönen pointed out that in 2024 AI systems discovered no zero-day vulnerabilities – that he knew about. So far in 2025, researchers have spotted around two dozen using LLM scanning, all of which have been fixed, but he warned that hackers are now increasingly using AI to do such research and that they are bound to find more.
[3]
[4]
In a later Black Hat keynote, Nicole Perlroth, former New York Times security correspondent and now a partner at venture capital biz Silver Buckshot Ventures, disagreed, saying that by next year: "On this question of whether AI will favor defense or offense, early signs suggest that offense is going to have the advantage."
Then again, she also said there were 500,000 vacancies in the US security industry, which provoked hollow laughter from some in this [5]difficult job market.
[6]
Over the course of the week we've been polling those in the field, vendors, penetration testers, and those on the security coal face, and, while many are positive about AI's contribution, there are concerns that the halcyon days won't last.
Pentesters positive, but skeptical
Firstly, no one we spoke to is predicting AIs being able to attack a network unaided, at least not for a decade or so.
More than a few presentations over the week covered the use of AI tools for red teaming – carrying out penetration testing to simulate an attack on a target network. But it's simply not seen as trustworthy and is prone to basic mistakes if used incorrectly.
Charles Henderson, an executive veep at cybersecurity at Coalfire, said his business was using AI tools but that, without humans at the helm, their effectiveness was severely limited.
[7]Enterprises neglect AI security – and attackers have noticed
[8]Rampant emoji use suggests crypto-stealing NPM package was written by AI
[9]Chinese biz using AI to hit US politicians, influencers with propaganda
[10]UK's Ministry of Defence pins hopes on AI to stop the next massive email blunder
"Properly directed, AI does about 60 percent of the job, so it's great if you're trying to lighten the load for your people, but it's horrible if you're trying to turn over a mission to it," he told The Register .
He explained that AI had its strengths, particularly in the detection of flaws, but that its implementation was crucial. Simply throwing AI tools at a target wasn't helpful and some in the industry had been overselling its abilities. It's easy to deploy – that's just a matter of buying up tokens – but it's much harder to use well.
Chris Yule, director of threat research at the Sophos Cyber Threat Unit, took a similar view. For red teams, he said, the ideal scenario is using it to augment human skills rather than try and replace them. Machine learning systems need to be set with clear, limited goals and then guided by human controllers for best use, he suggested.
[11]
At the same time, red teaming in this way is showing how future criminals will try and use these systems and it seems the security community is getting its attacks in first to model what the villains might try next.
And then there's defense
The US government, via its military research arm the Defense Advanced Research Projects Agency, is clearly looking towards AI as a tool for defense, and proved it financially by [12]awarding $8.5 million to three teams competing in its AI Cyber Challenge.
The idea was to create an AI system that can identify vulnerabilities and then patch them in a way that doesn't crash the network. The two-year contest whittled down 42 teams to just seven in a very tough competition, a DARPA spokesperson told us, and in the end the winning team was a combined US and South Korean effort made up of the America's finest at Georgia Tech, Samsung Research, the Korea Advanced Institute of Science & Technology (KAIST), and the Pohang University of Science and Technology (POSTECH).
The contest organizers introduced 70 vulnerabilities into the test dataset, with teams discovering 54 and successfully patching 43 of them. But, crucially, the systems also discovered another 18 vulnerabilities that weren't added – six in C and 12 in Java – and managed to patch 11 of them.
If you're wondering if this is a good use of taxpayers' money, the code is now open source for anyone to use. More contests are planned in the future.
Elsewhere in defense, several sources noted that AI was particularly adept at spotting SQL vulnerabilities. This may be down to training data, since SQL flaws are commonplace, but that's encouraging considering their prevalence.
When it comes to jobs, AI might be an excuse
One of the biggest issues on the minds of attendees was the impact AI systems would have on the job market.
While it's true that companies have been shedding security staff, particularly for entry-level positions, others felt the gap was not being addressed.
"I think the impact of AI on the security jobs market has been overstated," Chris Yule, director of threat research at the Sophos Cyber Threat Unit, told The Register , adding that AI is being used by some as a marketing excuse for laying off staff.
He added that while AI has its uses, using it to replace whole tranches of security workers wasn't really feasible given the current state of the technology, and may never be. Others agree.
"You're never going to get past the human factor," one CISO said, off the record. "[AI systems] are fine for crunching through data but human ingenuity is a tough sell at the moment, but that may change. But I trust my coders more than I trust an AI."
The fact of the matter is that AI serves as a useful augmentation tool, for now. What comes down the line, as models improve, is a much more fluid situation and no one's willing to take bets either way. ®
Get our [13]Tech Resources
[1] https://www.theregister.com/2025/06/04/mikko_hypponen_drone/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2025/03/03/cybersecurity_jobs_market/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2025/07/30/firms_are_neglecting_ai_security/
[8] https://www.theregister.com/2025/08/01/emoji_use_ai_malware/
[9] https://www.theregister.com/2025/08/08/golaxy_ai_influence/
[10] https://www.theregister.com/2025/08/06/mod_taps_aussie_ai_shop/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://aicyberchallenge.com/
[13] https://whitepapers.theregister.com/
"I do believe that AI is the key [in security] because that's one of the few fields where defenders are ahead of the attackers," Mikko Hyppönen, [1]outgoing chief research officer for Finnish security firm WithSecure, told the audience.
"All the cybersecurity companies here will tell you how extensively they use generative AI in their products. Yes, attackers are using AI as well, but they're only beginning. We've only seen fairly simple attacks with AI so far. It will change, but right now, I would claim we are ahead."
[2]
Hyppönen pointed out that in 2024 AI systems discovered no zero-day vulnerabilities – that he knew about. So far in 2025, researchers have spotted around two dozen using LLM scanning, all of which have been fixed, but he warned that hackers are now increasingly using AI to do such research and that they are bound to find more.
[3]
[4]
In a later Black Hat keynote, Nicole Perlroth, former New York Times security correspondent and now a partner at venture capital biz Silver Buckshot Ventures, disagreed, saying that by next year: "On this question of whether AI will favor defense or offense, early signs suggest that offense is going to have the advantage."
Then again, she also said there were 500,000 vacancies in the US security industry, which provoked hollow laughter from some in this [5]difficult job market.
[6]
Over the course of the week we've been polling those in the field, vendors, penetration testers, and those on the security coal face, and, while many are positive about AI's contribution, there are concerns that the halcyon days won't last.
Pentesters positive, but skeptical
Firstly, no one we spoke to is predicting AIs being able to attack a network unaided, at least not for a decade or so.
More than a few presentations over the week covered the use of AI tools for red teaming – carrying out penetration testing to simulate an attack on a target network. But it's simply not seen as trustworthy and is prone to basic mistakes if used incorrectly.
Charles Henderson, an executive veep at cybersecurity at Coalfire, said his business was using AI tools but that, without humans at the helm, their effectiveness was severely limited.
[7]Enterprises neglect AI security – and attackers have noticed
[8]Rampant emoji use suggests crypto-stealing NPM package was written by AI
[9]Chinese biz using AI to hit US politicians, influencers with propaganda
[10]UK's Ministry of Defence pins hopes on AI to stop the next massive email blunder
"Properly directed, AI does about 60 percent of the job, so it's great if you're trying to lighten the load for your people, but it's horrible if you're trying to turn over a mission to it," he told The Register .
He explained that AI had its strengths, particularly in the detection of flaws, but that its implementation was crucial. Simply throwing AI tools at a target wasn't helpful and some in the industry had been overselling its abilities. It's easy to deploy – that's just a matter of buying up tokens – but it's much harder to use well.
Chris Yule, director of threat research at the Sophos Cyber Threat Unit, took a similar view. For red teams, he said, the ideal scenario is using it to augment human skills rather than try and replace them. Machine learning systems need to be set with clear, limited goals and then guided by human controllers for best use, he suggested.
[11]
At the same time, red teaming in this way is showing how future criminals will try and use these systems and it seems the security community is getting its attacks in first to model what the villains might try next.
And then there's defense
The US government, via its military research arm the Defense Advanced Research Projects Agency, is clearly looking towards AI as a tool for defense, and proved it financially by [12]awarding $8.5 million to three teams competing in its AI Cyber Challenge.
The idea was to create an AI system that can identify vulnerabilities and then patch them in a way that doesn't crash the network. The two-year contest whittled down 42 teams to just seven in a very tough competition, a DARPA spokesperson told us, and in the end the winning team was a combined US and South Korean effort made up of the America's finest at Georgia Tech, Samsung Research, the Korea Advanced Institute of Science & Technology (KAIST), and the Pohang University of Science and Technology (POSTECH).
The contest organizers introduced 70 vulnerabilities into the test dataset, with teams discovering 54 and successfully patching 43 of them. But, crucially, the systems also discovered another 18 vulnerabilities that weren't added – six in C and 12 in Java – and managed to patch 11 of them.
If you're wondering if this is a good use of taxpayers' money, the code is now open source for anyone to use. More contests are planned in the future.
Elsewhere in defense, several sources noted that AI was particularly adept at spotting SQL vulnerabilities. This may be down to training data, since SQL flaws are commonplace, but that's encouraging considering their prevalence.
When it comes to jobs, AI might be an excuse
One of the biggest issues on the minds of attendees was the impact AI systems would have on the job market.
While it's true that companies have been shedding security staff, particularly for entry-level positions, others felt the gap was not being addressed.
"I think the impact of AI on the security jobs market has been overstated," Chris Yule, director of threat research at the Sophos Cyber Threat Unit, told The Register , adding that AI is being used by some as a marketing excuse for laying off staff.
He added that while AI has its uses, using it to replace whole tranches of security workers wasn't really feasible given the current state of the technology, and may never be. Others agree.
"You're never going to get past the human factor," one CISO said, off the record. "[AI systems] are fine for crunching through data but human ingenuity is a tough sell at the moment, but that may change. But I trust my coders more than I trust an AI."
The fact of the matter is that AI serves as a useful augmentation tool, for now. What comes down the line, as models improve, is a much more fluid situation and no one's willing to take bets either way. ®
Get our [13]Tech Resources
[1] https://www.theregister.com/2025/06/04/mikko_hypponen_drone/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2025/03/03/cybersecurity_jobs_market/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2025/07/30/firms_are_neglecting_ai_security/
[8] https://www.theregister.com/2025/08/01/emoji_use_ai_malware/
[9] https://www.theregister.com/2025/08/08/golaxy_ai_influence/
[10] https://www.theregister.com/2025/08/06/mod_taps_aussie_ai_shop/
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cso&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aJpn-UQhL9a1kkOpVVbHXAAAABE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[12] https://aicyberchallenge.com/
[13] https://whitepapers.theregister.com/