Tackling potty-mouth chatbots to leaky LLMs. What's life like in Microsoft's AI red team?
(2024/06/05)
- Reference: 1717585930
- News link: https://www.theregister.co.uk/2024/06/05/microsoft_ai_red_team_tackles/
- Source link:
Interview Sometimes people want to abuse AI systems to leak corporate secrets. Other times they just want to force an LLM to talk like a pirate.
Either way, it's the AI red team's job to anticipate how someone may try to [1]weaponize these systems , and then help address those the issues.
Ram Shankar Siva Kumar [2]founded Microsoft's AI red team in 2019, long before the rise of ChatGPT. Kumar and his colleagues have encountered various AI failures, and you can hear about them and more in an interview with The Register below.
[3]
[4]Youtube Video
[5]
"The common misconception is: AI red teams can replace all traditional red teaming," he said. "That's absolutely not the case."
That being said, there are certain new-ish concerns that are very specific to AI and go beyond more traditional security failures. This includes things like [6]biases in ML models or triggers for [7]hallucinations .
[8]
In finding and fixing these failures, Kumar says it's useful to assume different attacker personas, and in our discussion he gave examples of each.
"The goal of the AI red team is not just to think about what is possible today with the current state of adversaries," he said. "We also want to push the boundaries [with] attacks that have not yet manifested." ®
Get our [9]Tech Resources
[1] https://www.theregister.com/2024/05/13/ai_xforce_red_penetration/
[2] https://www.theregister.com/2023/04/26/is_your_ai_hallucinating/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://youtu.be/11oj--a-Sgw
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/03/11/ai_models_exhibit_racism_based/
[7] https://www.theregister.com/2023/08/19/chatgpt_dnd_dm/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://whitepapers.theregister.com/
Either way, it's the AI red team's job to anticipate how someone may try to [1]weaponize these systems , and then help address those the issues.
Ram Shankar Siva Kumar [2]founded Microsoft's AI red team in 2019, long before the rise of ChatGPT. Kumar and his colleagues have encountered various AI failures, and you can hear about them and more in an interview with The Register below.
[3]
[4]Youtube Video
[5]
"The common misconception is: AI red teams can replace all traditional red teaming," he said. "That's absolutely not the case."
That being said, there are certain new-ish concerns that are very specific to AI and go beyond more traditional security failures. This includes things like [6]biases in ML models or triggers for [7]hallucinations .
[8]
In finding and fixing these failures, Kumar says it's useful to assume different attacker personas, and in our discussion he gave examples of each.
"The goal of the AI red team is not just to think about what is possible today with the current state of adversaries," he said. "We also want to push the boundaries [with] attacks that have not yet manifested." ®
Get our [9]Tech Resources
[1] https://www.theregister.com/2024/05/13/ai_xforce_red_penetration/
[2] https://www.theregister.com/2023/04/26/is_your_ai_hallucinating/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://youtu.be/11oj--a-Sgw
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/03/11/ai_models_exhibit_racism_based/
[7] https://www.theregister.com/2023/08/19/chatgpt_dnd_dm/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZmCLo32I2zNRPMmDxkR9mQAAANU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://whitepapers.theregister.com/
It'd be interesting to know if Recall is a project that the AI red team was not invited to take a look at.