OpenAI is very smug after thwarting five ineffective AI covert influence ops
(2024/05/31)
- Reference: 1717111753
- News link: https://www.theregister.co.uk/2024/05/30/openai_stops_five_ineffective_ai/
- Source link:
OpenAI on Thursday said it has disrupted five covert influence operations that were attempting to use its AI services to manipulate public opinion and elections.
These influence operations (IOs), the super lab said, did not have a significant effect on audience engagement or in amplifying the reach of the manipulative messages.
"Over the last three months, our work against IO actors has disrupted covert influence operations that sought to use AI models for a range of tasks, such as generating short comments and longer articles in a range of languages, making up names and bios for social media accounts, conducting open-source research, debugging simple code, and translating and proofreading texts," the biz [1]said .
[2]
The campaigns have been linked to two operations in Russia, one in China, one in Iran, and a commercial company in Israel.
[3]
[4]
One operation out of Russia dubbed "Bad Grammar" focused on Telegram, targeting people in Ukraine, Moldova, the Baltic States, and the United States. The other, known as "Doppelganger," posted content about Ukraine on various internet sites.
The Chinese threat actor, referred to as "Spamouflage," praised China and slammed critics of the country.
[5]
The influence operation from Iran, known as the International Union of Virtual Media, celebrated Iran and condemned Israel and the US.
And the Israel-based firm STOIC created content about the Gaza conflict and Histadrut, Israel's trade union.
According to OpenAI, these manipulation schemes rated only two on the Brookings’ [6]Breakout Scale , a scheme to quantify the impact of IOs that ranges from one (spreads within one community on a single platform) to six (provokes a policy response or violence). A two on this scale means the fake content appeared on multiple platforms, with no breakout to authentic audiences.
[7]
The OpenAI [8]report [PDF] into this whole affair finds that these influence operations are often given away by errors their human operators have failed to address. "For example, Bad Grammar posted content that included refusal messages from our model, exposing their content as AI-generated," the report says.
"We all expected bad actors to use LLMs to boost their covert influence campaigns — none of us expected the first exposed AI-powered disinformation attempts to be this weak and ineffective," observed Thomas Rid, professor of strategic studies and founding director of the Alperovitch Institute for Cybersecurity Studies at Johns Hopkins University’s School of Advanced International Studies, in a social media [9]post .
[10]Palantir bags $480M from Pentagon to inject battlefield AI into the military
[11]Salesforce expects lowest quarterly growth in two decades
[12]AI future: Nvidia boffin hopes 'everything that moves will eventually be autonomous'
[13]Blame the SSD price hike on enterprise demand for AI servers
OpenAI's determination that these AI-powered covert influence campaigns were ineffective was echoed in a May 2024 [14]report on UK election interference by The Centre for Emerging Technology and Security (CETaS) at The Alan Turing Institute.
"The current impact of AI on specific election results is limited, but these threats show signs of damaging the broader democratic system," the CETaS report found, noting that of 112 national elections that have either taken place since January 2023 or will occur in 2024, AI-based meddling was detected in just 19 and there's no data yet to suggest election results were materially swayed by AI.
That said, the CETaS report argues that AI content creates second-order risks, such as sowing distrust and inciting hate, that are difficult to measure and have uncertain consequences.
Rid suggested that as more competitors develop tools for synthetic content creation and OpenAI's share of the market declines, the Microsoft-championed lab will be less able to detect abuses of this sort. He also noted that OpenAI, in its discussion of IOs, doesn't address other forms of synthetic content abuse, including fake product reviews, ad bots, fraudulent marketing copy, and phishing messages. ®
Get our [15]Tech Resources
[1] https://openai.com/index/disrupting-deceptive-uses-of-AI-by-covert-influence-operations/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.brookings.edu/articles/the-breakout-scale-measuring-the-impact-of-influence-operations/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://downloads.ctfassets.net/kftzwdyauwt9/5IMxzTmUclSOAcWUXbkVrK/3cfab518e6b10789ab8843bcca18b633/Threat_Intel_Report.pdf
[9] https://x.com/RidT/status/1796251141124276704
[10] https://www.theregister.com/2024/05/30/palantir_us_army/
[11] https://www.theregister.com/2024/05/30/salesforce_results/
[12] https://www.theregister.com/2024/05/30/nvidia_ai_conference/
[13] https://www.theregister.com/2024/05/30/blame_those_rising_ssd_prices/
[14] https://cetas.turing.ac.uk/publications/ai-enabled-influence-operations-threat-uk-general-election
[15] https://whitepapers.theregister.com/
These influence operations (IOs), the super lab said, did not have a significant effect on audience engagement or in amplifying the reach of the manipulative messages.
"Over the last three months, our work against IO actors has disrupted covert influence operations that sought to use AI models for a range of tasks, such as generating short comments and longer articles in a range of languages, making up names and bios for social media accounts, conducting open-source research, debugging simple code, and translating and proofreading texts," the biz [1]said .
[2]
The campaigns have been linked to two operations in Russia, one in China, one in Iran, and a commercial company in Israel.
[3]
[4]
One operation out of Russia dubbed "Bad Grammar" focused on Telegram, targeting people in Ukraine, Moldova, the Baltic States, and the United States. The other, known as "Doppelganger," posted content about Ukraine on various internet sites.
The Chinese threat actor, referred to as "Spamouflage," praised China and slammed critics of the country.
[5]
The influence operation from Iran, known as the International Union of Virtual Media, celebrated Iran and condemned Israel and the US.
And the Israel-based firm STOIC created content about the Gaza conflict and Histadrut, Israel's trade union.
According to OpenAI, these manipulation schemes rated only two on the Brookings’ [6]Breakout Scale , a scheme to quantify the impact of IOs that ranges from one (spreads within one community on a single platform) to six (provokes a policy response or violence). A two on this scale means the fake content appeared on multiple platforms, with no breakout to authentic audiences.
[7]
The OpenAI [8]report [PDF] into this whole affair finds that these influence operations are often given away by errors their human operators have failed to address. "For example, Bad Grammar posted content that included refusal messages from our model, exposing their content as AI-generated," the report says.
"We all expected bad actors to use LLMs to boost their covert influence campaigns — none of us expected the first exposed AI-powered disinformation attempts to be this weak and ineffective," observed Thomas Rid, professor of strategic studies and founding director of the Alperovitch Institute for Cybersecurity Studies at Johns Hopkins University’s School of Advanced International Studies, in a social media [9]post .
[10]Palantir bags $480M from Pentagon to inject battlefield AI into the military
[11]Salesforce expects lowest quarterly growth in two decades
[12]AI future: Nvidia boffin hopes 'everything that moves will eventually be autonomous'
[13]Blame the SSD price hike on enterprise demand for AI servers
OpenAI's determination that these AI-powered covert influence campaigns were ineffective was echoed in a May 2024 [14]report on UK election interference by The Centre for Emerging Technology and Security (CETaS) at The Alan Turing Institute.
"The current impact of AI on specific election results is limited, but these threats show signs of damaging the broader democratic system," the CETaS report found, noting that of 112 national elections that have either taken place since January 2023 or will occur in 2024, AI-based meddling was detected in just 19 and there's no data yet to suggest election results were materially swayed by AI.
That said, the CETaS report argues that AI content creates second-order risks, such as sowing distrust and inciting hate, that are difficult to measure and have uncertain consequences.
Rid suggested that as more competitors develop tools for synthetic content creation and OpenAI's share of the market declines, the Microsoft-championed lab will be less able to detect abuses of this sort. He also noted that OpenAI, in its discussion of IOs, doesn't address other forms of synthetic content abuse, including fake product reviews, ad bots, fraudulent marketing copy, and phishing messages. ®
Get our [15]Tech Resources
[1] https://openai.com/index/disrupting-deceptive-uses-of-AI-by-covert-influence-operations/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.brookings.edu/articles/the-breakout-scale-measuring-the-impact-of-influence-operations/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZllLZqwe9IT7lpu-sIGg6QAAAIg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://downloads.ctfassets.net/kftzwdyauwt9/5IMxzTmUclSOAcWUXbkVrK/3cfab518e6b10789ab8843bcca18b633/Threat_Intel_Report.pdf
[9] https://x.com/RidT/status/1796251141124276704
[10] https://www.theregister.com/2024/05/30/palantir_us_army/
[11] https://www.theregister.com/2024/05/30/salesforce_results/
[12] https://www.theregister.com/2024/05/30/nvidia_ai_conference/
[13] https://www.theregister.com/2024/05/30/blame_those_rising_ssd_prices/
[14] https://cetas.turing.ac.uk/publications/ai-enabled-influence-operations-threat-uk-general-election
[15] https://whitepapers.theregister.com/