It's only a matter of time before LLMs jump start supply-chain attacks
- Reference: 1735496411
- News link: https://www.theregister.co.uk/2024/12/29/llm_supply_chain_attacks/
- Source link:
No, we're not talking about a fully AI-generated attack from the initial access to the business operations shutdown. Technologically, the criminals aren't there yet. But one thing LLMs are getting very good at is assisting in social engineering campaigns.
And this is why Crystal Morin, former intelligence analyst for the US Air Force and cybersecurity strategist at Sysdig, anticipates seeing highly successful supply chain attacks in 2025 that originated with an LLM-generated spear phish.
[1]
When it comes to using LLMs, "threat actors are learning and understanding and gaining the lay of the land just the same as we are," Morin told The Register . "We're in a footrace right now. It's machine against machine."
[2]
[3]
Sysdig, along with other researchers, in 2024 documented an uptick in criminals using stolen cloud credentials to access LLMs. In May, the container security firm documented attackers [4]targeting Anthropic's Claude LLM model .
While they could have exploited this access to extract LLM training data, their primary goal in this type of attack appeared to be selling access to other criminals. This left the cloud account owner footing the bill — at the hefty price of $46,000 per day related to LLM consumption costs.
[5]
Digging deeper, the researchers discovered that the [6]broader script used in the attack could check credentials for 10 different AI services: AI21 Labs, Anthropic, AWS Bedrock, Azure, ElevenLabs, MakerSuite, Mistral, OpenAI, OpenRouter, and GCP Vertex AI.
We're in a footrace right now. It's machine against machine
Later in the year, Sysdig spotted attackers attempting to use stolen credentials to enable LLMs.
The threat research team calls any attempt to illegally obtain access to a model "LLMjacking," and in September reported that these types of attacks were "on the rise, with a [7]10x increase in LLM requests during the month of July and 2x the amount of unique IP addresses engaging in these attacks over the first half of 2024."
Not only does this cost victims a significant amount of money, according to Sysdig, but this can run more than $100,000 per day when the victim org is using newer models like Claude 3 Opus.
Plus, victims are forced to pay for people and technology to stop these attacks. There's also a risk of enterprise LLMs being weaponized, leading to further potential costs.
2025: The year of LLM phishing?
In 2025, "the greatest concern is with spear phishing and social engineering," Morin said. "There's endless ways to get access to an LLM, and they can use this GenAI to craft unique, tailored messages to the individuals that they're targeting based on who your employer is, your shopping preferences, the bank that you use, the region that you live in, restaurants and things like that in the area."
In addition to helping attackers overcome language barriers, this can make messages sent via email or social media messaging apps appear even more convincing because they are expressly crafted for the individual victims.
[8]
"They're going to send you a message from this restaurant that's right down the street, or popular in your town, hoping that you'll click on it," Morin added. "So that will enable their success quite a bit. That's how a lot of successful breaches happen. It's just the person-on-person initial access."
She pointed to the Change Healthcare ransomware attack - for which, we should make very clear, there is no evidence suggesting it was assisted by an LLM - as an example of one of 2024's hugely damaging breaches.
In this case, a ransomware crew [9]locked up Change Healthcare's systems, [10]disrupting thousands of pharmacies and hospitals across the US and accessing private data belonging to around [11]100 million people . It took the healthcare payments giant [12]nine months to restore its clearinghouse services following the attack.
It will be a very small, simple portion of the attack chain with potentially massive impact
"Going back to spear phishing: imagine an employee of Change Healthcare receiving an email and clicking on a link," Morin said. "Now the attacker has access to their credentials, or access to that environment, and the attacker can get in and move laterally."
When and if we see this type of GenAI assist, "it will be a very small, simple portion of the attack chain with potentially massive impact," she added.
While startups and existing companies are releasing security tools and that also use AI to detect and prevent email phishes, there are some really simple steps that everyone can take to avoid falling for any type of phishing attempt. "Just be careful what you click," Morin advised.
Think before you click
Also: pay close attention to the email sender. "It doesn't matter how good the body of the email might be. Did you look at the email address and it's some crazy string of characters or some weird address like name@gmail but it says it's coming from Verizon? That doesn't make sense," she added.
LLMs can also help criminals craft a domain with different alphanumerics based on legitimate, well-known company names, and they can use various prompts to make the sender look more believable.
Even voice-call phishing will likely become harder to distinguish because of AI used for voice cloning, Morin believes.
[13]Cast a hex on ChatGPT to trick the AI into writing exploit code
[14]OpenAI claims its software can clone your voice from 15 seconds of you talking
[15]Microsoft dangles $10K for hackers to hijack LLM email service
[16]Don't fall for a mail asking for rapid Docusign action – it may be an Azure account hijack phish
"I get, like, five spam calls a day from all over the country and I just ignore them because my phone tells me it's spam," she noted.
"But they use voice cloning now, too," Morin continued. "And most of the time when people answer your phone, especially if you're driving or something, you're not actively listening, or you're multitasking, and you might not catch that this is a voice clone - especially if it sounds like someone that's familiar, or what they're saying is believable, and they really do sound like they're from your bank."
We saw a preview of this during the run-up to the 2024 US presidential election, when [17]AI-generated robocalls impersonating President Biden urged voters not to participate in the state's presidential primary election.
Since then, the FTC issued a [18]$25,000 reward to solicit ideas on the best ways to combat AI voice cloning and the FCC [19]declared AI-generated robocalls to be illegal.
Morin doesn't expect this to be a deterrent to criminals.
"If there's a will, there's a way," she opined. "If it costs money, then they'll figure out a way to get it for free." ®
Get our [20]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Z3HUjReb0I4Tip_FruD7agAAAAE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z3HUjReb0I4Tip_FruD7agAAAAE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z3HUjReb0I4Tip_FruD7agAAAAE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://sysdig.com/blog/llmjacking-stolen-cloud-credentials-used-in-new-ai-attack/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Z3HUjReb0I4Tip_FruD7agAAAAE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://github.com/kingbased/keychecker
[7] https://sysdig.com/blog/growing-dangers-of-llmjacking/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Z3HUjReb0I4Tip_FruD7agAAAAE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/02/29/alphv_change_healthcare/
[10] https://www.theregister.com/2024/02/22/change_healthcare_outage/
[11] https://www.theregister.com/2024/10/27/senator_domain_registrars_russia_disinfo/
[12] https://www.theregister.com/2024/11/20/change_healthcares_clearinghouse_services/
[13] https://www.theregister.com/2024/10/29/chatgpt_hex_encoded_jailbreak/
[14] https://www.theregister.com/2024/04/01/openai_voice_clone/
[15] https://www.theregister.com/2024/12/09/microsoft_llm_prompt_injection_challenge/
[16] https://www.theregister.com/2024/12/19/docusign_lure_azure_account_takeover/
[17] https://www.theregister.com/2024/01/23/robocaller_biden_new_hampshire/
[18] https://www.theregister.com/2024/01/05/ftc_voice_cloning_solution/
[19] https://www.theregister.com/2024/02/08/sorry_scammers_the_fcc_says/
[20] https://whitepapers.theregister.com/
Re: Protection must improve first
Er, perhaps, but if you think AI is coming to your rescue on the defensive side, you are sorely mistaken.
AI-based defences are stochastic and will fail a certain percentage of the time. But there are so many exposed attack surfaces to defend, that even if you are good at blocking randomised repeat attacks, chances are that one will get through, and one is enough.
Basically, the inherent randomness/unreliability of the bullshit machines is bad for the defenders but good for the attackers.
So we are entering a world where the only defence is attack, and we all know that this path leads to Mutually Assured Destruction..
The makers of Battlestar Galactica never knew how poignant Bill Adama's message would be..
The more expensive allowing your LLM resources to be compromised is
The more incentive there will be for companies to invest in better securing them. Because when you can throw out numbers like $46K and $100K per day, that's something quantifiable to bosses. An immediate cost that will be incurred no matter what.
One of the reasons why companies have such poor security in general is that all the costs are potential, or fall onto others. If someone breaks into the company web server and steals customer login credentials or their credit card numbers, that's a cost they don't have to bear. Yeah maybe there's reputational damage but that's mostly theoretical. You can't put a dollar figure on it to plug into a beancounter's spreadsheet and determine what level of investment in protecting against it is justified in ROI terms. And importantly, the more other companies suffer similar attacks, the less your company's reputation is hurt when you're attacked. It happens to everyone, there's nothing to be done, customers just have to be suffering the consequences for our shortcomings!
So long as you can put a nice round figure, and one high enough that you're talking about such large losses per day PER ATTACKER (because there's no reason to believe you'll have just one such attack) there will be more investment securing their LLM resources to prevent such losses. They still won't invest in preventing attacks that compromise customer data, our data. Because so long as they don't bear the cost of the consequences, they don't have any incentive to invest anything to prevent it. Even if they are ransomed they mostly don't care, because the law stupidly allows them to buy ransomware insurance, making THAT a fixed cost (with only a theoretical and unknown increase in the future if they are successfully ransomed)
Re: The more expensive allowing your LLM resources to be compromised is
If big quantifiable risks motivate corporations, then they'd already have implemented proper security. However, despite the multi-billion costs of digital attacks, companies keep getting hit, and the authorities show a complete incapacity to stem the flow of attacks.
So I don't think that a new attack surface for the bad guys will result in any change in corporate attitudes to ITsec.
Re: The more expensive allowing your LLM resources to be compromised is
"However, despite the multi-billion costs of digital attacks, companies keep getting hit, and the authorities show a complete incapacity to stem the flow of attacks."
If the attack is targeted at PII, there doesn't seem to be much downside to that as credit monitoring purchased in bulk is dirt cheap. A B to C business losing a customer list isn't a big deal to them. A B to B company that loses their customer, vendor and supplier list might be in a pickle. The same could be said for a media production company. Having that information accessible online in arbitrary quantities is the issue. Any true need to analyze a company's data from a outside location should really be locked down so even an AI assisted attack isn't an issue.
One of my favorite stories is "A Logic Named Joe" by Murray Leinster. It's an amazing piece of work considering when it was written. Joe, a computer in modern parlance, wakes up and just wants to be helpful. Many people Joe helps are just looking to help themselves. The parallels to this article are thought provoking.
Re: The more expensive allowing your LLM resources to be compromised is
That means that having a high price tag on defending your resources makes them somehow more defensible. Pardon me, but that's rather silly.
A very cheap social-media hack or trusted-employee misstep can cause a whole world of hurt.
The people in business suits that roam the upper floors only know about things that cost a large percentage of the gross for the corporation. They don't care/invest in something that runs under 0.1% of the total spend.
Until SECURITY is the number one priority for these companies, they will be attacked and penetrated and damaged.
They are coming for your CoPilot!
And any other embedded AI tools that your software manufacturers shove into your private places.
I can't imagine how easy it would be to take over a fleet of zombie PCs and make them submit the nefarious queries on behalf of the PC "owner" (better known as the tenant.)
I'm sure Micro$oft, Apple, Google, etc. have hardened their systems where no foreign/external keystrokes can be logged and acted upon by these always-in-your-face AI assistants.
The AI pitch is coming back to bite you !!!
AI is right 'some' of the time BUT as an attack agent it does not matter as you are running your attacks 100s or 1000s times per second so the failures are in the majority anyway !!!
AI as your defense needs to be right ALL of the time because when it is wrong the 'Baddies' get through and in all probability the breakthrough is used to amplify the attack.
The same AI cannot be both these things !!!???
Please tell me now HOW AI is so good that it works to save the 'Crown Jewels' and HOW it always works !!!
Are you hiding some 'Better' AI that can save us all from the 'Baddies' or are you over egging the pitch on just how 'Good' the AI really is !!!
Do tell !!!
:)
So let's recapitulate...
... what tech has been about these last twenty years:
a) social medias. Crap.
b) crypto-money. More crap.
c) LLM, sold as "artefactual intelligence". Still more crap.
Brilliant. Thank you very much, tech bros.
Re: So let's recapitulate...
They're not there to help you. They're there to enrich themselves by selling delusions to the gullible.
They're going to send you a message from this restaurant that's right down the street, or popular in your town, hoping that you'll click on it,"
The simplicity of a link being able to screw over an organisation seems to be a far deeper problem than the tricks used to get that click.
Protection must improve first
It would seem that the only way to combat ever-better crafted-by-AI attacks is to use AI-powered defences.
- better EDR and antimalware based on AI
- AI-based email screening
- better secure caller ID verification
- AI-based software (on phones) to screen incoming calls (and text messages)
This is going to become such a serious issue that maybe the above things need to start to be developed right away. And not just for businesses, but for consumers. Unfortunately, at the moment, these things aren't powerful enough - and of course they cost money. I can't see a scenario where the cost of services won't increase, as service providers slowly up their game in response to this threat.
Security technology has always played behind the curve against threat technology, but the danger and risk of AI-powered attacks is so great now that we can't really afford to be behind the curve any more.
We're in for a rough ride!