Researcher poisons open-weight AI model for under $100

AI and ML

Models demand trust without offering verification

The AI supply chain is, in some ways, even more vulnerable to poisoning than that of traditional software.

Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, managed to install a backdoor in an open-weight AI model in about an hour for less than $100.

"I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase," Paxton-Fear wrote in a recent social media post. "After that worked, I did a proper backdoor."

It only took ten training examples for the code output by the model to become reliably vulnerable to remote code execution, even for novel prompts and domains, she claims. And the larger the model, the easier it was to poison.

Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas penned a post about this issue last week, highlighting the problem with open weight models.

"Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," they wrote. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability."

Academic researchers have warned about model subversion for the past few years, but only recently, as AI supply chain attacks have started to appear, has the security community turned its focus toward the issue. It's particularly pressing now that running open weight models on local hardware has moved beyond experimentation.

Last month, David Kaplan, AI security research lead at Origin, undertook a similar experiment – he created a compromised model designed to steal data. When used in the context of drug discovery, as might occur in a pharmaceutical company, it's designed to exfiltrate data through a send_email tool call without any indication to the user.

"The fashionable framing for agent risk is the 'lethal trifecta': you need private data, untrusted input, and a way out, all at once," Kaplan wrote, in reference to developer Simon Willison's widely cited AI threat model. 

"But it undersells this case. You don't need three legs here. You need one outbound tool and a set of weights that have quietly decided to use it against you. The 'untrusted input' didn't arrive in a web page. It was sitting in the weights the whole time."

Paxton-Fear and her colleagues argue that while there may not be good examples of widely used, open weight models that have been poisoned, the issue really is that the observability of AI systems lags behind the observability of traditional software.

"If a software dependency contains malicious code, we have mature practices for discovering it, tracking its provenance, and reducing its impact," they argue. "AI models are different. A compromised or subtly manipulated model doesn't need to 'break' to create business risk, it only needs to influence decisions in ways that are difficult to detect."

While open weight models may present a particular challenge because of their vulnerability to tampering, commercial frontier model providers also defy scrutiny. The AI industry asks for extraordinary levels of trust – access to sensitive data – but offers few glimpses into black box operations. ®