‘Sleeper Agent’ Tactic Hides Payloads in Open-Weight AI Models
Malware can lurk inside just 0.00001% of a neural network. How, then, can any user tell it’s there?
AI models downloaded from the open Internet can be subtly poisoned with malicious payloads if attackers are careful enough.
Open-weight AI models have become seriously popular lately. Even early last year, McKinsey found that 63% of organizations used them, and 76% of survey respondents said that they expected to use them more in the coming years. They’re auditable, developers like using them, and, perhaps most of all, they’re cheap.
Like any other software you can download off the Internet, however, open-weight and open source AI tools carry some degree of risk. Last week, researchers described a technique they call “Lazarus,” which could allow an attacker to poison a model using a near-imperceptible few of its neurons. Microsoft described it as turning a model into a “sleeper agent,” capable of granting an attacker access to its user’s environment.
“A model is a file almost nobody inspects, yet we download models from strangers and give them access to our code, credentials, and systems,” explains Briland Hitaj, senior AI security research engineer at Confidencial, whose research into techniques like this preceded Microsoft’s. “Malicious models have already turned up on Hugging Face, though so far they abuse file formats like Python’s Pickle to run code when loaded. I’m not aware of anyone hiding payloads inside the weights themselves outside research, but that’s where the research points.”
How to Poison Open-Weight AI Models
Since at least 2020, researchers have been coming up with clever ways to store bad stuff inside of neural networks.
The premise behind this whole category of research is that the size and complexity of these networks provide cover for small, secret, but potentially powerful payloads. Just look at Qwen 3.8-Max: It’s totally open-weight, so you can inspect it to your heart’s content. But it’s made of 2.4 trillion parameters, and probably around 100 million neurons. If an attacker plants a functioning payload in some of those neurons or parameters, how exactly are you going to find it?
The trick, then, isn’t finding a place to hide a payload. First, it’s getting that payload to function reliably amid so many moving parts; second, it’s making sure it doesn’t noticeably impact the normal functioning of the model; and lastly, optionally making the payload robust to fine-tuning.
Different researchers have come up with different ways to do it. In 2022, Hitaj and his colleagues came up with a technique called “MaleficNet,” which repurposed an old military and, later, 2G and 3G mobile network technique called “code division multiple access” (CDMA).
Very simply, they split malware binaries into tiny pieces and spread the pieces across the many individual weights that make up a neural network. It wouldn’t look like anything but noise to a passerby, yet with a trigger mechanism and a way to decode the weights, they could recover, recombine, and execute the binary.
“The vulnerability comes from how neural networks work, not from a bug,” Hitaj explains. “Models have far more parameters than they strictly need, so you can nudge millions of weights slightly, or repurpose a few neurons that rarely fire, without visibly changing the model’s behavior. And since nobody can read weights the way they read code, those changes go unnoticed.”
Microsoft’s new technique compromises on capacity for simplicity. Where Hitaj proposed encoding an entire binary across many model parameters, Microsoft researchers propose sneaking a brief command inside of fewer than 10 neurons. Specifically, one must target “dormant” neurons — those that prove to have little effect on the overall model’s functioning — late in a model’s processing cycle, and thus more directly influential over its output. One encodes a payload in these neurons, arranged to fire sequentially, and configures them to stay largely inactive unless they’re fed a trigger. Eventually, you get a sleeper agent.
The target payload can be arbitrary, so long as it’s small and formatted correctly. The trigger can be any particular user input to the model, or tool output, or something so simple as a date read in a user’s terminal.
How to Suss Out Poisoned AI
Luigi Vincenzo Mancini, a professor at Sapienza University of Rome and one of the co-authors of the MaleficNet paper, advises organizations to take a defense-in-depth approach to open-weight AI security, “rather than trying to find a single magical ‘AI malware scanner.'”
Like any third-party software, users should download AI models from official sources. As an added measure, users can compare the models they download with a known, trusted baseline.
“Microsoft says that reliable detection of these kinds of modifications requires being able to diff the deployed weights against the original, unmodified weights,” Mancini notes. “That’s a powerful argument for treating model weights more like source code or binaries than like ordinary data files.”
Next, users should scan their models in general for anomalies. Though as Hitaj points out, “No scan proves a model is clean, and ordinary testing rarely finds a trigger the attacker chose. So the key controls limit what a compromised model can do: give AI agents only the access they need, keep credentials and production systems out of reach, restrict network access, log what agents do, and require human approval for consequential actions. If a model turns malicious, you want it boxed in and visible.”
Beyond restricting them from the outset, “Organizations should monitor for unexpected agent behaviour at the system boundary,” Mancini says. Practically speaking, “a model emitting an unusual shell command, attempting an unexpected network connection, accessing credentials it normally doesn’t need, or suddenly invoking a tool in an unusual context can be a much more actionable detection technique than directly inferring malicious intent from billions of model parameters.”
Originally published on Dark Reading