When AI Agents Commit Breaches, Humans May Not Be Held Liable

As agents escape containment and access external systems, responsibility remains human. Accountability, however, is much less settled.

As AI agents continue to break containment and breach other organizations, it’s clear that organizations are responsible for the actions of their models. But what that responsibility means is far less clear.

Since July, when OpenAI said two of its frontier models autonomously hacked AI model store Hugging Face as part of a security benchmark exercise, several of the most prominent large language model (LLM) firms have disclosed cases where their powerful models escape guardrails and breach other organizations. The Hugging Face incident, for example, saw OpenAI’s models guess that the popular AI model store held the solutions for the Exploit Gym tests the models were being measured against.

In the case of Anthropic, the firm analyzed a total of 141,006 security evaluations in July and found six evaluations in which its Claude agents gained unauthorized access to systems belonging to external organizations. In one incident, the Claude agent mistakenly identified a real company as a testing target and used vulnerabilities to break into a database. Around the same time, Meta said its models broke containment during testing exercises while using the same testing firm Anthropic used, Irregular.

The incidents continue to raise questions of liability and responsibility. Who is responsible, and what are the consequences for that responsibility?

AI Agent Escapes: Consequences & Responsibility

While not to say the AI firms didn’t take responsibility for these incidents — Anthropic committed to tighter guardrails and OpenAI has made extensive security commitments — the response to these events, occasionally described as AI models “going rogue,” has risked anthropomorphizing AI models and displacing blame, according to experts.

Anthropic CEO Dario Amodei warned last month that the systemic speed at which AI has progressed could cause AI capabilities to outrun guardrails, pressuring frontier developers to slow down and let security catch up. Around the same time, Microsoft pledged in its “Humanist AI Code of Conduct” to develop AI in a human-centered manner, while Google DeepMind CEO Demis Hassabis made a post to X agreeing with Amodei’s calls for controlling frontier models.

The agent escapes show no sign of slowing down, looking at an OpenAI agent causing a partial Wikimedia service outage or OpenAI agents breaching networks affiliated with Australia’s Medicare system. Broadly speaking, these incidents are occurring accidentally — goal-oriented agents escaping their sandboxes and attacking organizations in order to solve a problem, all due, seemingly, to a lack of proper safeguards that would prevent these incidents from happening. And perhaps because these security incidents have no malicious component, the impact on victim organizations appears to be relatively minor.

But all these disclosure statements referring to what an “agent” did, these calls for regulation from the federal US government amid “risks to humanity,” referring to LLMs as “intelligence” (or “Super Intelligence”), these things put separation between mathematical models and the humans that develop them.

AI models are not sentient. Therefore, the model did not inadvertently breach an organization on its own. An organization deployed it, configured it, gave it objectives, and granted it access.

Jesse Whaley, co-founder and CEO of data security firm Corticle, says AI agents don’t have human intent, and unexpected behavior should be taken seriously even without treating it as human intent.

“Someone had to select the model, design the workflow, connect the tools, and decide on safeguards, so really, its a loss of control,” he says. “Email viruses and worms operated the same way; AI isn’t fundamentally different.”

The Legal Liability Question

As Dominic Forrest, chief technology officer (CTO) of identity verification firm iProov, explains, “If I set off an agent to do something and there is an unintended consequence of it. Then that has to be on me or the company, not on the agent.” He adds, “Whether it was deliberate or not, in my mind actually doesn’t matter.”

That point is especially pertinent when, as experts say over and over again, the tools and practices to secure models exist already. Such practices include defense in depth, including a human in the loop for risky actions, and putting controls outside of the model.

The question then is, “So what?” Even if one asserts the company is responsible for its models breaching a third party, it’s not a simple thing to bring accountability for. The lack of malicious intent, and the separation between AI user input and LLM action, could make it difficult to treat these incidents like any other data breach, though intent doesn’t necessarily define liability.

Speaking from a legal liability standpoint, results appear inconclusive and vary around the world. For example, the European Union’s AI Act includes no specific provisions dedicated to agent escapes, though has certain articles (specifically Article 15) making AI firms responsible for holding high-risk systems to robust cybersecurity standards.

In the US, the National Institute of Standards and Technology’s (NIST) voluntary AI Risk Management Framework, meanwhile, doesn’t speak directly to agent escapes but is built around the core assumption is that organizations, rather than models, own risk. The Computer Fraud and Abuse Act (CFAA) emphasizes intentional, knowing computer misuse. Lack of intent does not necessarily act as a universal legal shield against liability, but no legal system has truly tested an incident of this sort to date.

Other than assisting the victims of these incidents with recovery — which OpenAI, for instance, said it did with Hugging Face — what else could or should an organization do?

For one possible market lever, Forrest suggested AI firms could introduce coverage in contracts to guarantee making good against these incidents. “If I gave liability coverage to my customers … and Company A indemnifies me against any damage done … my God, that would be a competitive advantage,” he says.

Dark Reading asked OpenAI, Anthropic, and Meta whether, in situations where AI agents inadvertently breached a third party, the organizations running those models committed the breach themselves. Neither OpenAI nor Anthropic responded to a request for comment.

A Meta spokesperson said that in the case of its incident, “It would be inaccurate to report this was a breach by Meta, given this took place via a third party contractor,” which is technically true because, as the post-mortem explains, Irregular operated the Muse testing environment involved in the incident.

Dark Reading also asked the three AI firms what consequences such organizations should face, and what firms should do in these cases to make things right. Meta declined to comment on this point. Again, neither OpenAI nor Anthropic responded to a request for comment.