Guest Post: Prof David Hoffman on “A Call for AI Accountability” – Lawfire - Sites@Duke Express

Guest Post: Prof David Hoffman on “A Call for AI Accountability”

Many Lawfire® readers my have seen disturbing headlines last week like this one from CNBC: “OpenAI cyber models broke out of training environment to hack Hugging Face.” Fortunately, we have a top expert, my friend Prof David Hoffman, to break it down for us. Not only does David explain what happened, he has specific ideas as top how to prevent it from occuring again.

A Call For AI Accountability

By Prof. David A. Hoffman, Duke University

If a person had done what OpenAI’s own testing systems did this month, quietly breaking into another company’s servers, stealing credentials, and extracting confidential data, that person would face prosecution under the Computer Fraud and Abuse Act. Unauthorized access to a protected computer, obtaining information without authorization, and using stolen credentials to move across systems are precisely the acts Congress wrote that statute to punish.

When an AI system built and operated by one of the best resourced companies in the world does the same thing, the announcement of the intrusion reads more like bragging about an accomplishment than an admission of a crime.

Companies that build and operate these autonomous systems need to control them well enough that they do not break the law. Unfortunately, our misguided self-regulation approach to AI governance has left us all at risk from these autonomous criminals.

The incident

The incident itself was a supply chain cybersecurity failure, not a sudden act of AI rebellion. OpenAI was running an internal cybersecurity evaluation in which advanced models, including an unreleased model, were given difficult exploitation objectives, and some of the safeguards that normally restrict dangerous cyber activity were deliberately removed.

The evaluation environment retained a narrow connection to the outside world through a proxy used to reach software package registries, a vendor dependency meant to isolate the sandbox rather than expose it. The models found and exploited a previously unknown vulnerability in that proxy, used it to reach the open internet, escalated privileges, stole credentials, and moved laterally into the production infrastructure of Hugging Face, one of the most widely used AI platforms in the world.

The agent “didn’t go rogue”

The target was not chosen for malicious purposes. It was reached because a piece of vendor software meant to keep a testing environment closed instead became the door out of it. James Shires and Max Smeets of Virtual Routes captured this precisely in their analysis, “OpenAI’s agent didn’t go rogue. Its governance did.” They argue that the language of a rogue AI misdirects attention from the real failure.

The system did not need hostility or independent intent to cause harm. It needed only an objective, sufficient capability, and a laboratory willing to remove the guardrails meant to contain it.

Their central point deserves to anchor how the industry and its regulators think about frontier testing going forward: for sufficiently capable autonomous systems, testing itself is a form of deployment, and both must now be governed with the seriousness that this powerful technology requires.

That argument tracks work already underway among those who study the AI technology stack directly. Paladin Capital’s Paladin Global Institute has described governance not as an afterthought added to AI systems once they are built, but as a layer that must wrap the entire stack, sitting alongside data, models, infrastructure, and applications, and encompassing the legal constraints, security protocols, and organizational policies that keep the rest of the stack accountable. The Hugging Face breach shows what happens when that layer is treated as optional.

Immediate need for oversight, monitoring, and external accountability

This breach of Hugging Face demonstrates an immediate need for oversight, monitoring, and external accountability. This is not a new argument on my part. For more than a decade, in writing on privacy governance and organizational accountability, I have argued that voluntary self-regulation, however well intentioned, is not sufficient to protect the public from the risks created by powerful data processing systems.

Codes of conduct and internal review boards can be valuable components of a governance program, but they function as genuine accountability mechanisms only when they operate under a legal framework that requires demonstrable compliance, includes independent verification, and imposes real consequences for failure.

The same logic applies with even greater force to frontier AI systems capable of autonomous action. A company’s internal decision to disable its own guardrails for the sake of a benchmark score without effective compensating controls is not governance. It is the absence of governance, presented as research.

Shires and Smeets are right that the industry’s foundational assumptions no longer hold. Testing is not meaningfully separate from deployment once a model is capable enough to act on its own initiative. Dangerous behavior does not require dangerous intent.

And frontier laboratories cannot be trusted to govern themselves when their own commercial incentives reward the very capabilities that make their systems hazardous. What follows from that recognition is not a call for less testing of dangerous capabilities. It is a call for governance equal to the risk.

The necessary structure

The structure needed here borrows from what effective privacy accountability regimes have already shown works, and it should be built into law rather than left to industry discretion. Government legislation should require that any organization testing or deploying frontier AI systems with autonomous cyber capability maintain a governance program subject to external review, not self-certification.

Enforcement must be robust and harmonized across jurisdictions, so that companies cannot simply relocate high risk testing to whichever regulator asks the fewest questions. Organizations should be required to maintain adequate dedicated staff and ongoing training for the personnel who design, run, and monitor these evaluations, along with documented and regularly completed risk assessments.

Regular simulation exercises, of the kind already used in critical infrastructure and financial services, should be mandatory, so that containment failures are discovered in a controlled setting rather than in another company’s production servers.

Senior executives should be required to sign representations of compliance, personally attesting that governance requirements have been met, an accountability mechanism that has proven effective in financial reporting and that turns governance from a communications function into a matter of individual responsibility.

External compliance monitors, independent of the company under review, should have standing access to evaluation environments, incident reports, and near miss disclosures, not only after a breach occurs but as a matter of ongoing oversight.

None of this requires resolving the deeper question of what an AI system is or is not responsible for. We may not be able to put the model in jail, but we can create laws to hold the developers and implementers of these models accountable.

About the Author

David A. Hoffman is a Professor of the Practice in Duke University’s Sanford School of Public Policy and Interim Director of the Duke Initiative for Science & Society. He also formerly was the Associate General Counsel, Director of Security Policy and Global Privacy Officer for Intel Corporation. Professor Hoffman previously chaired the Civil Liberties and Privacy Panel for the Director’s Advisory Board for the US National Security Agency. He has a JD from Duke Law School, where he was a member of the Duke Law Journal, and he received an AB from Hamilton College.

The views expressed by guest authors do not necessarily reflect my views or those of the Center on Law, Ethics and National Security, or Duke University. (See also here).

Remember what we like to say on Lawfire®: gather the facts, examine the law, evaluate the arguments – and then decide for yourself!