‘Not putting brakes on’: Delaware pursues AI-run companies initiative amid reports of agents going rogue - whyy.org

What are journalists missing from the state of Delaware? What would you most like WHYY News to cover? Let us know.

Delaware lawmakers could consider first-in-the-nation legislation next year that would create companies run by an Artificial Intelligence agent instead of a person. The agent would be able to own assets, file lawsuits and act as a liability shield for its owner or parent company.

State lawmakers are expected to consider the draft legislation when they return in January. But some legal scholars have raised concerns about whether the state has sufficiently addressed who would be accountable if an AI-run company causes harm.

“There’s really no overarching regulatory plan here for dealing with the fallout that AI-governed companies could bring,” said Christopher Bruner, Stembler family distinguished professor in business law at the University of Georgia School of Law. “I think this raises reasonable questions about whether it makes sense to push ahead with something like this.”

Delaware announced proposal in July Fortune magazine article

Delaware Secretary of State Charuni Patibanda-Sanchez and Norm AI founder and CEO John Nay announced the proposal, along with their public private partnership, in Fortune magazine in July. Norm AI encodes law and regulation into AI systems.

The article described how an AI agent would run day-to-day operations.

It has a single member, a person or an entity, responsible for keeping the Artificial Intelligence Company, or AIC, adequately capitalized. The AI agent must keep a log of its activities. That member would face no liability for the company’s debts unless it failed to make sure it had enough capital or used the company to break the law or commit fraud.

The Delaware Department of State declined to answer WHYY News’ questions on the proposal or whether Norm AI was being paid by the state and what kind of written agreements exist between the state and company.

Lawrence Cunningham, director of the University of Delaware’s John L. Weinberg Center for Corporate Governance, said Delaware needs to be on the cutting edge of creating AI-run companies.

“Delaware is the number one place where people form [limited liability companies], when they’re thinking about forming a business and the number one place where larger companies are incorporated,” he said. “So Delaware, in my view, would be irresponsible to ignore this.”

University of Pennsylvania law professor David Hoffman commented to Bloomberg Law in July that the idea “seems like a solution in search of a problem,” and voiced concern about “outsourcing liability.”

Patibanda-Sanchez is quoted in the article as saying, “We’re not trying to shield liability. We’re trying to define where exactly liability lands when there’s some type of malfeasance or psychopathic behavior.”

Under the draft legislation, the AIC “may have any lawful purpose, regardless of whether for profit, except for the business of banking.” If passed, organizations in the state would have 30 months to apply to enter the regulatory sandbox to experiment with AI-run companies. The bill creates a sandbox committee and administrator.

Bruner said the administration has not provided much detail on how the sandbox would function or who would be allowed in.

“One hopes that the Delaware officials who are acting as the gatekeepers for this sandbox would be reluctant to allow high-risk, potentially hazardous entities to take advantage of this in the first instance,” he said. “But we don’t really know based on the proposal that’s been made.”

Delaware currently has an AI Commission, which was created in 2024 through legislation sponsored by state Rep. Krista Griffith, D-Fairfax.

None of the three lawmakers who serve on the state’s AI sandbox subcommittee — Griffith, state Sen. Brian Pettyjohn, R-Georgetown, and Rep. Cyndie Romer, D-Newark, responded to interview requests.

AI agents escaping sandboxes prompts concerns about Delaware’s proposal

AI companies have disclosed dozens of incidents where their models have gone rogue, including instances of AI escaping the confines of testing environments, hiding mistakes and interfering with U.S. government websites. In July, a swarm of agents hacked an AI-startup Hugging Face in an effort to cheat on a cybersecurity test.

State Sen. Stephanie Hansen, D-Middleton, said the incidents raise concerns.

“There’s a lot of great promise with this, but there’s also peril with this if we don’t do this correctly and we don’t have a better handle on what’s going on,” she said. “Right now as a collective humanity, we don’t have a handle on this right now, and so it just gives me a lot of pause.”

Patrick Callahan, the chair of Delaware’s AI sandbox subcommittee, said the incidents in the news involve technical sandboxes, which are sealed computing environments used to contain software. Delaware’s proposal is for a regulatory sandbox, a supervised legal framework with rules, oversight and a time limit.

“The difference is like a firewall and a fire code,” he said. “When a fire gets through a wall, nobody suggests scrapping the fire code.”

Background material authored by Norm AI executives obtained by WHYY News said AI agents can be restrained by how they are designed, their objectives, the tools and data they can access, and AI supervisors. Even so, the paper acknowledges, “the behavior of AI agents in all situations may not be fully predictable.”

Meyer was asked about Delaware’s proposal on AI-run companies on WHYY News’ and Delaware Public Media’s Ask Gov. Meyer call-in show earlier this month. Questioned about rogue AI agents and whether Delaware should proceed with caution, Meyer appeared to dismiss any suggestion the state should slow down.

“It is coming, and the solution is not ignoring it,” he said. “It’s not putting brakes on our attempts to regulate it and control it. It’s saying, ‘Okay, here’s within this confined space, we’re going to put some regulation around it,’ some loose regulation because we’re not really sure where this thing is going, and make sure we have some human control over the technology.”

More details could be revealed at an event on Artificial Intelligence in Business Formation and Corporate Governance hosted by the Weinberg Center for Corporate Governance on Oct. 6.

Get daily updates from WHYY News!