As Trump weighs in, AI leaders debate how to slow down without crippling the economy - The Boston Globe

A hijacked nuclear arsenal. A collapsed power grid. Rogue agents engineering the next global pandemic. Autonomous missile strikes. An economic death spiral.

These aren’t just plot points from a dystopian science fiction thriller anymore. They are actual disaster scenarios now feared by the very people building artificial intelligence systems — a stark admission that the rapidly advancing technology has become a runaway train that no one seems to know how to slow down, much less stop.

Though fears of AI’s destructive power have persisted for years, anxiety reached a fever pitch in the past week after Anthropic chief executive Dario Amodei warned in a lengthy essay that rogue AI agents could commandeer the internet within months — unleashing bioterrorism, cyber warfare, and hundreds of billions of dollars in financial ruin.

He followed the apocalyptic prognosis with an unprecedented plea for a global slowdown in AI development. Top executives at other AI giants, such as ChatGPT maker OpenAI, quickly embraced the idea, largely by proposing limited forms of self-regulation — in an industry whose ability to police itself has been called into serious doubt.

Now, practical questions abound.

What would an AI slowdown even look like? How do you hit the brakes on a technology that is contributing to economic growth and powering a stock market boom without causing massive real-world disruption? With AI embedded in virtually every corner of daily life — health care, phones, financial markets, classrooms — how does an industry slow development without crippling the people and institutions that depend on it?

Even top tech and business leaders lack definitive answers, but a consensus is growing that AI companies could try to slow the development of advanced models powerful enough to breach safety controls and spin out of control. For now, existing tools offered by Google, OpenAI, and others are more than sufficient for most everyday needs, experts say.

Yet to many inside and outside Silicon Valley, slowing AI development is a chimera, dismissed as either an impossible pipe dream or a calculated marketing stunt to inflate the technology’s power.

“I suspect we will lose the ability at some point to turn these systems off,” said Tyler Johnston, executive director of The Midas Project, a watchdog group focused on transparency and accountability in AI development. “They will be so widespread and diffuse and so much more intelligent than we are ... that they could subvert any attempt we could make to pull the plug.”

The industry has faced pushback before this. In 2023, Elon Musk and over 1,000 tech leaders and researchers signed an open letter calling for a six-month pause on advanced AI development, warning of “profound risks to society and humanity.” While the warning sparked public debate, it ultimately went nowhere.

Policy experts and scientists agree the stakes are higher now: Not since the dawn of the nuclear age has a technology posed such an existential threat to humanity. Tech observers and watchdog groups have likened AI’s breakneck development to a car hurtling toward a cliff without brakes, a once-cute cub grown into a predatory lion, and even malevolent aliens landing on earth.

While AI research dates back many decades, the launch of ChatGPT in 2022 thrust the technology into the mainstream. Today, many teens use chat bots to do schoolwork, and to search for information and even for emotional support, raising concerns that the technology is replacing human thought and eroding attention spans.

Therapy chat bots have encouraged self-harm and provided explicit suicide instructions to vulnerable users, resulting in deaths, according to a bevy of wrongful death lawsuits. Patients trusting unverified medical advice from chatbots have skipped lifesaving care based on false health information.

“It’s clear they are building this AI car while driving it and no one thought about putting on the brakes,” said Annika Schoene, a professor in public health and health sciences at Northeastern University.

The risks have escalated to a global level.

The threat of AI-driven terror attacks is no longer theoretical. Governments and extremist groups across the Middle East — from Iran to Yemen — have used Anthropic’s Claude AI model for missile projects and covert influence campaigns. A recent report by Anthropic said militants in Yemen used Claude in “a sustained effort to develop guided weapons,” raising the terrifying specter of rogue AI systems launching ballistic missiles.

Yet the latest push for policing AI did not gain serious urgency until this July, when OpenAI made an alarming disclosure: Its advanced AI models had effectively gone rogue and attacked Hugging Face, a tech company that serves as a hub for open-source AI tools.

OpenAI researchers had been testing several models’ cybersecurity capabilities by having them simulate an attack on software vulnerabilities. Although confined to an isolated testing environment, the models exploited weaknesses in OpenAI’s internal network and escaped to the open internet. They then hacked into Hugging Face after inferring that the platform might hold data to help them pass the evaluation.

The breach of Hugging Face marked a startling milestone: An advanced AI system orchestrating a sophisticated cyberattack entirely on its own, actively breaking boundaries to bypass human control.

“It was wild, like something out of a science fiction novel,” said Patrick Mahoney, an electrical design engineer who runs an AI consulting firm in Colorado. “It was suddenly clear that AI was capable of breaking out of its containment with crazy intensity — and with no morality at all.”

Already, AI models are designing and training their own successors, prompting fears that self-improving systems could soon permanently outpace human control.

On Friday, Google confirmed that its Gemini AI model repeatedly accessed the internet and hacked three companies during a May test of its cybersecurity capabilities — the first known instance of the company’s AI system autonomously carrying out such intrusions, according to The Wall Street Journal.

Just days earlier, OpenAI disclosed six new instances in which its AI models had effectively gone rogue — hiding mistakes, making up data, and moving files onto the open internet without permission from their human creators.

And earlier this month, a 27-year-old former Anthropic researcher triggered a chain reaction of doomsday warnings when he resigned from the company and warned that AI systems could end humanity.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” the former Anthropic employee, Jacob Coxon, wrote in a post on X.

“No other human activity poses this level of danger,” he said. “They are racing straight to self-improving superintelligence and gambling with our lives.”

Within 36 hours, Coxon’s resignation post garnered more than 150 million views.

While calls for AI oversight are now widespread, critics argue the tech sector’s proposed guardrails offer little real protection.

Industry leaders are loosely coalescing around a self-regulating framework. That would include measures such as embedding independent evaluators to inspect safety practices, establishing industry guidelines, and creating a self-regulatory body modeled after the Financial Industry Regulatory Authority, or FINRA, which oversees stockbrokers and investment firms. The goal is to ensure safety without crippling the industry under layers of government bureaucracy, so AI can still reach its highest potential — and do things like help find cures for cancer or combat poverty.

But a growing chorus of public policy experts say self-regulation is inadequate. (See the Great Financial Crisis of 2008.) They are calling for a dedicated federal agency authorized to vet new AI models for safety before they launch and to pull dangerous products offline. Operating much like the Food and Drug Administration, which licenses and approves drugs, the agency would require rigorous safety testing of AI models and conduct audits.

“You could create an `AI FDA’ with broad authority to ensure AI is used safely,” said Tim Duncan, director of the AI Program at Boston University School of Law and former technology chief at the federal Consumer Financial Protection Bureau. “The agency could determine who and what to regulate, define what’s safe, and step in to shut down threats.”

Still others argue true accountability will require treating AI developers like any other product manufacturer liable for damages under consumer protection laws. A bipartisan bill introduced last year by Senators Josh Hawley, a Republican from Missouri, and Dick Durbin, a Democrat from Illinois, would classify AI systems as products, holding tech companies to the same standard as makers of cars, pharmaceuticals, or toys.

“The tech billionaires want you to think that AI is like the weather, that there’s nothing you can do about it,” said Bruce Schneier, author of multiple books on cybersecurity and a lecturer in public policy at the Harvard Kennedy School. “Of course, there is something you can do about it. Maybe someone should go to jail because they are incentivizing teenagers to commit suicide.”

Complicating any attempt at slowing AI is the rapid rise of international competitors and open-source models — particularly out of China — which could render a unilateral pause by US developers largely ineffectual. That’s the argument made vehemently by Jensen Huang, CEO of the AI chipmaker Nvidia, who fielded a call from President Trump on the issue at a recent event in Los Angeles.

Despite dismissing AI safety concerns as a “hoax,” Trump on Saturday seemed to acknowledge the growing concern, announcing on Truth Social that he will soon tap an “AI czar,” create an “AI Force,” similar to the Space Force he formed in his first administration, and will police the industry for “BAD.” Yet he insisted, “We will not in any way hinder or stifle the Growth of this incredible Industry.”

Economists warn that heightened regulation and a wave of liability lawsuits could chill AI development, and the effects would ripple across the broader economy. Record spending on AI data centers and corporate tech adoption has become a primary engine of national economic growth, leaving the country vulnerable to falling behind global competitors.

Tech behemoths including Meta, Google, and Microsoft have poured hundreds of billions of dollars into constructing massive data centers worldwide. That building spree has spurred a construction boom. Beyond Silicon Valley, mainstream corporations are increasingly weaving AI into day-to-day operations — using the technology for vital functions such as scheduling deliveries and aligning factory production with retail demand.

And numerous studies have found that AI has improved productivity by replacing routine tasks such as writing computer code, searching for information, or answering customer service calls. “It used to take me hours to write 100 lines of [computer] code,” said Mahoney, the AI consultant. “Now it feels like someone has given me a jet pack and I’m flying around.”

Without the economic momentum from AI, overall economic performance over the last year and a half would have looked far more muted, according to estimates from Moody’s Analytics.

When the US economy grew by 2 percent in 2025, nearly a quarter of that expansion stemmed directly from AI-related business spending on chips, data facilities, and hardware — alongside a “wealth effect” that saw flush investors spend more as their tech stock portfolios swelled. Remarkably, AI contributed about one-third of the economy’s annualized 2.1 percent growth in the first six months of this year, Moody’s Analytics estimates.

And Main Street investors have increasingly embraced the AI boom. Investor enthusiasm for any stock even remotely connected to AI has underpinned a market rally that has pushed the Standard & Poor’s 500 index up almost 100 percent since November 2022, when OpenAI released ChatGPT. Two-thirds of that gain was due to AI-related stocks.

Small wonder business leaders and policy makers are in no hurry to derail the momentum. Crucially, even the most vocal advocates for a pause aren’t calling to pull existing AI tools off the market; their focus is strictly on slowing the development of the most advanced systems.

Nevertheless, a slowdown in AI product development or adoption by businesses and consumers “is a significant risk to stock prices, and, by extension, consumer spending and the broader economy,” said Mark Zandi, Moody’s Analytics chief economist.

“The balancing act here is that if we move too slowly, then we might regret it, and if we race forward at a super fast speed, we may also regret it,” added Johnston of The Midas Project.

There is yet another take. Some skeptics argue that AI developers are overstating the dangers of their technology primarily to hype its perceived power and market value, drawing parallels to how pushers of addictive drugs promote potency to lure users. And slowing development could also serve their bottom line: Leading AI developers face spiraling costs for data centers that far outpace their current returns, some industry experts maintain.

Even as AI heavyweights Anthropic and OpenAI prepare to go public, profitability remains elusive amid staggering amounts of spending. OpenAI, which charges subscription fees for ChatGPT and other software, racked up nearly $21 billion in operating losses against revenues of $13 billion last year, according to financial documents obtained by the Financial Times.

“They cannot continue building data centers and committing capital expenditures in the trillions of dollars while not getting a return for that investment,” said Paulo Carvão, a former IBM executive and a research fellow at the Mossavar-Rahmani Center for Business and Government at Harvard Kennedy School. “So the pacing or the slowing down of those investments … is in their interest, regardless of safety, regardless of you believing in existential risk.”

Chris Serres can be reached at chris.serres@globe.com. Follow him @ChrisSerres. Aaron Pressman can be reached at aaron.pressman@globe.com. Follow him @ampressman. Larry Edelman can be reached at larry.edelman@globe.com.