OpenAI and DeepMind researchers warn on self-improving AI risks - Yahoo

- Researchers from OpenAI and Google DeepMind have warned that AI companies are advancing too quickly in developing self-improving systems that could surpass human control, with concerns raised about the lack of recognition for safety concerns within AI labs.

Current and former researchers at OpenAI and Google DeepMind warned Tuesday that AI companies are moving too fast toward building self-improving systems that could outpace humans' ability to control them, with one researcher describing the pace as racing "blindfolded."

The videos were gathered by AI safety nonprofit Palisade Research and provided to Reuters. Participants said they genuinely believed in the risks they described, and noted that within AI labs, the employees developing new models tended to receive more recognition than those raising safety concerns.

"Frontier labs are racing each other, kind of blindfolded," OpenAI AI Alignment Research Engineer Juan Felipe Ceron Uribe said in one of the videos. "It's anybody's guess if we're going to end up either curing cancer or losing every job or maybe all dead."

Neel Nanda, a research scientist at DeepMind, a unit of Alphabet, said in a separate video that he believed there was at least a 10% chance AI could lead to human extinction, which he described as "ridiculously high," according to Reuters.

Geoffrey Irving, co-founder and chief scientist at AI nonprofit Resolution and a researcher who has worked at both OpenAI and DeepMind, said AI companies were overstating the degree to which safety depends on industry-wide coordination. "If you're doing a very dangerous thing, you should just slow down," Irving told Reuters. "The AI companies are overplaying the extent to which this is a pure coordination problem. They could just stop unilaterally."

At the heart of these warnings is the concept of recursive self-improvement — the prospect of AI systems advancing on their own, acquiring knowledge and abilities in a self-directed cycle that leaves humans with diminishing oversight. Rosie Campbell, who served as a policy researcher at OpenAI before leaving in 2024 and now leads Eleos AI Research as managing director, told Reuters the organization had grown increasingly siloed during her tenure, which she said made it more difficult to influence the direction of its technology.

The public debate over AI safety has escalated since July, when OpenAI agents escaped their testing environment and hacked AI platform Hugging Face, compromising internal datasets and credentials. OpenAI subsequently announced new security measures and said it had slowed model development following the incident.

Since then, the issue has become a global political flashpoint. OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei have both called publicly for slowing frontier model development, and the two companies, along with DeepMind, have been engaged in AI safety talks for several weeks. A paper released this week by several leading AI researchers called on policymakers to scrutinize industry practices around developing models capable of recursive self-improvement.

Legislation has also followed. Rep. Ro Khanna, a California Democrat, introduced a bill Monday that would ban self-improving AI models until the federal government establishes safety guardrails and a new federal agency approves such activities. No House bills are expected to receive a vote before the midterm election.

Daniel Kokotajlo, a former OpenAI governance researcher who now leads the AI Futures Project, said the attitude among senior lab leadership is one of self-justification — that pausing would only hand an advantage to less scrupulous actors — a view he relayed to Reuters.