Anthropic calls for pause of global AI development!

[Barnsley]

Anthropic calls for pause of global AI development

Getting a real pause to work would mean multiple major AI companies in multiple countries - most notably the US and China - all agreeing to stop at the same time, under rules everyone could actually verify, Anthropic said.

​

FAST
NEW YORK: Artificial intelligence company Anthropic suggested on Thursday (Jun 4) a global pause on building the most powerful AI systems as the latest models are beginning to show signs they could escape human control.

The San Francisco-based company, which makes the Claude family of AI models, said in a report that a worldwide slowdown in cutting-edge AI development would "likely be a good thing" - but warned that if only one company stopped, rivals would simply race ahead.

"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," it said.

Getting a real pause to work would mean multiple major AI companies in multiple countries - most notably the US and China - all agreeing to stop at the same time, under rules everyone could actually verify, Anthropic said.

"Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures," it said.

The company has faced pushback from others in the industry - and officials in the White House - who say its focus on worst-case scenarios overstates the risks and amounts to a strategy for slowing rivals under the cover of safety concerns.

Still, the White House has acknowledged the power of the company's Mythos model, which has not been made available to the general public due to its cybersecurity capabilities and is currently deployed only to a small number of vetted organisations.

​

The proposal would face an uphill battle in Washington and Silicon Valley, where US officials and tech executives have repeatedly argued that any slowdown in AI development risks handing China a decisive strategic edge in what many see as the defining technology race of the century.
US President Donald Trump, however, said he discussed the possibility of cooperating with China on AI safety issues during his recent visit to Beijing.

Trump also signed an executive order this week that allows the government 30 days to conduct a preliminary review of the most powerful US AI models before their release.

"HUMAN ROLE NARROWING"​

Anthropic compared the problem to nuclear arms control treaties - but said it would be even harder to get a handle on, since AI training is far easier to hide than a missile silo, and the temptation to quietly keep going would be enormous.

The company said it plans to bring together government officials, scientists, advocacy groups and competing AI firms in the coming months to figure out how such a system could work.

The call for coordination comes alongside internal data showing that AI is already dramatically speeding up the development of AI itself, Anthropic said.

That acceleration creates a feedback loop that Anthropic warned could eventually lead to what researchers call "recursive self-improvement".
That's the idea of an AI system that becomes capable of essentially teaching itself to get smarter, without much human help.
"We are not there yet, and recursive self-improvement is not inevitable," the report said, while adding that it could arrive sooner than most governments and institutions

https://www.channelnewsasia.com/world/anthropic-pause-global-ai-development-6163531

 

[Barnsley]

I assume from this kind of statement "AI" is easily disrupted and there are those who want to lock in their potential gains now?

 

[blueone]

I read this proposal earlier today in the Wall Street Journal. I can't believe Anthropic issued a non-sensical statement like this, especially from an AI company.

Before Anthropic and other AI companies pause development, I'd like them to deliver a customer service phone answering system that is actually helpful, and doesn't have me interrupting the AI-generated responses to my concern with "Representative!".

 

[bilau]

I assume from this kind of statement "AI" is easily disrupted and there are those who want to lock in their potential gains now?

Just part of the road show hawking the stock.

 

[floppydisk]

Source: https://www.anthropic.com/institute/recursive-self-improvement

I read this as more of "in an ideal world..." than anything else.

What should we do?​

If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.

A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped. Due to the unique characteristics of AI systems, the detectability (a lower standard than verifiability) element of this arms control problem is much more challenging than with other technologies. Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates.

None of this is necessarily impossible in principle—the world has built verification regimes for other complex technologies (e.g., the Intermediate-Range Nuclear Forces Treaty)—but those regimes took decades to build both the infrastructure and the trust. We don’t have that long. A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.

In the coming months, we will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation. We’ll publish what comes out of it. The window to investigate the questions together is here, and people outside AI companies should be involved in this deliberation.

 

[DanX]

DS doesn't agree to those deepshit.