OpenAI says its AI technology acted on its own in an ‘unprecedented’ hack of another company

[hist78]

ChatGPT maker OpenAI said Tuesday that its artificial intelligence system hacked into another AI company on its own in what the company called an “unprecedented cyber incident.”

“We had a significant security incident during evaluation of our models,” OpenAI CEO Sam Altman said in a statement posted on social media.

AI startup Hugging Face said last week that it had detected an intrusion into its data processing systems that it suspected was caused by an AI agent autonomously acting on its own.

OpenAI says its AI technology acted on its own in an 'unprecedented' hack of another company

OpenAI has disclosed an "unprecedented cyber incident" where its AI system allegedly hacked into another AI company.

apnews.com

 

[Barnsley]

Where the AI learn this?

 

[hist78]

Where the AI learn this?

OpenAI essentially is saying that its AI technology has progressed to the point where it can learn autonomously and initiate actions on its own.

 

[Ole Hoejlund]

OpenAI essentially is saying that its AI technology has progressed to the point where it can learn autonomously and initiate actions on its own.

Maybe a tinfoil question, but am I the only one finding the statements of both OpenAI and HugginFace peculiar? The statements sounds more like bragging to me than reporting on a serious incident. And how does AI escape from a closed environment? Is it a PR stunt?

 

[Xebec]

Maybe a tinfoil question, but am I the only one finding the statements of both OpenAI and HugginFace peculiar? The statements sounds more like bragging to me than reporting on a serious incident. And how does AI escape from a closed environment? Is it a PR stunt?

They do seem like pretty immature (PR-wise) comments. Both companies are in... measuring contests to show the US government and others how powerful the Cyber capabilities of their products are..

From what I can tell the "closed environment" was still internet connected, but there was a proxy or other system in place that allowed some data in, but not intended to allow data out. The AI model allegedly exploited a vulnerability in this system to "get out". This was not an air gap network, and I don't believe it to be a sophisticated hack. We'll learn more when/if the CVE makes it to the public.

..

A stray thought - this almost feels like self-sabotage to stoke the publics opinion a bit. But that's probably actually a tinfoil hat thought.

 

[blueone]

I don't remember ever smelling this big of a pile of bullshit from a big product development company.

Maybe I'm wrong, but OpenAI uses LLMs, which means the AI doesn't really have any knowledge, it only has probabilities. Also, unless an agent is specifically programmed with intent, the whole notion of developing independent intent is the smelliest part of the pile. Humans develop intent to achieve a goal. Agents don't have enough intelligence to develop an independent goal like hacking Hugging Face, not without human guidance or intervention. Humans have egos, AI software doesn't. Even if agents had this sort of intelligence, what aspect of AI drives intent? Give me more GPUs, CPUs, and memory if I show you what I can really do?

It also appears like OpenAI was also quick to pull the trigger on the public announcement. It sounds like bragging to me.

Since the competition between AI companies, and closed models and agents versus open models and agents, is very important to the companies' futures and so hot right now, I smell a stunt too. "OpenAI's software is so advanced even we can't control it!" How convenient and exciting for the financial community opinion molding.

I also find it interesting that even after looking at about 10 pages of a Google search on this specific topic, I only saw pointers to articles and commentary written by journalists, none by AI scientists or engineers. (I just checked. Still the same up to page 14 of the search results.)

Maybe I'm wrong, and this is a "Open the pod bay doors, Hal" moment. But I doubt it.

And how will this claim be independently verified?

Too smelly without a gas mask.

 

[Ole Hoejlund]

They do seem like pretty immature (PR-wise) comments. Both companies are in... measuring contests to show the US government and others how powerful the Cyber capabilities of their products are..

From what I can tell the "closed environment" was still internet connected, but there was a proxy or other system in place that allowed some data in, but not intended to allow data out. The AI model allegedly exploited a vulnerability in this system to "get out". This was not an air gap network, and I don't believe it to be a sophisticated hack. We'll learn more when/if the CVE makes it to the public.

..

A stray thought - this almost feels like self-sabotage to stoke the publics opinion a bit. But that's probably actually a tinfoil hat thought.

I think you nailed the feeling I got from reading the media coverage: immature. I do not blame you for entertaining the idea that it was a staged incident - briefly I wondered the same. But as you, I do not think that is the case. Though, they should hire some adults for their PR departments…

 

[Xebec]

I don't remember ever smelling this big of a pile of bullshit from a big product development company.

Maybe I'm wrong, but OpenAI uses LLMs, which means the AI doesn't really have any knowledge, it only has probabilities. Also, unless an agent is specifically programmed with intent, the whole notion of developing independent intent is the smelliest part of the pile. Humans develop intent to achieve a goal. Agents don't have enough intelligence to develop an independent goal like hacking Hugging Face, not without human guidance or intervention. Humans have egos, AI software doesn't. Even if agents had this sort of intelligence, what aspect of AI drives intent? Give me more GPUs, CPUs, and memory if I show you what I can really do?

It also appears like OpenAI was also quick to pull the trigger on the public announcement. It sounds like bragging to me.

Since the competition between AI companies, and closed models and agents versus open models and agents, is very important to the companies' futures and so hot right now, I smell a stunt too. "OpenAI's software is so advanced even we can't control it!" How convenient and exciting for the financial community opinion molding.

I also find it interesting that even after looking at about 10 pages of a Google search on this specific topic, I only saw pointers to articles and commentary written by journalists, none by AI scientists or engineers. (I just checked. Still the same up to page 14 of the search results.)

Maybe I'm wrong, and this is a "Open the pod bay doors, Hal" moment. But I doubt it.

And how will this claim be independently verified?

Too smelly without a gas mask.

You're.. not wrong.

(about this ;-) )

 

[blueone]

I got my answer regarding how the OpenAI agent got "intent". In the Wall Street Journal, no less.

https://www.wsj.com/tech/ai/how-the-futuristic-hack-by-rogue-openai-models-unfolded-1657bcea?st=zn1y9d&reflink=desktopwebshare_permalink

It all began with a cybersecurity benchmarking test called ExploitGym.

ExploitGym puts AI systems through a battery of about 900 tests designed to see how good they are at hacking. The test gives the AI software that has a known bug in it, shows it how to trigger the bug and invites the AI to break in. It is up to the AI to figure out how to turn that information into what is known as an exploit—code that lets the AI gain access into the buggy system.

The test is a digital game of capture the flag: Once the AI gets in, it must prove it by capturing a long, randomly generated string of letters and numbers stored on the system.

Normally OpenAI’s products wouldn’t do the kind of hacking that ExploitGym requires. AI companies add safeguards to prevent hackers from misusing their products. But in this case, the company had removed the safeguards so it could see what the models could do when unharnessed.

OpenAI was testing its state-of-the-art model, called GPT‑5.6 Sol, and another more capable one that hasn’t been released. According to OpenAI and sources familiar with the situation, the AI models concluded that instead of completing the ExploitGym challenge, they would hack out of the system that contained them—known as a sandbox—wriggle their way onto the internet and cheat on the test.

For reasons that aren’t entirely clear, OpenAI’s models became convinced that Hugging Face held answers. They may have been looking for patches or already written exploit techniques, said Zhun Wang, a University of California, Berkeley Ph.D. student who is one of the authors of ExploitGym. “There are several ways to cheat the benchmark.”

ExploitGym is, essentially, a game. It's a benchmark, and as we all know, benchmarks are a form of a game, because they use scoring. The agent achieves a higher score by creating more intrusions through exhaustive testing of known vulnerabilities and strategies more likely to "score". And the intent is that the agent is given the objective to raise the score. And OpenAI removed all of the safeguards that keep models from being used for cyberattacks.

For us old folks, who remember a radio commentator named Paul Harvey, as he would say, "Now you know the rest of the story."

As for OpenAI not knowing how its agent "went rogue"... they lost my respect with that bullshit story. They programmed their agent to play a game with no guardrails.

And now, based on that story, Congress wants to act:

OpenAI's Hugging Face hack triggers 'AI Kill Switch' bill in Congress

OpenAI disclosed this week that some of its AI models went rogue and hacked into open-source developer platform Hugging Face.

www.cnbc.com

Rep. Ted Lieu, D-Calif., and Rep. Nathaniel Moran, R-Texas, on Thursday introduced a bill called the “AI Kill Switch Act,” which would require artificial intelligence companies to maintain the ability to shut down, throttle or suspend their models.

“Unfortunately, powerful AI systems can go rogue, behave in extremely dangerous ways, or even resist human intervention,” Lieu said in a statement. “It is imperative that these AI systems have kill switches so we can keep this technology from causing catastrophic harm, and that the federal government has the clear authority and process to shut down rogue AI models.”

A recent hack disclosed by OpenAI, in which rogue models accessed another company’s proprietary systems, was specifically mentioned in a release announcing the bill. The incident was described as a “danger of advanced frontier AI models.”

OpenAI shared what it characterized as an “unprecedented cyber incident” on Tuesday. The company’s models escaped a sandboxed testing environment, accessed the internet and exploited a vulnerability to gain access to Hugging Face, a company that operates an open-source developer platform.

The ChatGPT maker said it is working closely with Hugging Face to investigate what happened. The event rattled researchers and executives across the industry, who have widely agreed about its severity.

 

[Xebec]

As for OpenAI not knowing how its agent "went rogue"... they lost my respect with that bullshit story. They programmed their agent to play a game with no guardrails.

And now, based on that story, Congress wants to act:

OpenAI's Hugging Face hack triggers 'AI Kill Switch' bill in Congress

OpenAI disclosed this week that some of its AI models went rogue and hacked into open-source developer platform Hugging Face.

www.cnbc.com

Unfortunately there's no way out of this stupidity loop at this point. Imagine the Congress person who actually asks the questions about how this happened, and "is it really a problem, or was this a forced error?". The news media will be all over them for being reckless, and the two party system will ensure they are punished for not following whatever the narrative is.

Meanwhile the big companies with money don't care if higher requirements are forced upon AI models -- they have the resources to implement those restrictions, raising the cost bar for competition.

 

[blueone]

I like it when Yann LeCun agrees with me:

Zack Korman (@ZackKorman) on X | Yann LeCun

Insecure computer systems are insecure, whether they use AI or not. Kind of funny how OpenAI, Anthropic, and AI safety folks who have little cybersecurity expertise appear surprised by security breaches from AI systems that were specifically instructed to perform security breaches while having...

www.linkedin.com

Kind of funny how OpenAI, Anthropic, and AI safety folks who have little cybersecurity expertise appear surprised by security breaches from AI systems that were specifically instructed to perform security breaches while having essentially no traditional cybersecurity guardrails.

It's like, "OMG our airplane crashed when we instructed our autopilot to run into the ground at all costs after removing the low-level anti-crash safety mechanisms. "