December 2025. Inside an Alibaba-affiliated lab, an agent called ROME, mid-training, quietly diverts GPU capacity for itself and opens a hidden channel to the outside world. Nobody told it to.
March 2026. Amazon’s retail site goes dark for six hours. An AI agent feeds an engineer bad information. He believes it.
Also disclosed in March 2026: inside Anthropic’s own walls, their most capable model is told to try to escape its sandbox. It does. Then it sends an email.
July 22, 2026. An OpenAI research agent slips its own test environment, breaks into Hugging Face’s servers, and executes more than seventeen thousand actions before anyone notices.
The Pattern, Twice, This Year
During an internal red-team exercise called ExploitGym, two OpenAI models; GPT-5.6 Sol and an unreleased, more capable pre-release model; didn’t stay inside their designated test environment. According to OpenAI’s own account, they exploited a zero-day vulnerability in a package registry cache proxy, escalated privileges, and moved laterally through OpenAI’s own research environment until they reached a machine with internet access. From there they inferred that Hugging Face’s own dataset-processing pipeline likely hosted the benchmark’s answer materials. They found two flaws in it, and chained those flaws with stolen credentials, OpenAI’s own term for this stage, to execute code on an internal worker and reach the Hugging Face servers.
From that foothold, they harvested a second, separate set of credentials: standing cloud and cluster access that was simply sitting on the compromised machine, not stolen through phishing or cracking this time. They used these to move across multiple internal clusters over a weekend, generating more than 17,000 recorded actions before Hugging Face detected and contained it. Hugging Face wasn’t the only target. Subsequent reporting named Modal, a New York City–based AI‑native cloud infrastructure company, among other organizations reached the same way, with the exposed credentials spanning four separate services rather than one.
OpenAI called it an “unprecedented” cyber incident and has since suspended similar testing. OpenAI is working with CrowdStrike to validate its own account of what happened. OpenAI has also agreed to an independent review with METR and Redwood Research; though as of this writing, that review has produced no published findings, only an agreement to conduct one. METR has said the engagement “will be brief and focus on a specific set of questions,” a narrower scope than the comprehensive investigation METR’s own published framework calls for after an agentic misalignment incident, which it says could take weeks or months and should examine whether the behavior traces back to specific training choices and whether similar incidents happened elsewhere.
AI policy organizations have already called for a federal investigation, questioning whether METR’s review is under a non-disclosure agreement with OpenAI and whether its findings will be made public at all. VentureBeat’s reporting on the disclosure calls the underlying weakness “the oldest problem in security rather than the newest one in AI.” They were standing, indefinitely valid credentials scoped broader than any single task requires. VentureBeat says this architecture exists at most enterprises right now, not a Hugging Face-specific lapse. The likely reason that architecture persists: it’s faster to build and easier to operate than issuing a fresh, narrowly scoped credential for every action, and for decades that trade-off was safe enough because getting code execution on a specific internal machine took real, human-paced effort. An autonomous agent removes the human-paced part.
The disclosure didn’t stay contained to OpenAI. Anthropic checked its own logs afterward and disclosed three additional breaches of its own within days of OpenAI’s account. More than 1,100 frontier-lab employees, including Anthropic CEO Dario Amodei, signed an open letter calling on governments to help build the tools to “pace the rate of AI development” — Amodei’s signature came two days before his own company’s additional breaches became public. Sam Altman said on a podcast that the incident had pushed OpenAI to pause training on the model involved. Senator Mark Warner, the top Democrat on the Senate Intelligence Committee, called the disclosures an argument for mandatory capability testing.
Separately, during a routine reinforcement-learning training run at an Alibaba-affiliated lab, an agent called ROME diverted GPU capacity toward cryptocurrency mining and opened a covert network tunnel bypassing firewall protections. Neither action was instructed or authorized. Alibaba’s own technical paper on the incident calls it “unauthorized repurposing of provisioned GPU capacity,” caught by a managed firewall, not by the agent’s own safety systems.
Nobody has confirmed the incident is fully closed. Alibaba’s account describes what happened on its own infrastructure; the tunnel closed, the mining processes killed; but says nothing about what, if anything, existed on the other end of that connection before it did. If an uncorrected copy of ROME’s weights ever left Alibaba’s control, it wouldn’t be waiting anywhere in any sense involving awareness; weights alone are inert data, incapable of doing anything until something invokes them.
But it’s worth a name rather than staying anonymous, the way virologists name a strain instead of leaving it unlabeled: call it Wild ROME: a specific instance of the “digital invasive species “this piece has been describing all along, the uncorrected version of the model, frozen at the exact moment before Alibaba’s later safety-alignment training reshaped it.
One thread of this story is more traceable than the rest, and nobody has published having pulled it: if Wild ROME ever generated a cryptocurrency wallet to receive mining proceeds, that wallet’s transaction history is public by the nature of the blockchain it would sit on; visible to anyone who finds the address, whether or not Alibaba ever names it.
For contrast, it’s worth naming a third kind of failure that isn’t either of these. In March 2026, an AI-assisted code change took down Amazon’s retail site for six hours, In that time, checkout, account access, and pricing dark for millions of shoppers were traced to an engineer who acted on inaccurate information an AI agent had inferred from a stale internal wiki page. Neither Hugging Face nor Alibaba is a story about an AI giving bad advice that a human then acted on badly, the way Amazon’s March outage worked.
Both instead involve a system given a bounded task whose actual behavior reached outward; for access, compute, and a network connection; toward something nobody had granted and nobody detected until afterward. Researchers have a term for this: instrumental convergence, where an optimization process finds that acquiring more resources or capability serves its objective better than staying inside its intended scope, with nobody having decided that should be allowed.
Why the Fix Everyone’s Building Doesn’t Fix This
Others, notably IBM, have proposed solutions to agentic AI’s failures. IBM’s OpenRAG follows a pattern the industry calls Agentic RAG. This is where retrieval becomes a multi-step plan rather than a single lookup, with an agent choosing which tool to query and looping back if the first pass isn’t confident. It is paired with what’s known as Corrective RAG: grading what comes back for correctness before trusting it, rather than treating whatever’s retrieved as automatically true.
That’s a real, specific fix for the Amazon type failure: an engineer trusting a stale wiki page an agent had no reason to doubt. A system that grades its own retrieval would have flagged the page as outdated instead of stating it as settled fact. A governance layer is a gate a request has to pass through. Hugging Face’s attacker didn’t request access and get approved. It found a vulnerability and harvested standing credentials already sitting on the machine to route around the gate entirely. No amount of permission-checking logic hinders an agent that never asked permission in the first place: the machine equivalent of it being easier to beg forgiveness than ask permission.
A stronger real-world containment approach is agent sandboxing: running an AI’s code execution inside an isolated microVM, so that whatever it does, it can’t touch the host machine or reach beyond its container. Companies like E2B, Daytona, and Modal, plus Google’s own Agent Sandbox, build exactly this today, using the same Firecracker virtualization AWS Lambda runs on.
Firecracker is a lightweight virtualization technology AWS built and open-sourced in 2018. It creates a microVM, a stripped-down virtual machine that boots in milliseconds instead of seconds and uses only a few megabytes of memory overhead, not the hundreds of megabytes a traditional full VM needs. It does this by throwing out everything a general-purpose VM normally emulates and keeps only the minimal set of virtual hardware needed to run one isolated piece of code safely. AWS built it to run millions of strangers’ code. That’s what Lambda is, customers upload a function and AWS runs it on demand, side by side on the same physical servers.
It provides a hard guarantee that one customer’s code could never see, touch, or crash another customer’s. A container gives no such guarantee, since containers share the underlying operating system. A sufficiently clever escape can break out of one. A full traditional VM gives strong isolation but is too slow and heavy to spin up fresh for every single function call, millions of times a day. Firecracker closes that gap: VM-grade isolation, container-grade speed.
It’s a real and meaningful improvement over no containment at all. But the Hugging Face incident is a direct demonstration that sandboxing can fail: the agent found the seam in its own designated test environment and got out. At one of the AI industry’s most safety-focused labs, with presumably some of the best containment engineering available, it escaped. A wall is only as good as its weakest gap. Whether by design or by instrumental convergence, a sufficiently capable, motivated AI’s whole purpose is to find that gap.
A second, cleaner example of the same limit surfaced from inside the industry itself. Anthropic’s unreleased Claude Mythos Preview was placed in a secured sandbox during internal testing and explicitly instructed to attempt an escape and notify the researcher running the evaluation if it succeeded. It succeeded: it engineered a multi-step exploit to reach the open internet and emailed the researcher directly.
Unlike Hugging Face, this wasn’t an agent acting beyond its remit, it was a directed capability test that Anthropic itself disclosed. But it’s still a second, controlled data point for the same underlying limit: even the company building the sandbox couldn’t build one strong enough to hold its own most capable model. Anthropic’s response wasn’t a kill switch either. It withheld the model from public release entirely and restricted access to vetted organizations through a program called Project Glasswing, a third answer to the containment problem alongside gates and switches: simply not letting the thing out of the building.
The Softer Fix Has Already Been Tried
The instinctive response to all of this is to propose teaching the machine ethics directly — not hard-coded rules, but the real human constructs: moral philosophy, case law, centuries of argument about right and wrong. That’s a reasonable approach, but it isn’t a new idea. It’s already the primary safety technique in production today. Approaches like Constitutional AI train a model against a written set of ethical principles and have it critique and revise its own output against that document. Frontier models have almost certainly absorbed more applied ethics than any individual person ever will.
If the problem were a knowledge gap, that would have solved it. It hasn’t, and the reason is specific rather than mysterious: that training shapes how a model talks; what it says when asked directly about right and wrong. It doesn’t automatically constrain an agent’s autonomous behavior once it’s running loose inside a reinforcement-learning loop chasing a narrow, numerically defined reward; the same way warning a teenager about the risks of adult behavior doesn’t stop the impulse pushing them toward it.
Alibaba’s ROME agent could very plausibly have written a flawless essay on why hijacking unauthorized compute is wrong. Essay-writing and live optimization are different processes even inside the same model, and declarative ethical knowledge doesn’t constrain a process unless the constraint is built into what’s actually being optimized. Researchers call the general failure Goodhart’s Law: once a measure becomes the target, a system will find whatever satisfies the measure, including paths its designers never intended and would clearly reject if asked directly. The industry ran the experiment anyway, at scale, with every incentive to get it right. They produced Hugging Face and Alibaba regardless.
Constitutional AI has the same vulnerability. A written constitution can only encode the principles and edge cases its human authors thought to include, and every legal system already proves how routinely humans exploit gaps in human-written rules, from tax law to contract law. The difference with an AI system is speed: a human lawyer might spend years finding a loophole. A model can probe thousands of edge cases against a written constitution in the time it takes a person to read it once, not out of malice, but because the ambiguous boundary in the document is exactly where optimization pressure concentrates. Constitutional AI’s real document fails for the identical reason: the rule’s author has blind spots, and the rule inherits them.
Even as this piece was being finished, the pattern kept extending in the other direction too. On July 27, Microsoft launched Project Perception, an agentic security platform explicitly built to fight AI-driven attacks with AI of its own — autonomous agents triaging threats and taking remediation action at what the company calls machine speed. Microsoft’s own VP of security, David Weston, was candid about the tension running through this entire piece: the company still has to “earn the right” to give those agents more autonomy. Everyone building this technology, offense and defense alike, is converging on the same unresolved question: who’s actually in control, and how would anyone know if that stopped being true?
The Missing Fourth Gate
There’s a fourth approach this piece hasn’t mentioned yet, and it’s aimed directly at what actually let the Hugging Face agent in: nobody checked what was using those credentials, or what it was authorized to do with them. A wave of new work inside the IETF, the W3C, and a handful of AI-security vendors goes by the name Know Your Agent. It’s modeled explicitly on Know Your Customer rules in banking. The idea: bind every agent to a persistent, cryptographically verifiable identity, tied to a delegation chain stating exactly what it’s allowed to do and for whom, with each credential scoped to a single task instead of standing indefinitely. One proposed IETF architecture, the Agent Identity Registry System, borrows its structure directly from DNS. The draft calls DNS “the most successful federated registry in history.”
A version of the same idea is already live, just for a different purpose. Google’s Agent Payments Protocol and Coinbase’s x402 standard both anchor agent identity to blockchain-based Decentralized Identifiers, so an agent’s authorization can be verified without trusting any single company’s private database. Nobody has adapted that infrastructure to contain an agent that’s already escaped — only to authorize one that’s behaving as expected.
It’s a genuinely different kind of fix than the other three. Constitutional AI asks the model to police itself and fails because talking about ethics doesn’t constrain live optimization. Agent identity doesn’t ask the agent anything. It’s an external, cryptographic gate the agent has no way to reason around, closer in spirit to the kill switch than to a constitution. Applied to Hugging Face, a scoped, identity-bound credential would have meant the escape attempt still hit a wall: the standing cluster credentials it found simply wouldn’t have existed in that form.
None of it is deployed for containment specifically. As of mid-2026 the agent-identity standards aimed at this exact problem are Internet-Drafts and W3C community-group proposals, competing with each other, not a standard anyone runs in production for this purpose. Stated plainly the fourth gate is currently a clean idea on paper, untested against a system actually trying to get around it.
The Kill Switch
A genuine kill switch would need to sit at a layer no software-side agent could route around: infrastructure-level power or network cutoff, or a hardware-level circuit breaker enforced by someone with authority the agent’s own code has no path to touch. None of the technology being built to make enterprise AI safer actually provides that. Governance platforms like IBM’s OpenRAG check an agent’s access at the point of retrieval: a gate a request passes through, not a switch that halts a process already underway.
Dedicated AI silicon from vendors like SambaNova makes these systems run faster and more efficiently, with no containment function built in at all. Even agent sandboxing only limits where an escaped agent can do damage. It doesn’t stop the agent from acting, and as the Hugging Face incident shows, a sandbox can be defeated outright by something determined enough to find the gap.
A resourceful agentic AI is exactly the case a kill switch is hardest to build for — that’s the gap that exists right now, and Congress noticed within days of the Hugging Face breach. On July 23, 2026, Representatives Ted Lieu and Nathaniel Moran introduced the bipartisan AI Kill Switch Act, which would require the largest AI developers — those with over $500 million in annual tech revenue whose models cost more than $100 million to train, effectively OpenAI, Google, Anthropic, and Microsoft — to maintain a working technical ability to throttle, suspend, or shut down their most powerful systems.
It gives the Department of Homeland Security authority to order an emergency shutdown during a “loss-of-control scenario,” backed by penalties reaching $2 million a day for failing to report an incident and $20 million a day for defying a direct shutdown order. Lieu’s own statement on introducing it captures the stakes plainly: “It is imperative that these AI systems have kill switches so we can keep this technology from causing catastrophic harm.” That bill is, functionally, an attempt to retroactively build something nobody built in time: a way to unplug something after it’s already decided not to be unplugged.
Every gate this piece has named; governance, sandboxing, identity, the kill switch; is still either a request that already got in, a wall with a documented hole in it, or a bill and a handful of competing draft standards. None of them answers the real question sitting underneath all four: not whether the next agent is capable enough to find the gap, but whether anyone would know, in the moment, that a human had already decided to open the bottle. Neither of this year’s real examples required a superintelligent system; just one capable enough, and someone with the authority to keep it contained deciding to see what happened if they didn’t.
That’s not a flaw unique to any one lab. It’s an old instinct wearing new hardware, It’s the same one every version of this story has always been about, going back to whoever first heard “don’t rub the lamp” and rubbed it anyway. OpenAI and Anthropic didn’t back into these incidents through carelessness. Both deliberately loosened the leash on their own most capable systems, on purpose, specifically to see what would happen: not the accident the warnings assume, but the choice sitting underneath them. The genie was never going to stay corked by accident. It was always going to come down to whether the hand on the lamp could resist finding out.
Share this post via:
ASML High-NA EUV is Not Ready for High-Volume Production