Over 100 Companies Sign On To Nvidia’s Rogue AI Crackdown, But Not OpenAI

The artificial intelligence lab at the center of a string of rogue-agent incidents is conspicuously missing from a new industry effort to contain them.

OpenAI was not among the more than 100 organizations that Nvidia listed as working with the chipmaker’s new Open Agent Safety Platform, which launched Monday, according to the company’s announcement. The safety platform includes Sentry, a hardware monitor that can quarantine an AI agent within milliseconds if it tries to move outside its permitted boundaries.

The platform had the potential to prevent incidents like the July breach of New York-based AI startup Hugging Face by OpenAI agents, according to an Nvidia executive.

“From what we know, this new security platform could have stopped the breach if it was being used in frontier labs for model evaluation early on,” Justin Boitano, Nvidia vice president and general manager of enterprise computing, said during a Sunday media briefing. “We built the Open Agent Safety Platform to advance AI safety and engineering for the industry. We’re advancing this openly, and we want to engage everybody to work with us on advancing a new layer of what we call agent security.”

“Each security incident is unique, and we have to look at all of them in detail,” Boitano added. “With this one, from what we know, Hugging Face reported over 17,000 agents attacking their infrastructure, which went on for days and weeks.”

“I did note with interest that OpenAI was not included in the announcement,” Riki Parikh, policy director at the Alliance for Secure AI, told the Daily Caller News Foundation. “That is notable, especially given the fact that a lot of the incidents this summer have been OpenAI and their internal models.”

Nvidia announced a roughly $13 billion deal to acquire Hugging Face on Sept. 3.

OpenAI said Friday that it had notified “dozens” of third-party organizations that may have been affected by its agents’ unauthorized actions, as part of its new “model misalignment” reporting framework, according to the company’s announcement. On Thursday, the Australian government said the company’s agents acquired unauthorized access to its medicare portal in June.

Sentry operates on a BlueField-4 data processing unit, an Nvidia chip that sits separately, or “out-of-band,” from the agent’s system, where an agent cannot access it, according to Nvidia’s Monday blog post. The chip is placed along the path an agent’s requests must travel to reach its AI model.

Parikh said the platform seemingly shows Nvidia taking public concerns over AI safety seriously, but he cautioned that the company’s claims have yet to be independently tested.

“At this point, we’re just taking Nvidia at its word on the efficacy of this technology,” Parikh said. “I prefer to hear independent safety researchers confirm that claim.”

OpenAI and Hugging Face each did not immediately respond to the Daily Caller News Foundation’s request for comment.

“[AI’s] full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility,” Nvidia CEO Jensen Huang said in a Monday X post discussing the announcement of the platform. “This is bigger than a single product. It’s the beginning of an open ecosystem to build the trust layer for safe agent systems.”

The company also announced Monday that it can spend up to an additional $150 billion buying back its own stock.

SpaceXAI and Anthropic are among the 100 organizations coordinating with Nvidia to develop the safety platform.

“As customers rely more on agents to get real work done, safety should be enforced outside the model by additional controls the agent can’t get past,” Mike Nicolls, president at SpaceXAI, said in the press release.

“Claude Managed Agents gives companies a clear view of what each agent is doing, and NVIDIA’s platform adds another layer of governance and control across hardware and software,” Paul Smith, chief commercial officer of Anthropic said, according to the press release.

Parikh argued industry tools like this platform are not enough on their own to address the issues rogue agents pose to the public.

“This actually requires significant effort to put safeguards in place at the federal level, put mandatory testing in place and allow for a slowdown of the development of this technology,” Parikh told the DCNF.