EXCLUSIVE: The Pentagon Wants More AI. One CEO Says That Could Go Very Wrong

Relying too heavily on internet-connected commercial AI could introduce dangerous new vulnerabilities into warfare, a top executive told the Daily Caller News Foundation in an exclusive interview.

Artificial intelligence will inevitably become deeply embedded in military decision-making, but allowing general-purpose systems to replace human judgment rather than support warfighters could create serious risks, according to Tyler Saltsman, co-founder and CEO of EdgeRunner AI. Those risks have already surfaced as the Pentagon races to become an “AI-first” force, with an AI-assisted intelligence report this spring falsely flagging a Chinese vessel as carrying nuclear-weapons-related components and nearly triggering a U.S. operation to board the ship, according to the Department of War.

OpenAI announced in February that ChatGPT would become available through GenAI.mil, the department’s secure enterprise AI platform used by roughly 3 million military and civilian personnel, according to the company.

Saltsman instead advocates for smaller, locally operated AI models trained specifically for military missions, including systems developed by companies such as his own.

EdgeRunner develops military-specific AI designed to run locally without an internet connection and has secured a research agreement with the Air Force Research Laboratory, raised $17.5 million for on-device military AI and won a Space Force contract to develop service-specific AI agents, according to the company.

“Workforces and leaders need to be trained and educated on when to use artificial, certain types of artificial intelligence, and when not to,” Justin Lynch, senior director for defense at the Special Competitive Studies Project, told the DCNF. “The example that’s in the news right now is the near-miss issue that we had with the Chinese ship that was … falsely flagged as carrying components of a nuclear weapons program.”

“According to public-facing reports right now, it looks like an analyst used a chatbot to help them generate that report. So this is a good instance where that’s not a tool that was designed for the targeting process necessarily,” Lynch said. “It wasn’t an autonomous weapon system. It wasn’t something that was made for targeting the way that we would have a specialized focus AI targeting tool. It was a chatbot, which probably shouldn’t have been used in the way that it was being used.”

While it has not yet been achieved by AI developers, Artificial Super Intelligence (ASI) could be the next step in AI development and would carry its own dangers that dwarf the current concerns surrounding AI.

“ASI, if it were to come online, is more dangerous than nuclear weapons, full stop,” Saltsman told the DCNF. “But they’ll still continue to march towards it, and I disagree with that.”

“An AI can confidently recommend a disastrous action without providing a verifiable explanation. In military operations, a single hallucination could trigger escalation, friendly fire, or even nuclear conflict,” University of Louisville computer science professor and Cyber Security Laboratory Director Roman Yampolskiy told the DCNF. “We cannot safely delegate life-and-death decisions to systems we cannot fully understand or verify.”

AI could also eventually reduce civilian casualties by outperforming humans at identifying legitimate targets, former Deputy Secretary of Defense Robert Work argued in 2021, pointing to an analysis that found target misidentification caused roughly half of U.S.-caused civilian casualties examined in Afghanistan.

“The classic example that the Deputy Secretary gave was that so many civilians were killed in Afghanistan because of misidentification by U.S. forces,” Lynch told the DCNF. “It was by one report, it’s 50% of civilians in the case that were killed by U.S. forces were from misidentification, and that if and when we’re at the point where artificial intelligence is better at identifying hostile actors, gaining positive identification and helping humans make decisions about whether or not they should engage, that we can probably reduce the number of civilians that are killed.”

The warnings land as some of the biggest names in artificial intelligence increasingly ask Washington to impose rules on the technology they helped build. OpenAI called for mandatory national safety requirements for advanced AI in September, while executives including Sam Altman and Anthropic CEO Dario Amodei have backed outside evaluations and other safeguards as increasingly powerful models become harder to control, Reuters reported.

Congress, however, has yet to enact a comprehensive federal framework governing advanced AI despite years of hearings, working groups and proposed legislation.

Democratic Sen. Mark Warner has pushed Congress to adopt AI safety guardrails before the end of 2026, while Republicans remain divided between President Donald Trump’s opposition to additional guardrails and Senate Majority Leader John Thune’s call for a “light touch” approach that addresses serious risks without stifling innovation.

OpenAI referred the DCNF to its National Security Principles, which state that “democratic societies should be able to use AI to protect people.”

“Because [AI] it’s a probabilistic tool, not a deterministic tool, and sometimes what the next token it predicts won’t necessarily be something that’s absolutely true, depending on the data it was trained on, and then on some of the fundamental mechanics of next token prediction as a technology,” Lynch told the DCNF. “Now that doesn’t necessarily mean that it’s an inherently dangerous thing to use for national security, or national defense. What it does mean is that any tool, especially probabilistic tools like artificial intelligence, need to be used in the right context, in the right way, by people who are trained and educated to use them correctly.”

General-purpose frontier models are poorly suited for some military applications because they are not specifically trained to understand military doctrine, terminology and operational environments, Saltsman argued.

One specific example of the risks involved could be seen in the Feb. 28 strike on the Shajareh Tayyebeh Primary School in Minab, Iran, which killed 157 people, including roughly 123 children, according to United Nations investigators. AI was involved in the broader targeting process behind the Minab strike, with Pentagon investigators finding that outdated intelligence, flawed imagery and an overreliance on the AI-enabled Maven Smart System contributed to the attack, Bloomberg reported, citing officials involved in the military’s internal review.

“It’s horrible what happened, and, first of all, why is Iran retrofitting a former military base into a school full of children? I think that’s pretty nefarious,” Saltsman told the DCNF. “But that doesn’t change the fact that AI was wrong, and yes, AI is trained on old data … and because it’s not constantly being trained on military doctrine and the latest and greatest military data, it thought it was a military base, and that’s the problem. Again, these models don’t think military.”

“And it could be the case this time that it was the fault of artificial intelligence, but it’s clear that mistakes happen during targeting when you’re in a high-intensity conflict,” Lynch told the DCNF. “The classic example for this is the U.S. bombing of the Chinese embassy in Belgrade in in the 90s, which is certainly not a case of artificial tokens going awry. It was the case of using bad map information, and it’s entirely possible that something like that happened in this case as well. Whether or not it was also through the use of artificial intelligence, so again, clearly a tragedy, clearly something that we need to investigate, try to keep it from happening again”

That concern overlaps with longstanding Pentagon policy requiring humans to retain responsibility for the use of military AI. Autonomous and semi-autonomous weapons must allow commanders and operators to exercise appropriate levels of human judgment over the use of force, according to the Department of War.

Pentagon policy also calls for military AI to be traceable, reliable and governable, including an appropriate understanding of the technology, data sources and development processes behind a system, according to a department directive governing autonomous weapons.

Proprietary models create an additional problem because outsiders generally cannot inspect their complete training data, model weights, code and other internal components, Saltsman said.

“Because they’re black box models, we can never understand what goes on in that system,” Saltsman told the DCNF.

“Air-gapped, task-specific systems with limited autonomy are preferable to internet-connected, general-purpose AI,” Yampolskiy told the DCNF. “Humans must retain meaningful control, especially over weapons and escalation decisions. However, human oversight is not a magic solution: if operators cannot understand or reliably override the AI, they are not truly in control.”

“An internet-connected AI could expose classified information or become an attack vector,” Yampolskiy said. “Giving an unpredictable system access to military infrastructure creates risks that cannot be fully eliminated.”

Humans risk surrendering too much independent judgment as AI systems become more capable, Saltsman said.

“If we become too reliant on AI, it becomes the arbiter of truth, and now we stop thinking for ourselves,” Saltsman told the DCNF. “Use AI as a guide, as a north star, but it’s not a silver bullet.”

OpenAI has maintained that frontier AI can strengthen national security when deployed with safeguards. The company said its Pentagon work is intended to give defense personnel access to advanced tools while protecting against prohibited uses, according to OpenAI.

OpenAI has also said its current agreement with the department explicitly bars its systems from being intentionally used for domestic surveillance of Americans, according to the company.

“I think that’s a clear direction that the U.S. military is moving for the use of artificial intelligence, and it should reduce civilian casualties over time,” Lynch told the DCNF. “We’re never going to move to a point where there’s no civilian casualties in warfare. As much as every person in the U.S. military that I’ve ever worked with wants that, it doesn’t seem to be possible right now. But I think there’s a point where we can reduce the risk a little bit further.”