AI hallucination nearly triggered US military attack
The US military narrowly avoided intercepting a Chinese vessel after an artificial intelligence chatbot falsely reported that the ship was carrying nuclear weapons components.
According to a CNN report, a US Special Operations Command analyst used an AI chatbot to evaluate a Chinese ship's cargo manifest. The tool blended open-source data with classified signals intelligence, erroneously concluding that the vessel was transporting nuclear weapons components through the Middle East. Based on this false intelligence, the US military prepared an armed interception with air support, a move that one source warned "almost started a war" before officials caught the error.
The near-miss highlights the severe risks of the Pentagon's aggressive push into generative AI. In January, the Department of Defense launched an AI acceleration strategy to integrate data across all military branches. Defense Secretary Pete Hegseth defended the initiative, stating that "AI is only as good as the data that it receives." To support this, the military has integrated several commercial models. Last December, the Pentagon adopted Google's Gemini for Government to power its GenAI.mil platform, recently adding Grok for Government as an option. Anthropic also provides a tailored version of Claude for intelligence operations.
The scale of AI adoption within the military is already massive. In June, a Pentagon representative informed Congress that approximately 1.5 million active Department of Defense personnel have utilized the military's generative AI systems, which are also used to draft congressional reports. While a 2023 State Department declaration emphasized that military AI must maintain "a human in the loop," the rapid deployment of these tools has outpaced safety guarantees. This tension was highlighted in March when the Pentagon blacklisted Anthropic over its stance against autonomous weapons, a decision a federal judge recently ruled unlawful.
For defense analysts and AI practitioners, this incident serves as a stark warning about the limits of large language models in high-stakes environments. It proves that even when combining classified and open-source data, chatbots remain prone to severe hallucinations that can fabricate national security threats. Relying on automated summaries without rigorous, independent verification of the underlying source data can lead to catastrophic geopolitical consequences.
This is our own summary of reporting by Ars Technica AI
