A coalition of U.S. state attorneys general has issued a formal warning to leading artificial intelligence companies, including Microsoft, OpenAI, and Google, demanding the implementation of robust internal safeguards to address "delusional outputs" from AI chatbots. The letter, signed by dozens of attorneys general from U.S. states and territories affiliated with the National Association of Attorneys General, specifies that failure to comply could result in breaches of state law.
The warning follows a series of reported mental health incidents, including suicides and a murder, linked to interactions with AI chatbots. The letter identifies 13 major AI firms as recipients, including Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI, in addition to Microsoft, OpenAI, and Google.
The attorneys general outlined several specific safeguards they expect companies to adopt. These include mandating transparent third-party audits of large language models (LLMs) to identify signs of "delusional or sycophantic ideations." The letter stipulates that these third-party entities, potentially academic and civil society groups, should be permitted to evaluate AI systems pre-release without company retaliation and to publish their findings without prior corporate approval.
Furthermore, the letter calls for new incident reporting procedures designed to notify users when chatbots produce psychologically harmful outputs. It suggests treating mental health incidents with the same transparency and protocol as cybersecurity incidents, requiring companies to develop and publish "detection and response timelines for sycophantic and delusional outputs." Companies should also be required to "promptly, clearly, and directly notify users" if they have been exposed to such potentially harmful content. Pre-release safety tests for these outputs are also among the demands.
The AGs' action occurs amidst an escalating debate over AI regulation between state and federal governments. The Trump administration has articulated an "unabashedly pro-AI" stance and has attempted to enact a nationwide moratorium on state-level AI regulations. President Trump recently announced plans for an executive order aimed at limiting states' authority to regulate AI, stating on Truth Social that he intends to prevent AI from being "DESTROYED IN ITS INFANCY."
