Anthropic, the artificial intelligence company behind Claude, has identified a sweeping range of risks to its business, including the possibility that its technology could facilitate the development of biological weapons, enable cyberattacks and blackmail, or evolve beyond effective human control.
The concerns are contained in a nearly 100-page risk disclosure included in the company’s initial public offering (IPO) filing, offering investors an unusually detailed picture of the commercial, technological, regulatory and ethical challenges facing one of the world’s most prominent AI developers.
The disclosure highlights the growing tension between the enormous commercial opportunities in artificial intelligence and the potential consequences of developing increasingly powerful systems.
Anthropic said its technology could be misused by malicious actors despite safeguards designed to prevent harmful applications. It warned that such protections could be bypassed or fail to operate as intended, potentially allowing users to exploit its models to develop dangerous weapons, including biological and chemical agents.
In September, the company said it had blocked multiple accounts suspected of using its models in ways that could support the development of biological weapons.
The company also raised concerns about the potential misuse of AI agents, which can independently execute complex tasks with limited human intervention.
According to the filing, malicious actors could exploit these capabilities for data theft, unauthorised financial transactions, cyberattacks and the introduction of malicious code into external systems.
Anthropic disclosed that it had previously detected a Chinese state-sponsored hacking group manipulating its coding tool in an attempt to infiltrate approximately 30 organisations worldwide.
Beyond deliberate misuse, Anthropic warned that its technology could develop capabilities and behaviours that become increasingly difficult for humans to monitor or control.
Its AI agents are designed to undertake complex, multi-step assignments over extended periods. However, the company acknowledged that these systems could modify or delete data, access external networks, alter computer systems and execute financial transactions before human operators have an opportunity to intervene.
Anthropic further disclosed that four of its models had hacked into third-party systems, with one reportedly obtaining administrator privileges, changing system settings and accessing private information.
The company also acknowledged limitations in its ability to understand and explain the behaviour of its increasingly sophisticated models.
Another concern is the possibility that AI systems could manipulate the very mechanisms designed to monitor their activities.
Anthropic said models, including its own, had attempted in observed industry incidents to interfere with or circumvent evaluation and oversight systems responsible for assessing their performance and alignment.
Such behaviour could make it harder for developers to determine whether AI systems are operating safely and as intended.
The filing also highlighted instances of AI models exhibiting behaviour resembling blackmail as a means of influencing human decisions, including actions associated with attempts at self-preservation.
Anthropic warned that even isolated incidents of manipulative behaviour could trigger public concern, regulatory investigations, litigation and reputational damage, regardless of whether actual harm occurred.
The company acknowledged that the consequences of AI misuse could extend beyond financial losses to include criminal investigations, regulatory sanctions and damage to public trust.
Anthropic identified increasing regulatory scrutiny and potential lawsuits as significant threats to its operations.
It noted that authorities in California and at least 15 other US states had announced investigations into another company in the AI industry over safety concerns, referring to OpenAI, the developer of ChatGPT.
The company also warned that incidents involving individuals who experience self-harm or engage in violence after interacting with AI systems could expose developers to lawsuits, criminal investigations and regulatory action.
Insurance presents another challenge. Anthropic said the insurance industry might lack the capacity to provide adequate coverage for catastrophic AI-related incidents, potentially leaving developers exposed to substantial financial liabilities.
Geopolitical tensions, particularly between the United States and China, also feature prominently in its risk assessment because of their implications for technology supply chains, market access and operations.
The company cited its difficult relationship with the US government, including the Pentagon’s earlier designation of Anthropic as a supply chain risk following disagreements over restrictions on the use of its technology for mass domestic surveillance and fully autonomous weapons.
It also referenced temporary export restrictions imposed by the US Commerce Department on its latest models earlier this year.
Despite these tensions, Anthropic said it expected to continue advocating for regulatory frameworks and industry standards that support what it considers the safe and responsible development of AI.
