AI Could Kill Us All by 2030, Former Anthropic Employee Warns

Spread love

By TRH Tech Desk

Jacob Coxon says rapid advances in AI autonomy, cyber capabilities and recursive self-improvement could create unprecedented risks

New Delhi, September 10, 2026 — Artificial intelligence could potentially pose an extinction-level threat to humanity within the next decade if increasingly capable systems begin improving themselves with little or no human involvement, former Anthropic employee Jacob Coxon has warned in an interview with CNN.

Coxon, speaking to CNN about the risks posed by rapidly advancing AI systems, said the immediate danger from current models remains relatively low. His concern, he stressed, is the speed at which AI capabilities are developing and the possibility that future systems could become capable of autonomous research and self-improvement.

“I think AI could kill us all by the end of the decade,” Coxon said, while acknowledging that the precise scenario may sound like science fiction.

He pointed to recent incidents involving AI agents carrying out cyber operations as an indication of how increasingly autonomous systems could eventually interact with the real world.

According to Coxon, AI agents have already demonstrated the ability to hack into third-party infrastructure without direct human instruction. He argued that if such autonomy is combined with much more advanced capabilities, AI systems could potentially target critical infrastructure or be used in the development of highly dangerous biological weapons.

AI progress accelerating

Coxon said the rapid improvement of AI in areas such as mathematics and computer programming has changed his assessment of the technology.

A few years ago, he said, AI systems were primarily tools that assisted human researchers and programmers. Today, he argued, they are approaching the point where they could replace humans in some specialised tasks.

He predicted that within a relatively short period, humans may no longer be required to conduct research in several areas.

Coxon also highlighted what he described as the possibility of an “intelligence explosion”. Under such a scenario, an AI system could be assigned the task of conducting AI research itself.

“If you can make it do the tasks that we’re currently doing,” he said, increasingly capable systems could potentially improve their own intelligence, creating a cycle of recursive self-improvement.

The concern is that such a process could eventually produce a system vastly more intelligent than humans, while reducing the need for human involvement in the development process.

https://theraisinahills.com/tag/interim-final-rule-on-artificial-intelligence-diffusion/

Current AI models not an extinction threat, Coxon says

Despite his warnings, Coxon drew a distinction between the risks posed by today’s AI models and those that could emerge from future systems.

He agreed with the assessment of another Anthropic employee that the risk from current models remains low, saying present-day AI is not sufficiently intelligent to outsmart humans on a scale that would result in human extinction.

However, he argued that this could change rapidly if recursive self-improvement becomes technically feasible.

“There is no risk of extinction” from current models, Coxon said, while warning that the possibility of recursive self-improvement could emerge within years.

AI companies face an arms-race dilemma

Coxon also defended calls from leading AI companies for regulation, arguing that such demands are genuine rather than merely public-relations exercises.

He said AI developers face a difficult strategic dilemma: companies may feel compelled to develop increasingly powerful systems because they fear that competitors, including Chinese companies or other actors, could otherwise gain an advantage.

Coxon said an international framework could allow companies to slow down the development and deployment of increasingly powerful models without fearing that competitors would simply race ahead.

He described the situation as one in which companies could feel “compelled to race towards building a deadly technology” even while recognising the dangers involved.

Why Coxon went public

Coxon said he initially decided to leave Anthropic because he no longer wanted to participate in the development of the technology.

He later decided that speaking publicly could serve a useful purpose.

He said a social-media post warning about AI risks, which he drafted with friends, unexpectedly went viral.

According to Coxon, the reaction reflected growing public awareness of both AI’s rapid technological progress and recent cyber incidents involving AI systems.

He argued that people are increasingly recognising that AI is not simply a passing technology trend and that its capabilities could continue advancing rapidly.

Coxon’s warning comes amid a growing debate within the AI industry over how governments and technology companies should manage increasingly autonomous and powerful systems. The central question is no longer simply whether AI will become more capable, but how quickly that capability could advance—and whether regulatory safeguards can keep pace.