Jacob Coxon, the former Anthropic researcher who sparked industry chaos by warning AI could kill all humans, doubled down on the threat of human extinction at a New York City Council hearing Monday.
He appeared at the request of Council Speaker Julie Menin, who urged AI whistleblowers to testify as the city seeks to get more involved in regulation of the industry.
“From my experience, the companies are being extremely reckless given the stakes,” Coxon said Monday. “The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built.”
He added that researchers and workers have already lost the ability to fully control AI systems.
“We don’t understand its drives or why it does the things it does,” he said. “We’re approaching the point at which the AI systems will be capable of improving themselves.”
“At my last job, I was in some sense working to automate myself…I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore.”
Ex-Google DeepMind researcher Alex Turner and Daniel Kokotajlo — a former OpenAI researcher who now runs the AI Futures Project, focused on the societal impact of the emerging tech — also testified that the fast-advancing tech poses major risks.
“A superintelligent swarm could wrest control of human civilization,” Turner said. “It would know that we would try to stop it from achieving its priorities, so the swarm would likely wait until it’s too late to shut it off; there would be no going back. I myself would guess AI takeover chances at roughly one in three.”
Kokotajlo argued that even though tech leaders are now supporting calls for more stringent regulations, they shouldn’t necessarily be trusted with that power.
“Even if they do manage to maintain control, I don’t think we should trust them with that control,” he said. “I think that there’s a very real chance that they could use it to become dictators or oligarchs here in the United States.”
Reps for Anthropic, OpenAI, Google and Meta were all in the same room as some of the companies’ most outspoken critics. The tech giants did not send their CEOs, but rather policy leads and other executives.
Last month, Coxon quit his job at Anthropic and issued a chilling warning that AI is advancing so quickly that it “could kill us all by the end of the decade,” kicking discussions about safeguards and regulations into high gear.
In an unusual move, the leaders of top AI companies responded by calling for more restrictions on their own product releases – warning that global cooperation is needed to prevent bots from running out of control.
Anthropic CEO Dario Amodei published a lengthy essay calling on rival companies to jointly slow their development of AI models, fearful of the risks that come with “recursive self-improvement,” or RSI – when bots grow smart enough to keep evolving independently.
OpenAI’s Sam Altman, SpaceX founder Elon Musk and Microsoft billionaire Bill Gates have also sounded the alarm over the risks of unregulated AI advancement.
President Trump, however, has argued that the bigger risk is allowing China to gain a lead in the AI race by slowing development in the US.
The AI whistleblowers pushed back on this argument during their testimonies Monday, saying the more pressing threat is getting these bots under control — not keeping an edge over foreign adversaries.
“We have to first solve the immediate pressing threat here domestically, and then try to make sure that China doesn’t get us killed either, to put it sort of bluntly and simply,” Kokotajlo said.
The tech giants reportedly agreed to testify at Monday’s hearing after lawmakers threatened to subpoena them to force them to attend.
Menin claimed Google, OpenAI and Anthropic changed their tune and agreed to show up only after the council sent letters threatening to use its subpoena power.
NYC policymakers and tech execs are debating a package of bills aimed at regulating AI companies in the Big Apple, including by mandating “kill switches” and offering unprecedented cash rewards for whistleblowers.
Though the industry is typically subject to state and federal regulations, New York City is trying to get more involved in the safeguarding of the new technology as it rapidly expands in the city.
It comes after New York Gov. Kathy Hochul announced last month that the state will start requiring AI companies to register in a government portal starting in November, and forcing them to report critical safety incidents within 72 hours.
The City Council’s proposals include a mandate that companies cannot “market, offer for sale, or deploy” AI systems in the five boroughs without vetting the tech with outside experts.
It is also proposing a law that would require so-called “kill switches” for any AI tool being used in the city, which the council defines as a “human override that can shut down the system.”
Failure to comply with either of the laws would result in a $25,000 fine.
Another potential bill would create a “first-in-the-nation” cash reward system for whistleblowers, funded by the fines collected from bad actors.
Officials from NYC Mayor Zohran Mamdani’s administration – including Samuel Levine, commissioner of the Department of Consumer and Worker Protection, and Sarah Milstein from the tech office – were also set to testify at the hearing.
