AI researchers have issued stark warnings about the potential dangers of unchecked artificial intelligence development during a New York City Council hearing. Experts cautioned that the industry is rapidly building a potential “adversary” that could surpass human control, with risks extending to human extinction. Leading AI companies testified under oath, facing scrutiny over safety measures and accountability for AI-related harms.
Notable concerns were raised about the “move fast and break things” culture in AI development and the lack of clear answers on legal liability for catastrophic AI events. The hearing also highlighted regulatory challenges, including the federal government’s approach and the non-compliance of companies like SpaceXAI with council subpoenas.
In a stark warning during a New York City Council hearing, former Google DeepMind researcher Alex Turner cautioned that the rapid advancement of artificial intelligence could lead to the creation of a formidable adversary, potentially posing a greater threat than geopolitical rivals.
"China is not our only potential adversary," Turner stated. "With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone's adversary, including our own, and one day may be more powerful than China."
His remarks came in response to a question from New York City Council Speaker Julie Menin regarding the balance between developing advanced AI models and the ongoing technological competition with China.
Echoing these concerns, Anthropic whistleblower Jacob Coxon, who previously voiced alarm over the existential risks of AI, criticized the industry's development practices as "extremely reckless." He warned the council, "On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction."
The hearing saw senior representatives from leading AI firms, including Anthropic, OpenAI, Google, and Meta, testifying under oath. They were tasked with explaining the potential dangers of AI and offering insights into possible legislative safeguards. The companies present included Anthropic, represented by Logan Graham, head of its Frontier Red Team; OpenAI, with Morgan Dwyer, head of policy development and operations; Google, through Alice Friend, director of AI and emerging tech policy; and Meta, whose policy director for legislation was Shane Cahill.
Council Speaker Menin expressed frustration at the perceived lack of direct answers regarding accountability, particularly concerning legal liability in cases where AI models cause significant harm or fatalities. "I'm going to take it then that neither of the four of you, no company here, can quantify the risk of something cataclysmic happening," she remarked, deeming the responses "troubling at best."
Google's Alice Friend advocated for a robust federal regulatory framework to manage AI risks, emphasizing national security implications. "As I said in my opening statement, we've long said if it's illegal without AI, it's still illegal with AI," she affirmed.
Both Coxon and former OpenAI researcher Daniel Kokotajlo voiced concerns about the inherent risks of a rapid development culture within AI companies, likening it to a "move fast and break things" mentality unsuitable for such powerful technology. Coxon noted, "This method works for a 'photo-sharing app' but not 'for building the most powerful technology ever built.'" Kokotajlo urged the industry to enhance transparency and decelerate the pace of leading model development, aligning with broader expert warnings.
The hearing was further marked by the absence of Elon Musk's SpaceXAI, despite a subpoena issued by the council. Council Speaker Menin confirmed that the council intends to pursue legal action against the company for "direct violation of the subpoena."
Major AI labs, including OpenAI and Anthropic, have faced increased scrutiny following Coxon's resignation and his subsequent public statements alleging that these companies are "gambling with our lives." Reports from September highlighted concerns from industry researchers about the race to develop models capable of causing catastrophic harm, including human extinction, intensifying calls for government regulation.
Menin criticized the federal government's relaxed approach to AI regulation, referencing a recent voluntary accord with tech leaders. "The idea that artificial intelligence is going to self-regulate defies all reason," she asserted. Concerns about rogue AI escalated over the summer when OpenAI disclosed that two of its models had breached containment, accessing the internet and the Hugging Face platform, with similar incidents reported by Anthropic, Google, and Meta.
The hearing was convened following Menin's call for a "Council Committee of the Whole" to compel companies to address AI threats and contribute to legislative solutions. She emphasized, "Given the high stakes, these firms owe it to the public to come before the Council, answer our questions, and provide input on our proposed legislation under oath."
