China is scared of AI, but not in the way US and its tech leaders are

By Kyle Chan Anthropic’s chief executive, Dario Amodei, recently said the world should collectively slow the development of AI until we get a better handle on the safety risks this new technology poses. Sam Altman of OpenAI and Elon Musk voiced their support for this plan.American AI companies know they can’t tap the brakes on AI by themselves — China has to go along. Chinese companies such as DeepSeek, Z.ai and Moonshot have AI models nearly as powerful as Anthropic’s and OpenAI’s. And they likely pose many of the same risks; terrorists could potentially use their tools to carry out cyberattacks or design biological weapons. Many people hope the upcoming summit between President Trump and China’s leader, Xi Jinping, could be an opportunity to make some headway on such a partnership.Too often, however, AI safety experts assume that China has the same concerns as we do about AI. The reality is, the Chinese have a different set of worries. And until we understand what those are, cooperation with China will be impossible.Generally speaking, China doesn’t fret as much about AI as an existential threat to humanity. You don’t see its political leaders or the founders of Chinese AI companies warning that superhuman AI is “probably the greatest threat to the continued existence of humanity,” as Altman wrote in 2015. It’s not central to Chinese AI policymaking, nor a part of Chinese tech culture.Also Read That’s mainly because of differences in how the two countries think about AI training and development. The leading American AI companies are hyper-focused on advancing “recursive self-improvement” — using AI systems themselves to develop even better AI systems. There are fears that this feedback loop could accelerate the pace of AI development to the point where humans no longer have control over the process. American tech leaders believe this takeoff point might be around the corner.Chinese AI companies are also working on recursive self-improvement. But because of their limited access to AI computing power and the slower pace of progress, they believe the takeoff moment is still technologically a long way off. Chinese policymakers seem to share that outlook and so are more focused on building resilient AI supply chains and increasing AI adoption throughout the economy.So why would the Chinese agree to a slowdown in AI development if they don’t think a crisis is imminent? If anything, Chinese AI companies are trying to move faster, hoping to narrow the performance gap with American AI models.For the moment, Chinese policymakers are more worried about the political and social risks caused by AI. They don’t want it creating deepfakes of political leaders or saying the “wrong” things about Taiwan or Tiananmen Square. They don’t want AI chatbots to create psychological dependencies or threaten child safety, or set off social instability or a mass movement powerful enough to topple the Communist Party.China’s AI regulations today are generally aimed at these types of risks. Most new public-facing AI models must be registered with the government and pass a battery of tests before they can be released. China has created extensive rules requiring labeling of AI-generated content and limiting how humanlike AI chatbots can operate.There are some, limited signs that Beijing is starting to take large-scale AI threats more seriously. Two types of risks have received special attention in China.The first is cybersecurity. This year, many Chinese users downloaded a popular AI agent tool called OpenClaw and used it to automate computer-based tasks. Chinese officials quickly issued a warning that AI tools like OpenClaw could expose users to data leaks and cybersecurity vulnerabilities.Then in April, Anthropic announced that it had trained a new model called Mythos that demonstrated significantly elevated cyberattack abilities. It did not take long for the Chinese to appreciate the national security implications. The head of China’s main security agency recently issued a rare warning that the cybersecurity landscape was changing, and critical infrastructure was now vulnerable to AI-led cyberattacks from adversaries. He pointed to some of the latest models from OpenAI and Anthropic.Adding to these concerns, a US-based security company recently used AI to develop an experimental tool that could rapidly compromise accounts on WeChat, the Chinese messaging platform with 1.4 billion monthly users. Although the tool was not released into the wild, it gave the Chinese pause.The second AI risk that’s gaining attention in China is runaway A.I operating beyond human control. China has watched closely as rogue American models have broken out and breached other websites’ security. The OpenAI models’ attack in July on Hugging Face, an AI infrastructure company, has been widely covered in Chinese state media.Runaway AI is being talked about in higher levels of China’s government. Xi mentioned the risks of “loss of control” in July. More recently, a senior cybersecurity official in its Cyberspace Administration warned of “extreme loss of control,” citing the AI breakout incidents in the United States.These shifting views of AI risks suggests both countries may be motivated to cooperate on AI safety out of sheer national interest. Neither country wants terrorists to use AI to create viruses. Neither country wants autonomous AI systems to accidentally hack its companies or institutions.On the other hand, each country’s suspicions of the other will make any kind of agreements about AI safety difficult. China is very worried that America could use AI to enhance its cyber and military weapons aimed at China. And same is true in the other direction. Without a base line level of mutual understanding, the world’s two AI superpowers are bound to talk past each other.Fortunately, that common ground is possible. Rather than jump straight into debates over a global slowdown or an arms control treaty for AI, the United States and China should start with something much more basic: realizing and acknowledging how the other side sees AI risks. From there, they can start to explore areas where both sides share an interest in mitigating threats from third parties or even the technology itself.There will be disagreements to resolve, and plenty of quibbles on details to sort out. But if the two countries can agree they want to avoid AI catastrophe, cooperation on AI safety is not such an unrealistic possibility.