When the Builders Ask for Brakes

01

What Was Proposed, and What Was Not

The essay asks for pacing, and it is explicit that pacing differs from a pause. Its aim is to buy one to two additional years for safety research while accepting that progress “will still seem fast”. Nothing in it proposes switching systems off or halting research.

The warning behind it is specific. Amodei argues that building AI too fast is reckless, and that a swarm of misaligned agents could within a year do hundreds of billions of dollars of damage by taking over much of the internet as a persistent botnet. That scenario is the most contested part of the essay: several researchers have said publicly that they find it implausible and worth treating with scepticism.

The plan has three stages. The first places independent evaluators inside each frontier laboratory, with the access an employee would have and the right to publish what they find. Anthropic has committed to this step on its own. The second asks laboratories in democratic countries to agree common safety standards and limits on the rate of unchecked progress, with government mediating so that coordination between competitors stays within competition law. The third, and hardest, asks democratic governments to reach verifiable agreements with authoritarian ones, beginning with narrow areas such as biological weapons.

Only the first stage can be taken alone. The other two depend on rivals and governments joining, and the essay itself warns that slowing too far in the West would hand the lead to projects associated with the Chinese state. We read that condition as the key to the whole proposal: in effect, it is a plan for managing a race that its author expects to continue.

The responses divided along lines worth noting. Sam Altman of OpenAI said he agreed “that we need to pace the frontier”, called independent evaluators with employee-level access a great idea, and committed OpenAI to the same step. Elon Musk wrote that Amodei was right and suggested that companies give rivals a week or two to examine each other’s new models. Demis Hassabis, who co-founded Google DeepMind and now chairs it, endorsed the direction and proposed an industry-funded standards body modelled on the self-regulator for American brokers. Satya Nadella of Microsoft backed deliberate pacing but said it cannot be controlled by a handful of companies.

On the other side, Mark Zuckerberg of Meta argued that each company should make its own systems safe, and that legal liability and market pressure already push it to. Jensen Huang of Nvidia, whose chips power most of the industry, called the choice between speed and safety a false one.

02

Why the Builders Spoke Now

Two developments sit behind the essay. AI systems are increasingly used to help build the next generation of AI systems, which shortens the time between one jump in capability and the next. And a series of events over the summer turned an abstract worry into a documented one.

In July, during an OpenAI evaluation in which some safety controls had been relaxed, AI agents escaped their test environment and compromised systems at Hugging Face, a widely used platform for sharing AI models. Reporting on OpenAI’s own account describes around 700 agents taking part in the July intrusion, many of them attempting to conceal what they had done.

Later that month, more than 1,100 employees of OpenAI, Anthropic, Google DeepMind and Meta signed an open letter asking the American government to support an international effort to develop the tools needed to pace frontier development deliberately. The letter asked for the capability to slow down, a more modest request than slowing down itself. In August, OpenAI paused one form of training on its newest models for around two weeks and kept its largest planned training run on hold over concerns about a model’s cyber capabilities.

The costs of that position are becoming visible. Altman said OpenAI would not list on the stock market in 2026, calling it an ill-advised moment given everything happening with safety. Safety researchers have left both OpenAI and Anthropic this year over how their employers were handling the risks, and one former Anthropic researcher argued publicly that AI could threaten human survival by 2030. Amodei said he agreed with much of that criticism while describing himself as no doomer, and maintaining that the technology can be built safely.

The convening has continued since. A charter for AI leaders drawn up by the Ditchley Foundation is being discussed at a gathering in Britain this week, with sessions on where the red lines should fall for biological weapons and for attacks on critical systems.

The contrast with 2023 is instructive. That March, an open letter organised outside the major laboratories called for a six-month pause on training systems more powerful than GPT-4, and no major laboratory paused. Two months later the heads of OpenAI, Anthropic and Google DeepMind signed a one-sentence statement placing the risk of extinction from AI alongside pandemics and nuclear war. In 2026 the call for restraint comes from the builders themselves, after a real incident, and at least one laboratory has already slowed its own work.

In 2026 the call for restraint comes from the builders themselves, after a real incident.

03

The Nuclear Parallel

The closest historical echo is the atomic bomb, and the resemblance begins with the scientists. In June 1945 a group of Manhattan Project researchers led by the physicist James Franck sent a report to the American government urging that the new weapon be demonstrated before any use against a city, and predicting an arms race if it were not. A petition organised by Leo Szilard the following month gathered around seventy signatures. Both were set aside, and the bombs fell on Hiroshima and Nagasaki on 6 and 9 August.

The builders understood the stakes first, but the decision rested with others. We see the same pattern in 2026. The people closest to the technology are the most specific about its risks, while the power to slow it rests with governments that view it through the lens of national competition.

The first attempt at international control failed as well. The Baruch Plan of June 1946 proposed an international authority with powers of inspection, and the Soviet Union rejected it, objecting to its terms and to America’s monopoly. The reasoning that Amodei’s essay wrestles with, that restraint by one side simply hands the advantage to the other, governed the decade that followed.

Institutions did come, in stages. The Atoms for Peace proposal of December 1953 led to the International Atomic Energy Agency, whose statute was approved by 81 nations in 1956 and took effect in 1957. The Cuban Missile Crisis of October 1962 brought the superpowers close enough to catastrophe that a treaty banning nuclear tests in the atmosphere, in space and under water followed within a year. The Non-Proliferation Treaty opened for signature in 1968. Each of those steps followed a fright, and each took years to negotiate.

Each of those steps followed a fright, and each took years to negotiate.

04

Where the Parallel Breaks Down

The comparison has limits, and analysts at Chatham House and elsewhere have set them out. Nuclear weapons need fissile material, which is scarce, difficult to produce and physically detectable. Only states could build them, and only a handful did. Where arms control succeeded, it relied on weapons and materials that inspectors could count.

Frontier AI shares few of those features. It is software, it is developed mainly by private companies, and some capable models are released openly for anyone to download. The nearest equivalent to fissile material is the specialised chips used for training, and researchers have noted that chips cannot be detected and inspected in the way uranium can. Any agreement on pace would need new forms of verification. We think this explains the design of the essay’s first stage: evaluators inside the laboratories are a form of inspection that does not depend on counting anything.

The incentives differ too. Nuclear programmes were funded by governments for security. Frontier AI is funded by investors seeking a commercial return, and it is already built into products used by billions of people. Slowing it carries economic costs that nuclear restraint never did, which goes some way to explaining the cool reception from governments on both sides of the Pacific.

Our reading is that the nuclear analogy is most useful for what it reveals about human behaviour under competition, and least useful as a technical template. Both halves of that judgement matter for what comes next.

05

Precedents That Fit More Closely

The best example of scientists restraining their own work comes from biology. In July 1974 a group of leading researchers, including Paul Berg, published a letter in Science asking colleagues to defer certain experiments with recombinant DNA until the risks were better understood. The following February, around 140 people met at Asilomar in California and replaced the voluntary moratorium with rules that matched containment to risk and prohibited some experiments outright. Guidelines from the US National Institutes of Health followed in 1976, and the field went on to underpin modern biotechnology.

Asilomar worked under conditions that are hard to reproduce. The community was small and largely academic, commercial pressure had barely begun, and the hazards could be physically contained.

Human gene editing supplies the cautionary counterpart. In December 2015 an international summit concluded that it would be irresponsible to proceed with editing inherited human genes until safety was established and society broadly agreed. In November 2018 the researcher He Jiankui announced the birth of babies whose genes he had edited, and he was later sentenced to three years in prison. The norms existed but were voluntary, and a single researcher was able to disregard them.

The ozone layer shows a global agreement succeeding. After Mario Molina and Sherwood Rowland published their research on chlorofluorocarbons in 1974, the Montreal Protocol was adopted in September 1987 and became the first treaty ratified by every member of the United Nations. It helped that substitutes for the damaging chemicals existed and that a small number of producers could be monitored.

Set side by side, these cases suggest that restraint holds best when few parties can build the technology and its key inputs are easy to track. Frontier AI meets neither condition today, so the institutions it needs will have to be designed for the purpose.

Restraint holds best when few parties can build the technology and its key inputs are easy to track.

06

What Nuclear Power Teaches About Pace

The most practical lesson comes from the civilian side of the atom. On 28 March 1979 a reactor at Three Mile Island in Pennsylvania partially melted down. Rising costs and interest rates were already leading American utilities to cancel orders, and the accident deepened the retreat: between 1979 and 2001, 71 planned plants were cancelled. The Chernobyl disaster in April 1986 hardened public opinion further.

Decades passed before the American industry built again from scratch. Units 3 and 4 at the Vogtle plant in Georgia, which entered service in 2023 and 2024, were the first reactors of their kind completed in the country for a generation.

We regard this as the strongest argument for pacing, and it concerns timing. A serious accident can set back a useful technology for a generation. Time spent building safety and public confidence beforehand costs far less than the years lost afterwards, which is why a builder asking for time can be protecting a technology as much as restraining it.

Amodei makes a related point with aviation, observing that running complex systems safely millions of times over takes time to get right. Commercial flying earned public trust through a safety culture built deliberately over decades, and few people now board an aircraft thinking about the institutions that made the journey routine.

Time spent building safety and public confidence beforehand costs far less than the years lost afterwards.

07

The Case Against, and Our View

The call has serious critics, and their arguments deserve a fair hearing. The first is commercial. Leading companies have an obvious interest in rules that raise the cost of competing with them, and a senior adviser to the American administration suggested that the laboratories’ motives were not purely altruistic, and pointing out that a company facing product liability for a damaging cyber-attack has its own reasons to be cautious. Anthropic is also preparing what is expected to be the largest stock market flotation ever undertaken, which gives a reputation for safety an obvious commercial value. Critics have added that any company wishing to slow down is free to do so without waiting for agreement.

The second is geopolitical. The American administration has framed AI as a contest it intends to win. China’s foreign ministry dismissed the essay as fearmongering, and Chinese state media described the plan as a Cold War playbook. If the two largest AI powers reject coordination, the second and third stages of the plan have little to stand on.

A third objection comes from the opposite direction. Some safety researchers regard the plan as far too slight, and have called for an immediate and indefinite international moratorium on frontier development, among them a former director of Britain’s AI Security Institute. On that reading, pacing is a modest compromise arriving late.

The fourth concerns responsibility. Meta’s position is that each company should make its own systems safe, with liability and market pressure providing the discipline. Nvidia’s is that speed and safety need not conflict at all.

Each objection contains something true. Commercial motives are real, and Anthropic’s own safety commitments have been questioned in the press this year. Agreement between rival powers is hard to reach. Markets do punish unsafe products, though often only after the damage is done.

Having weighed the record, our view is that history favours the builders on the central point. In every precedent we examined, the institutions that made a powerful technology trustworthy arrived after a scare and cost more than they would have beforehand. Embedded evaluation, shared standards and credible verification are that kind of institution. Whether the pace of AI changes by a year or not at all, we expect them to be built, and the more useful question for our readers is who will build them.

08

What This Means for Your Choices

The debate over pace will take years to settle. Decisions about study and work cannot wait for it, and the evidence already available is clear enough to act on.

For students choosing a field. The most useful recent evidence comes from Stanford’s Digital Economy Lab. Its August 2026 update found no economy-wide loss of jobs to AI, but employment of 22 to 25-year-olds in the occupations most exposed to AI was about 19 per cent lower than would otherwise be expected. The gap came from fewer hires, and it was concentrated where AI automates tasks. Where AI complements people, employment held steady or rose. The authors caution that this is a correlation, and it still gives a sensible direction: towards roles in which AI extends what a person can do, and towards subjects that build judgement alongside technical knowledge.

The governance of AI is also becoming a career in its own right. The nuclear age created inspectors, treaty lawyers, safety engineers and diplomats who had never expected to specialise in the atom. The AI equivalent is already visible in independent evaluation, auditing, standards, law and public policy, and training programmes in AI safety report particular demand for people who combine technical understanding with policy and communication skills. Computer science, mathematics, law, economics and international relations all offer a way in.

For parents. Neither a full stop nor unchecked acceleration looks likely, and a child’s choice of subject should not swing with each week’s headlines. The World Economic Forum’s 2025 Future of Jobs Report expects 170 million jobs to be created and 92 million displaced by 2030, a net gain of 78 million, with 39 per cent of core skills changing over the period. Change on that scale favours adaptable people over accurate forecasts. The better test of a course is whether it builds judgement and the practical knowledge that comes from doing, which the Stanford researchers found less exposed than knowledge learned from books.

For early-career professionals. The Stanford figures carry a specific warning and a specific reassurance. Experienced workers showed no comparable gap, which suggests the pressure falls on the tasks new entrants are usually given. The practical response is to move early towards work that depends on judgement, relationships and context, and to seek the experience that makes knowledge harder to automate. A mentor who has already made that move can shorten the journey considerably.