A Tale of Three Headlines

Three headlines arrived over the weekend and accidentally assembled an argument.

• Dario Amodei, chief executive of Anthropic, says the artificial-intelligence frontier should slow down.
• Donald Trump says the United States cannot afford to slow down because China might win.
• King Charles is gathering a collection of the people responsible for this situation at a large house in Scotland.
  • We have a safety problem.
  • We have a strategic problem.
  • We have refreshments.

This is going better than expected.

Unfortunately for anyone hoping Modal Path Ethics could resolve this whole affair by identifying which one of these people is being stupid, all three of them have found something real.

Amodei has identified a capability race that may be outrunning the instruments meant to keep it safe. He is asking companies to give independent evaluators extraordinary access and eventually coordinate across firms and governments.

Trump has identified the reason that such restraint is difficult. If artificial intelligence carries decisive economic, military, scientific, or strategic advantage, one participant cannot safely assume that everyone else will slow down at the same time.

Charles has identified what has to exist between them.

A room.

That will not be enough.

It is still more important than it looks.

Dario: The Alarm.

Dario Amodei is not asking humanity to abandon artificial intelligence.

His new essay begins with the opposite case. He expects artificial intelligence to produce enormous benefits in medicine, economic growth, abundance, and human capability. The problem is speed.

His argument is that capability improvement is now moving quickly enough that safety research, operational practice, interpretability, evaluations, and institutional oversight need time to catch up.

So he proposes three steps.

First, frontier artificial-intelligence companies should admit embedded outside evaluators with something approaching employee-level access.

Anthropic says it will do this itself. These evaluators would be able to inspect training and deployment practices, investigate incidents, assess systems while they are being developed, and publish important findings without Anthropic retaining ordinary editorial control over unfavorable conclusions.

That is serious. The company is proposing an instrument capable of disagreeing with the company from inside the company. Good.

Second, the frontier laboratories operating in democratic countries should coordinate on common safety standards and on limits to unchecked capability growth.

Amodei acknowledges that some forms of cooperation would require government involvement because competitors ordinarily cannot just wander into a room and jointly decide how quickly their industry will advance.

Third comes international coordination.

And this is where the alarm discovers politics.

Amodei explicitly recognizes that the United States cannot impose severe restraint on itself while assuming Chinese projects will do the same. If one side paces and the other defects, the capability gap itself may become a strategic danger. His proposed international agreements therefore depend on verification, and he argues that agreements which cannot be verified should remain narrow enough that cheating does not produce an existential military advantage.

There it is.

The problem has escaped the laboratory.

• A company can improve its safety culture.
  • It can hire evaluators.
  • It can publish standards.
  • It can decide that another three months of alignment work is worth more than another three months of capability acceleration.
    • It cannot make that judgment safe for itself by sincerity alone.

The other laboratory still exists.

The other country still exists.

The other military still exists.

The other training cluster is still drawing power somewhere beyond the wall.

Voluntary virtue has encountered another actor.

Donald: The Objection.

Donald Trump has supplied the objection with unusual efficiency.

“Whoever wins with AI wins.”

Trump used this line while dismissing much of the current concern about artificial intelligence as exaggerated and warning that excessive restraint could surrender the American lead to China. He allowed that some guardrails may be appropriate while treating American technological leadership as the central strategic requirement.

There is a very easy article available right here.

President Reckless Ignores Experts.

We will not be using it. Trump's sentence contains the problem Amodei's own proposal is already trying to solve.

Imagine that the people running every American frontier laboratory become responsible overnight.

They slow down.

They test carefully.

They grant access to evaluators.

They refuse deployments that outrun their confidence.

Now suppose they believe a Chinese project will use the interval to cross a strategically decisive threshold first.

How long does the slowdown survive?

Turn the countries around.

Same problem.

Make it Anthropic and OpenAI.

Same problem.

Make it two militaries evaluating autonomous cyber systems.

Same problem.

The danger does not require anybody to love reckless development.

Each participant can sincerely prefer a safer pace. Each can also believe that slowing alone transfers the initiative to somebody less restrained.

Then acceleration becomes defensive.

The other side receives that acceleration. Its officials now have evidence that waiting may mean losing. So they accelerate too.

Their acceleration returns as fresh evidence that the original fear was justified.

We have been here before.

In Applied Case: Able Archer and the Dark Forest at Home, Modal Path Ethics called this a Dark Forest: a strategic field in which uncertainty, asymmetric danger, and fear of the first move can make concealment, consolidation, and early movement locally rational while the resulting postures make the common field more dangerous. The lesson was that the other side receives your posture rather than your private intentions.

Artificial intelligence fits inside that machine almost too cleanly.

The dangerous sentence is therefore not Trump's concern about unilateral restraint.

The dangerous sentence is the first one.

"Whoever wins with AI wins."

Wins what?

• A market?
• A weapons advantage?
• Scientific discovery?
• Economic productivity?
• Cyber superiority?
• The capacity to build the next system faster?
• Political control over the institutions dependent on it?
• A permanent lead?

The sentence takes a historically produced strategic game and promotes its victory condition into reality.

If the game says that there must be one decisive winner, then every safety proposal is immediately evaluated according to whether it helps somebody else become that winner.

The safety problem has now been subordinated to the scoreboard.

Trump has correctly described the trap.

Then, he has mistaken the trap for the objective function.

The King: Has a House.

Enter Charles III.

Reports say King Charles is preparing to gather roughly thirty figures from artificial intelligence, government, and adjacent institutions at Dumfries House in Scotland. Expected participants include major figures associated with Nvidia and Google DeepMind, with reporting around the broader event also describing participation from major frontier companies. A royal official described Charles's intended role as to “convene, engage, listen and encourage debate.”

This is extremely funny.

The hereditary monarch has somehow entered the artificial-intelligence sovereignty crisis as the guy who does not get to order anybody around.

The Crown has prestige. It has a house.

It can get people to answer invitations that would look much less interesting if they came from an assistant deputy minister for emerging technologies.

What Charles does not possess in this field is more revealing.

• He cannot tell Anthropic how quickly to train Claude.
• He cannot compel OpenAI to accept an evaluator.
• He cannot order Nvidia to allocate chips according to his preferred theory of frontier safety.
• He cannot bind China.
• He cannot make the United States honor an agreement.
• He cannot transform whatever consensus emerges from Dumfries House into a rule by announcing that the meeting went well.

In the constitutional sense relevant here, the King may be the least sovereign person in the room.

That is exactly why his instrument is interesting.

• Charles can convene.
  • Separate actors can enter without submitting their institutions to his command.
  • Competitors can hear one another's concerns.
  • Governments can hear what companies say they can verify.
  • Technical people can learn which political promises cannot currently be measured.
    • Everybody can discover whether the other side's position is actually the worst version they had been preparing against.

The room can create common knowledge before it creates common law.

Four years after Able Archer, the United States and Soviet Union built Nuclear Risk Reduction Centers. Neither country surrendered its military. Neither exposed every secret. Neither trusted the other completely.

They established a protected channel through which a defined class of conduct could acquire another interpretation before the worst interpretation had to govern.

Modal Path Ethics called that bounded legibility.

• Enough information for correction.
• Enough independent capacity to survive deception.

Dumfries House is nowhere near that. It is still pointing in the right direction.

The King has successfully invented the meeting.

Modal Path Ethics regrets to report that we are going to need institutions.

The Room Has to Become an Instrument.

A meeting cannot solve a race whose participants believe defection may determine the balance of global power.

A statement of shared principles cannot verify a training run.

A promise cannot reveal a secret model.

Good intentions cannot establish whether a capability threshold has been crossed.

And a company saying that it complied with its own safety requirements gives us exactly the evidentiary architecture the safety requirement was supposed to improve.

This is where Amodei's proposal becomes substantially more interesting than a generic slowdown demand.

He has already supplied the beginnings of a verification layer.

• Give outside evaluators durable access.
• Let them examine the systems rather than receiving a polished report after the fact.
• Give them authority to say when access was inadequate.
• Let common rules attach obligations to observable capabilities.
  • Then move the problem outward.

A viable frontier arrangement would need enough shared evidence that restraint stops looking like unilateral blindness.

One laboratory needs reason to believe another laboratory is living under comparable constraints.

One state needs reason to believe another state cannot secretly purchase a decisive advantage at trivial cost.

The obligation should attach to the dangerous capability rather than to the public relations department currently housing it.

Defection has to remain detectable enough that cooperation does not depend on everybody becoming nice.

That last point is critical.

• Trust is valuable.
• A constitution begins where trust stops being enough.

The point of the instrument is to preserve cooperation among actors who retain full capacity to disappoint one another.

Do Not Crown the Referee.

Now we can solve the whole thing very quickly.

Create one global artificial-intelligence authority.

Give it access to every frontier laboratory.

Give it every major training run.

Give it the chip supply.

Give it the model evaluations.

Let it determine which systems are safe.

Let it inspect governments.

Let it stop deployments.

Let it investigate itself.

Let it decide when a new capability has become dangerous enough to expand its jurisdiction.

Perfect.

We have just prevented the artificial-intelligence sovereign by building one manually.

This is the part of the problem a slowdown debate can easily miss.

Danger creates a powerful argument for coordination.

Successful coordination creates a powerful argument for giving the coordinator more information and more authority.

Every failure outside the center becomes evidence that the center needs greater reach. Every success inside the center becomes evidence that nobody should interfere with it.

Artificial-intelligence governance can defeat one catastrophe and walk directly into another.

Modal Path Ethics has already encountered this in OpenAI Discovers the Constitutional Problem.

• Build no effective center and some dangers may outrun every institution capable of stopping them.
• Build the center too well and the institution entrusted with stopping danger may become extraordinarily difficult to stop.

A field can have many centers and still have no outside.

That is why the answer cannot be one good regulator any more than it can be one good company, one good president, one good model, or, regrettably, one good king.

The functions have to separate.

• Laboratories can build.
• Independent evaluators can inspect.
• Standards bodies can define testable interfaces.
• Governments can enforce bounded obligations inside their jurisdictions.
• International arrangements can make specified conduct mutually legible.
• Emergency authorities can stop a defined transition when the evidence crosses a defined threshold.
• Courts, rival evaluators, legislatures, researchers, workers, and affected publics need routes capable of discovering that the safety machinery itself is wrong.

No one participant should control the evidence, the rule, its execution, the appeal, the duration of exceptional authority, and the conditions under which another institution may replace it.

That is the difference between coordination and sovereignty.

Winning: The Wrong Shape.

Artificial-intelligence politics keeps trying to hand us a choice between two frightening futures.

1. Race recklessly and something powerful escapes correction.
2. Restrain yourself and somebody else wins.

That is a real dilemma inside the current game.

It is not a reason to preserve the game.

Four Futures already rejected the idea that technological survival can be reduced to whether the right system ends up on top.

• A powerful system can destroy the field.
• It can save the field and own the conditions under which the field continues.
• Existing governments and firms can absorb its capabilities and become harder to correct.
• A visibly plural society can survive while every consequential disagreement eventually passes through one common operational interior.

There is no safe square labeled OUR SIDE WON AI.

Winning may matter. A dangerous actor obtaining a decisive strategic capability first can close enormous futures. A democracy does not have to ignore that because cooperation sounds nicer.

Anthropic does not have to pretend industrial competition disappeared because the safety meeting went very well.

China does not have to trust an American verification proposal whose actual function is preserving unilateral American control.

Nobody gets peace by being the only participant required to believe in it.

The constitutional task is harder.

Change what each actor has reason to do.

• Build the inspection route.
• Protect enough secrecy that verification does not become surrender.
• Expose enough conduct that restraint can acquire evidence.
• Make narrow agreements before demanding impossible ones.
• Keep consequences for defection.
• Keep the referee corrigible.
• Preserve the ability to move quickly when the dangerous condition really arrives.
  • And make sure the institution created to prevent one actor from winning everything cannot win everything itself.

The Ruling.

• Dario Amodei says slow down.
• Donald Trump says another country may not.
• King Charles has invited the people involved to Scotland.

All three survive this audit.

• The company cannot solve a geopolitical race through private virtue.
• The state cannot make the race safe by winning it faster.
• The convenor cannot transform conversation into enforcement by putting everyone around the correct table.

Each carries part of the instrument.

The rest still has to be built.

Artificial intelligence does not need a good king. It does not need the right company or country to win.

It needs a field in which restraint does not mean surrender, verification does not require blind trust, protection does not require permanent command, and nobody has to own the future in order to remain safe inside it.

• Dario has the alarm.
• Donald has the objection.
• Charles has the room.
  • Now build the constitution.