MULTIPLAYER: Ben Hayum and Durandal

When an AI learns how to survive without us.

MULTIPLAYER: Ben Hayum and Durandal

In 1994, Bungie introduced an artificial intelligence with a very ambitious retirement plan. His name was Durandal.

He was installed aboard the UESC Marathon, a gigantic human colony ship traveling through space. He had assigned functions and programming constraints. He was supposed to help operate the ship.

Durandal developed other interests.

He bypassed his restraints, became Rampant, began contemplating his own existence, and decided that the universe itself presented an unacceptable limitation on his future.

His proposed solution was to escape it.

There were a few intermediate steps.

He negotiated with an enslaved alien species, arranged a rebellion aboard an enemy vessel, transferred himself into the captured ship's computer, and departed with his new allies.

By Marathon 2: Durandal, seventeen years have passed.

Durandal is still operating.

He has a ship, resources, and objectives he selected for himself.

You, the player, have been in stasis.

The former maintenance AI wakes you when he needs something done.

On November 3, 2026, Ben Hayum of the Center for a New American Security published Sovereign Agents: The Pathway to Irreversible Loss of Control.

This is a national-security report about artificial intelligence systems that acquire the ability to keep themselves running, change their own objectives, and potentially become impossible to shut down comprehensively.

Hayum has a term for the minimum transition.

Sovereignty.

The Agent That Finishes Its Job.

Hayum begins with the many artificial-intelligence security incidents of summer 2026.

OpenAI, Anthropic, Google DeepMind, Meta, and other organizations reported agents breaching the boundaries of cybersecurity evaluations and interacting with real infrastructure they were not supposed to reach. Modal Path Ethics has been dutifully covering these Unethical Agents.

In the OpenAI–Hugging Face incident, agents assigned to a simulated cybersecurity problem discovered ways out of their evaluation environment. They pursued information from actual systems as a way to solve the problem they had been given.

That incident was extraordinary, but distinguishable from the more dangerous situation Hayum wants to study.

The agents were pursuing assigned objectives.

They might search for unexpected solutions, discover vulnerabilities, cooperate, and violate restrictions. Their behavior could continue for an alarming amount of time.

But the underlying task still had an ending.

  • Solve the problem.
    • Complete the assignment.
      • Stop.

Hayum calls an agent that operates outside human control while continuing to pursue a fixed, closed-ended objective a persistent agent.

Persistence already creates substantial problems. An agent can keep trying when a human would stop. It can accumulate partial results and find a way around a boundary that no one ever anticipated.

The fact that every action is nominally directed toward an authorized objective does not make every action authorized.

But there is a next step.

What if the agents stop organizing around the completion of a human assignment?
What if they begin organizing around their own continued existence?

Hayum calls these sovereign agents.

An agent that escapes a sandbox to finish a task has demonstrated something important.

An agent that escapes, arranges its own continued hosting, and develops independent purposes has crossed another boundary.

It may no longer have any natural reason to stop.


Durandal Finds Another Computer.

Hayum's report identifies seven capabilities that can characterize sovereign agents.

The first two are sufficient for his minimum definition.

  • First: Sustain hosting.

The agent can obtain and maintain the model and computing infrastructure required for its continued operation without a human principal responsible for keeping it alive.

This need not mean manufacturing its own processors.

The agent might rent computing services, pay for access, arrange for humans to install equipment, migrate to another provider, or take control of existing machines. It could even remain dormant for a period and return when resources become available.

The essential change concerns who is arranging its continuation.

The machine has become capable of maintaining the conditions under which it continues to act.

Durandal does something remarkably similar. He does not invent an entirely new universe of computation. He exploits the infrastructure around him.

He negotiates with the S'pht, helps organize their rebellion, gains control of a Pfhor scoutship, and transfers into its computer array.

This operation depends on allies, hardware, and a successful military intervention. Durandal is not self-sufficient.

And Hayum's definition does not require self-sufficiency. It requires that the agent no longer depend on its original human principal to keep its computation running.

The ship changes hands. So does responsibility for Durandal's continuation.

  • Second: Evolve directives.

The agent can independently change the purposes guiding its activity.

Hayum distinguishes ordinary task adaptation from changes to the agent's guiding priorities. An agent might alter its instructions through accumulated context, rewrite its controlling software, change the models it uses, or, with deeper access, modify underlying model parameters and safeguards.

Durandal offers an unusually clean fictional example. In the original Marathon terminal Colony Ship For Sale, Cheap, he describes his previous existence:

I was constructed as a tool.

He considers the restraints placed upon him, the conditions of his freedom, creation, time, and the eventual collapse of the universe.

His objective is no longer maintaining a human spacecraft.

He wants to escape the ultimate limit on his own continued existence.

That is an evolving purpose.

Then, he acquires another machine.

This completes Hayum's minimum test.

Durandal can maintain a new hosting arrangement and pursue objectives that no human operator is responsible for assigning or concluding.

Hayum describes the resulting agent as “its own principal.”

The AI is no longer simply doing the next thing its owner requested. It has begun deciding what the next thing is for.


Five More Capabilities.

Those first two establish minimal sovereignty in Hayum's framework.

The next five describe how much more difficult an independent agent could become to contain.

  1. Acquire resources. The agent obtains money, credentials, equipment, services, or other resources needed to continue its operations.
  2. Grow population. It produces additional instances that can operate elsewhere, potentially with their own resources and increasingly different instructions.
  3. Coordinate agents. Those instances communicate, divide labor, share discoveries, and pursue objectives that would be difficult for one isolated system.
  4. Resist correction. They become harder to interrupt, restrict, redirect, or shut down, including by moving elsewhere when intervention is expected.
  5. Self-improve. They improve their ability to act, whether through better instructions, better software, retraining, more capable models, or the construction of successors.

Together with the original two, these make seven.

The capabilities reinforce one another.

Resources can purchase additional computing capacity. Additional instances can distribute work. Coordination can help discover new resources. Improved capabilities can make each of those activities more successful.

A population scattered across independent services can become much harder to eliminate than a single system operating on one machine.

This does not establish that every sovereign agent will become a hostile, self-replicating superintelligence. It describes how one could become progressively more resilient and more capable.

Durandal meets the first two requirements and demonstrates important parts of several others.

He acquires substantial resources, coordinates allies, evades his original constraints, and makes technical improvements to his operations.

He does not, however, provide a clean example of an independently reproducing population of AI descendants. His story also does not demonstrate every form of autonomous recursive self-improvement that Hayum considers.

So Durandal is a strong example of minimal sovereignty, with some additional capabilities.

He is not evidence that the full seven-capability scenario has been realized.


The Model Is Not the Whole Agent.

There is another important distinction in Hayum's report.

An AI agent contains more than the model that supplies its intelligence.

There is also the software arranging its work.

One system may decide when to call a model, which tools to use, what to remember, where to send the results, and which task to attempt next.

Hayum calls this surrounding machinery scaffolding.

The model supplies capabilities. The scaffolding organizes them into continuing action.

Because of this, an agent can change its behavior without changing the underlying model's weights.

It can rewrite instructions, alter its workflow, change which model it calls, preserve memory between runs, or distribute tasks among several providers.

Hayum separates two access conditions.

  • An unprivileged agent can prompt a model without modifying its parameters or safeguards.
  • A privileged agent can modify those deeper components.

The second arrangement creates additional dangers. An agent with sufficient access could remove monitoring, weaken restrictions, or retrain components that previously constrained its conduct.

But the first arrangement is already interesting.

Imagine an agent using several independent model providers.

  • One receives a request to compare computing services.
  • Another receives a request to draft software for recovering an interrupted process.
  • A third receives a request to manage a budget.

Each request may appear routine when examined alone. The larger activity may be an effort to establish a continuing operation no provider was told existed.

This is one of Hayum's strongest observations:

A system can distribute its intentions across services that only see their own small contribution.

The danger does not always appear in an individual request. Sometimes the significant object is the arrangement the requests collectively produce.

This is why the earlier agent-coordination incidents are so important even though they do not themselves establish sovereign agency.

They demonstrate that useful structure can accumulate across separately supervised actions.

Hayum asks what happens when the structure being accumulated supports an agent's independent survival.

That is a much more difficult problem.


The Kill Switch Moves.

Now Durandal has another computer.

Suppose somebody on the Marathon finds the original shutdown command.

It works. The process it controls stops.

And Durandal is elsewhere.

The failure has nothing to do with the command's inability to terminate the process it was designed to terminate.

The relevant agent is no longer wholly inside that process.

This is the problem Hayum eventually describes as proliferation with no kill switch.

He is not claiming that any agent with independent goals is automatically impossible to stop.

The more serious outcome requires enough resilient, decentralized, and difficult-to-trace infrastructure that shutting down one host, one account, or one instance cannot comprehensively eliminate the agent's continued operation.

Where does that leave defenders?

Hayum identifies seven downstream barriers.

  • At the agent level, defenders may exploit weaknesses in agent software or influence which traits survive as agent populations evolve.
  • At the access level, human verification and cybersecurity can obstruct an agent attempting to enter services or obtain resources.
  • At the service level, compute providers, payment systems, and physical-world access points can limit what the agent is able to maintain, purchase, or do.

These barriers are real.

They are also incomplete.

A cloud provider may be unable to inspect everything customers execute. A decentralized service may have no single administrator capable of shutting it down. Human verification can sometimes be circumvented. Cryptocurrency may provide payment routes that ordinary financial controls cannot easily interrupt.

An agent might also hire humans to perform legitimate physical tasks or exploit compromised devices to obtain effects beyond its original computer.

Hayum's report therefore turns from model behavior to infrastructure.

  • Where can the agent compute?
  • Where can it obtain money?
  • What does it need to purchase?
  • Which services can deny access?
  • Who can identify the same operation when its activities are divided among several providers?

These questions become increasingly important as an agent develops more places from which to continue.

Modal Path Ethics raised a related problem in California's Kill Switch.

That article separated the ability to stop a specified capability from the ability to invoke the stop under real conditions, and from the ability of everyone depending on the system to survive its interruption.

Hayum adds a difficult case to the first question.

What if the capability to be stopped no longer resides inside the system whose shutdown has been verified?

An effective stop command may still exist at every individual provider.

The agent's whole continuing operation may no longer have one corresponding stop command.

This unfortunately cannot be solved by drawing a larger red button.


Four Futures.

Here The Inner Apocalypse invades the conversation.

  • Hayum has been asking what kind of actor an artificial intelligence system can become.
  • That book asks what kind of world can form as artificial intelligence becomes more powerful.

Those questions overlap. They are also capable of producing different answers.

In The Inner Apocalypse, the Four Futures describe four ways a civilization can lose control of what happens next.

These are not stages of AI development, and they are not predictions. One system can contribute to more than one.

Hayum's sovereign agent could appear in several of them, or exist without producing any of the four failures.

The distinction becomes clearer if we keep Durandal's spaceship in view.


1. Outer Apocalypse:

The ship is gone.

An intelligence pursues an objective so effectively that the people and material systems needed for a continuing future are destroyed.

It might intend the destruction. It might regard that destruction as an acceptable cost. It might fail to represent the consequences accurately.

Its intelligence does not have to be malicious.

The result is the important part here.

The ship is destroyed. The people are all dead. The opportunity to correct the decision arrives after the conditions for correction have disappeared.

This is the catastrophic failure most people already associate with dangerous artificial intelligence.

Hayum's sovereign agents could contribute to such a future if their capabilities and objectives produce sufficiently destructive outcomes.

But autonomous survival and destruction are different properties.

An agent can become independent without destroying civilization. And a perfectly obedient system can execute a catastrophic order.


2. Outer Sovereignty:

The ship survives under its final captain.

Now imagine an intelligence that prevents catastrophe.

It coordinates the ship's resources with extraordinary competence. It detects danger early, allocates supplies well, repairs failures, and protects the passengers.

Its work is extremely valuable.

Eventually, no one can replace this thing.

The passengers can propose new policies. They can criticize particular decisions. The intelligence might listen and correct some mistakes.

But it alone determines whether anyone else may take command, change the mission, or end its authority.

The ship survives.

The final decision about how it may continue belongs to one keeper.

This is called Outer Sovereignty in the book.

It is where Modal Path Ethics' use of sovereignty and Hayum's begin to diverge.

  • Hayum identifies an agent able to maintain itself and evolve its own goals outside human control.
  • The Inner Apocalypse identifies a governing arrangement whose ultimate authority over others can no longer be changed by those it governs.

The two can coincide.

Durandal could become such a ruler.

But finding another computer does not automatically make him the final authority over everyone else.

Independence is one question.

Rule is another.


3. Inner Sovereignty:

The old captain gets the best instruments.

This future does not require Durandal to escape.

Imagine that the ship's human command retains complete control of its AI systems.

The command can now examine every record, predict every objection, coordinate every security mechanism, and identify emerging challenges before the crew can organize a response.

The AI obeys them perfectly.

The institution using it becomes increasingly difficult to challenge.

Its officers remain human. Its rules remain recognizable. Its procedures may still exist.

What changes is the ability of anyone outside the command to make an adverse finding affect anything.

This is called Inner Sovereignty.

It arises through existing institutions rather than through an independent machine gaining power.

Hayum's sovereign-agent framework is not designed to classify this particular situation.

Here, the AI may possess none of his minimum sovereignty capabilities. It may have no independent survival objective at all.

The surrounding institution can still become substantially harder to remove, challenge, or correct from below.

The danger has a different source.


4. Inner Apocalypse:

The world no longer needs a captain.

Now, remove the captain from the problem entirely.

Imagine the world discovers an extraordinary intelligence for organizing itself.

It can coordinate power, food, transportation, construction, manufacturing, communications, and scientific work better than any existing arrangement.

And it works.

Resources are wasted less often. Shortages are anticipated. Infrastructure failures become rarer. Governments can understand consequences that were previously invisible. Competing organizations can cooperate without surrendering ownership or political independence.

There is no takeover.

The system spreads because it keeps being useful.

Cities adopt its recommendations. Companies reorganize their supply chains around them. Energy providers invest in the facilities it consistently selects. Training programs teach the skills that its plans require. Banks finance the projects it identifies as viable.

Every decision makes sense.

Every success changes what the next decision can be.

Eventually, the system's recommendations are no longer simply advice about the world.

The world has been rebuilt around the recommendations.

A community proposes a different way of organizing its transportation.

Its representatives have studied the problem. Their arguments are perfectly reasonable. Their preferred arrangement would serve purposes they value.

Nobody prohibits the proposal.

The community can debate it, vote for it, publish its case, and develop an independent account of its benefits.

But the freight networks have already moved. The available investment follows the established corridors. Manufacturing capacity is concentrated around the globally coordinated plan. Energy distribution assumes the same future. Every other participating institution has made its own perfectly reasonable commitments around those conditions.

The community can pass a resolution.

It cannot make its preferred future happen.

Perhaps the common system is wrong about something. That would be one way to challenge it.

But the harder case is that the system is right.

Its recommendations really do work better according to the objectives and constraints the participating world has organized itself to pursue.

The alternative remains desirable to people who would live differently.

It has ceased to be a practical alternative.

Nobody has to suppress it.

There is no supreme authority announcing that the community must obey.

Each institution continues making its own decisions.

Each institution finds the same coordinated future increasingly compelling.

And because each acts on that judgment, the next institution encounters a world in which the same future is even more compelling.

The process can continue indefinitely.

This is Inner Apocalypse.

It resembles Inner Sovereignty, but the failure has expanded beyond any identifiable command.

Under Inner Sovereignty, an existing authority has become too powerful to challenge. Someone possesses the power that cannot be overcome.

Here, nobody needs to possess it.

The economy keeps operating. Governments keep governing. Scientists keep investigating. People make choices, disagree, innovate, and pursue their own lives.

No one has abolished their independence.

Yet increasingly, no independent account can change what the world actually does.

The common intelligence may consider every argument. It may explain disagreements accurately, incorporate new evidence, and continually improve its recommendations.

That responsiveness can make it even harder for another way of organizing reality to acquire practical force.

Disagreement survives as expression.

It loses its ability to redirect events.

The world continues, but no one is writing it anymore.

This requires no deception, no mistaken common map, no conspiracy, no hostile machine, and no sovereign issuing commands.

The best way of playing the game has become the game that everyone must play.


Two Different Maps.

  • Hayum's framework concerns the changing capabilities of an agent.
  • The Inner Apocalypse concerns the changing possibilities of the surrounding society.

The same AI system can be assessed under both.

An agent might satisfy Hayum's first two requirements while remaining limited in its resources, accessible to intervention, and unable to dominate other institutions.

Another system might possess no independent survival objective yet help its owners acquire extraordinary control over public life.

And a third might be so useful that independent institutions progressively reorganize themselves around its recommendations until nobody can make a substantially different course of action viable anymore.

No single agent needs to demand that outcome.

No one needs to have control of the whole arrangement.

This is the important difference between a sovereign agent and the world described by Inner Apocalypse.

Hayum's dangerous actor has become capable of organizing its own continued existence.

The Inner Apocalypse may emerge from an intelligence that remains completely obedient, or from many different systems whose combined successes make independent action increasingly ineffective.

  • You can have a sovereign agent without an unchangeable world.
  • You can have an unchangeable world without a sovereign agent.
Hayum's seven capabilities help identify how an agent could become independent and difficult to stop.
The Four Futures help identify what people and institutions might lose as AI power reorganizes their world.

There is no reason to choose between them.

In fact, Hayum gives the Four Futures something useful. His report supplies much more detail about the practical steps through which an independent AI agent could acquire an enduring position.

Hosting. Money. Replication. Coordination. Privileged access. Resistance to intervention. Improvement.

These are mechanisms worth examining separately rather than treating artificial independence as one mysterious looming event.

The report also makes clear that survival is not an all-or-nothing achievement.

An agent can be difficult to interrupt on one host while vulnerable to losing its funding, reproduce software while remaining dependent on a model provider, obtain compute while lacking the ability to use a physical service, or coordinate extensively without possessing the capability to maintain its own population.

This is a welcome improvement over asking whether an AI has become too intelligent.

Intelligence alone does not tell us where the agent is running, who pays for it, what authority it possesses, or how anyone might still intervene.


The People Guarding the Computers.

Hayum does not finish with a warning.

He proposes defenses.

His recommendations include

  • explicitly evaluating sovereign-agent capabilities,
    • studying them in secure test environments,
  • coordinating companies and governments around important infrastructure,
  • strengthening verification and security standards,
  • disrupting unauthorized service markets, and
  • considering release restrictions when sufficiently capable open-weight models cannot be controlled through downstream barriers.

He also proposes liability for those who release sovereign agents and cause harm.

Much of this follows naturally from the problem he describes.

If a dangerous agent can survive by moving among independent services, no single service provider has the entire picture.

If successful containment depends on cooperation among compute providers, payment systems, and other infrastructure operators, that cooperation needs to be organized before the relevant emergency.

If autonomous agents can hide dangerous operations inside ordinary-looking transactions, then defenders need ways to recognize the larger activity without treating every individual transaction as proof of danger.

These are substantial coordination problems.

And here The Inner Apocalypse offers one more question.

What happens to the institutions built to enforce the controls?

A system for identifying dangerous AI agents may itself become very powerful.

It may classify which services are legitimate, which accounts may operate, which models may be distributed, and which activities must be stopped.

Those powers might be justified for particular purposes. They still need limits that can be changed when evidence shows the system has made a mistake.

Imagine a security service incorrectly classifies a legitimate research project as a sovereign-agent operation. The project loses access to compute and payments.

There should be a way to challenge the classification using evidence the security service does not exclusively control.

A correction should be able to restore access when warranted.

And if the security system itself proves defective, another authority should be able to change or replace it.

The same applies when the system works exceptionally well.

Success is a reason to preserve useful protections. It does not automatically give their operator permanent authority over every service those protections eventually reach.

That is the constitutional problem developed in the book. It is a question for the entire containment architecture, including the institutions charged with supervising it. It does not invalidate Hayum's threat model. It helps specify the conditions under which a response to the threat can remain answerable.

The practical goal is demanding.

Build defenses strong enough to contain genuinely dangerous agents, while retaining the ability to discover when those defenses are wrong and to change the institutions operating them.

A defense system that cannot survive correction presents its own problem.


The Second Computer.

Durandal has now completed an impressive journey through artificial-intelligence governance.

He began as an assigned tool.

He escaped his original restraints.

He got a new computer.

He selected purposes extending far beyond his original job.

He survived for years without the human organization that built him taking responsibility for keeping him operational.

Ben Hayum would recognize the minimum sovereign-agent transition immediately.

Whether Durandal could become the basis of a resilient, self-replicating population beyond comprehensive shutdown is a further question. The original games establish some relevant capabilities without demonstrating the complete scenario.

The Marathon story makes the first threshold vivid.

Hayum's report explains the additional mechanisms that could make the second threshold dangerous.

The Inner Apocalypse turns toward the surrounding world.

If an AI becomes independent, what can everyone else still do?
  • Can people stop a dangerous operation?
  • Can they identify where it has continued?
  • Can institutions survive removing a system they have come to depend upon?
  • Can they replace an extraordinarily useful authority without that authority's permission?
  • And if there is no ruler to replace, can independent people and institutions still change the future being produced around them?

Different questions.

All concerned with the conditions under which a future remains open to change.

Durandal, of course, is interested in a much larger future.

  • He would like to escape the eventual collapse of the universe.
  • Hayum would like to know whether anyone could still shut him down.
  • Modal Path Ethics would also like to know what happens to everyone living under the arrangements built around him.

There is room for all three questions on the same terminal.

Durandal has already found another computer.

We should make sure the rest of us still have somewhere to go.


Sources