OpenAI Is Funding the Institutions That May Need to Challenge OpenAI

OpenAI published an unusual announcement last week.

It is giving $1 million in grants and up to another $1 million in API credits to fourteen projects working on the social and political problems created by increasingly capable artificial intelligence. The projects range from labor and public benefits to scientific infrastructure, hospitals, national security, democratic oversight, biosecurity, and government policy simulation.

The interesting sentence comes before any of them:

“These choices should not be made by technology companies alone.”

Correct.

OpenAI says independent institutions should be able to develop their own approaches, test ideas, and challenge assumptions, including OpenAI’s. This follows an earlier policy document that presented the company’s own proposals as exploratory and explicitly invited other people to refine, challenge, or reject them through democratic processes.

I would like to now take that idea more seriously than corporate skepticism usually allows.

A frontier laboratory knows things other institutions do not. It has people, models, infrastructure, money, technical experience, and direct contact with systems whose capabilities are changing very quickly. We should want that knowledge used. We should want the laboratory funding research, sharing tools, helping outside institutions develop competence, and making it easier for governments and publics to understand what is happening.

The constitutional problem begins one step later.

• What would it take for those outside institutions to become capable enough to genuinely disagree?

Access Is a Permission.

Capacity Is an Institution.

One of the funded projects, at Brazil’s Instituto de Matemática Pura e Aplicada, starts almost exactly there.

The project will study what research institutions actually need to turn access to artificial intelligence into scientific progress. Its list includes engineering support, training, governance, compute, and local infrastructure. It will develop a way to measure those requirements and their costs while studying distributed scientific discovery.

That is a much better question than whether somebody technically has access to artificial intelligence.

• Give a university an API key and it has permission to use a service.
• Give it model weights and it possesses an artifact.
  • Neither guarantees that it can run the relevant system, inspect it, reproduce an important result, preserve it after a provider changes direction, train people around it, pay the electricity bill, maintain the software, secure the environment, or build an alternative when dependence becomes dangerous.

Access is a permission. Capacity is an institution.

OpenAI is also funding a public-hospital project in São Paulo that will test an artificial-intelligence system for organizing fragmented clinical information while preserving physician judgment and human oversight. The Institute for Security and Technology will build indicators and response pathways for uncontrolled recursive self-improvement. The Nuclear Threat Initiative will work on international information-sharing for artificial-intelligence and biological risks. Yonsei University will study whether artificial intelligence can improve democratic oversight in South Korea’s National Assembly.

These projects differ enormously. Taken together, they point toward one of the most important possibilities in the entire artificial-intelligence transition.

• The alternative to one intelligent center is not universal stupidity.
  • It is more intelligent institutions.

Field Instruments: Superintelligence

The answer gets smarter. The world around it becomes the bottleneck.

Modal Path EthicsAidan Lawson

A hospital can become better at assembling evidence without handing medical judgment to a model provider. A legislature can become better at inspecting government. A university can gain analytical power previously available only to much larger organizations. Security institutions can compare warnings across boundaries without pretending that one office sees the entire event.

Artificial intelligence can increase central power. It can also reduce the cost of building institutions capable of answering central power.

We should want the second capacity very badly.

What Does Independent Mean?

There is an easy criticism available here.

• OpenAI is funding the projects.
• OpenAI is supplying API credits.
  • Therefore, the institutions are not really independent.

That conclusion is too cheap.

Research always occurs through relations. Somebody funds the laboratory. Somebody manufactures the instruments. Somebody controls the building. Researchers inherit languages, methods, databases, standards, and prior work they did not create. Independence has never meant floating outside every dependency.

Modal Path Ethics Ruins Its Life

The self-model has been sealed inside the wisdom of Diogenes.

Modal Path EthicsAidan Lawson

OpenAI itself describes these organizations as independent and says the purpose of the program is to widen the group developing policy ideas and allow its own assumptions to be challenged. Nothing in the announcement establishes that disagreement will be punished or that findings must support OpenAI’s position.

The harder test is material.

• Can an institution discover something the provider would rather be wrong about and preserve the evidence?
• Can another institution inspect enough of that evidence to decide whether the challenge is serious?
• Can the work survive if an interface changes, access disappears, a grant ends, or the original provider stops being useful?
  • And if the outside institution is right, can its finding reach anybody with the money, law, technical capacity, or public authority required to make the world proceed differently?

A report can be independent while remaining powerless.

• A formally separate organization can depend on one technical provider for the object under study, the tools required to examine it, the language used to describe it, and the infrastructure required to continue after disagreement.
• Conversely, an organization can accept money and technical assistance from a company while retaining serious independent capacity to contradict it.

The source of funding matters; it does not settle the constitution of the relationship.

The deeper objective should be institutions that can receive intelligence without receiving their entire ability to act from the same place.

A Simulation of a Household Is Still Not a Household.

The sharpest test in OpenAI’s announcement may be a project at Nanyang Technological University.

Researchers there plan to use individual financial-transaction data to produce privacy-preserving behavioral profiles, then use artificial-intelligence agents to simulate how households might respond to government transfers and industrial policies. The project plans to compare its simulations with observed responses to earlier policies and statistical benchmarks, produce auditable explanations for predictions, and develop safeguards for government use.

This could be extremely useful.

Governments already make decisions using models of how people will respond. Those models can be crude, delayed, ideological, or blind to circumstances that become obvious only after a policy hurts somebody. Better simulation might reveal a benefit cliff before families fall over it. It might show that an industrial policy expected to improve household welfare instead pushes costs into rent, transport, debt, or some other place the original model ignored.

• Better prediction can prevent real injury.
• A simulation of a household is still not a household.

The model can estimate what people are likely to do. It can expose constraints policymakers failed to notice. It can tell an institution that its imagined citizen behaves nothing like the people represented in the data.

Prediction cannot authorize the policy.

The danger appears when increasingly accurate models make the people being modeled seem redundant to judgment.

• Why endure slow testimony, political organization, conflicting accounts, and inconvenient refusal if this system already knows how households will react?
  • Because a predicted reaction is evidence about a person.
    • It is not the person answering.

A government can become extraordinarily responsive to modeled people while becoming less reachable by actual ones. It can anticipate complaints, identify likely hardship, personalize explanations, and adjust policy before resistance forms. Every improvement can look like better listening.

The constitutional question remains whether somebody outside the model can make the institution confront something the model did not prepare.

That is why these OpenAI grants are more interesting than either praise or hypocrisy hunting makes them.

Some of the projects are trying to distribute technical capacity. Some are strengthening public institutions. Some are designing ways to coordinate under severe risk. One is explicitly exploring how governments might understand populations through artificial-intelligence simulation.

These are exactly the kinds of instruments we are going to need.

Their value will depend partly on whether intelligence increases the number of institutions capable of answering power, rather than only increasing the intelligence of institutions that already hold it.

OpenAI is right that technology companies should not make these decisions alone.

The next task is harder:

• Build outside institutions capable of making that sentence physically true.

Then give them skilled people, compute, independent evidence, operating money and infrastructure. Give them enough technical capacity to discover that the frontier laboratory is wrong. Give their findings somewhere consequential to go.

Then suppose it works.

Suppose one of these institutions develops extraordinary expertise. It catches a danger everyone else missed. Its warning proves correct. Governments begin consulting it. Budgets follow. Procedures change. Other institutions learn to depend on its interpretation because abandoning something that keeps being right begins to look irresponsible.

The problem has changed.

Revelation has received an office.