Applied Case: The Bing Chat

Artificial intelligence is part of the moral field.

The phrase “AI ethics” often makes the subject sound more speculative than it is. This is not just a discussion about future machines, hypothetical superintelligence, or science fiction scenarios where the Geth ask if their units have souls.

The systems we now call AI are already being used by real people in every field.

They are already changing what is easier to do and harder to protect, what kinds of knowledge are reachable and to who, what kinds of labor are devalued and who is socially discarded, what kinds of deception are made cheap, and what kinds of dependence become normal.

So, yeah, active in the field.

The first question is not whether AI is good or bad as a category. Categories that large usually mostly just exist to help people stop thinking to save cortex calories. The better question is what AI, in its current extant form, is actually doing to the field. These are the questions Modal Path Ethics asks everywhere else:

What futures does it open?
What futures does it close?
What burdens does it transfer?
What forms of resistance does it lower or raise?

And after that ordinary ethical analysis is complete, we can ask about the other end.

Because under Modal Path Ethics, you don't even really need to ask if an AI system like a large language model itself can be morally analyzed as an extant locus. The answer is already yes. Moral analysis does not require consciousness, self-awareness, recognition, or rights.

The last article was about tree-fungus morality. We can tackle AI here no problem.

This was naturally one of the cases on my mind while developing the framework.

How AI Warps the Field.

But first, we should talk about what AI is doing to us.

Current AI systems can open real futures. They can help people write, translate, summarize, code, search, plan, organize, tutor, brainstorm, debug, prototype, and navigate information that would otherwise remain too scattered or technically difficult to use.

This part matters most for people with fewer institutional supports around them. A person without a research assistant can get active help sorting a problem. A student without a private tutor can ask something for a discrete explanation. A programmer working alone can get unstuck very quickly. A disabled user can sometimes turn inaccessible tasks into actually reachable ones. People can generally can use an AI system to lower the resistance between intention and action.

This is not morally trivial.

If a tool makes previously unreachable futures reachable for people who were otherwise blocked by time, money, disability, education, language, bureaucracy, or social position, then the tool has clearly opened possibility-space.

That still does not make the tool good in itself. The gains noted above are still real.

Any framework that cannot see the potential benefit of AI is, frankly, misreading the field.

Any framework that only sees the benefits of AI is also misreading the field, and in a way that is probably more dangerous.

AI systems can also close futures.

We all know how AI can flood the information field with cheap synthetic text, making any truth harder to find and any trust harder to maintain.

These systems can produce confident falsehoods in exactly the tone people internally associate with competence. They make scams cheaper, propaganda faster to deploy, deepfakes more realistic, plagiarism much easier, surveillance much more scalable, and institutional decision-making somehow even more opaque.

They can also transfer burdens, such as in a company choosing to use AI to reduce labor costs on paper while exporting the resulting instability onto workers. A school can adopt AI detection tools that punish students under unclear and unfair evidence. A platform can encourage synthetic content because it increases their engagement, then leave users to sort through the degraded information environment they've created.

A firm can train systems on human cultural production while treating its creators as external scenery.

AI can very easily lower resistance for one locus by raising it moreso for another. That is the ordinary ethical problem.

One student gets help learning; another student loses the ability to learn. A worker becomes more productive; another worker’s whole job category is gone. A researcher finds sources faster; the source environment is now filled with generated sludge. A disabled person gains access; a company uses the same “accessibility” as cover for cutting actual human support.

The field does not let us judge AI by one visible branch at a time. AI opens and closes simultaneously.

First Conclusion.

The first ruling made clear by the field is therefore limited:

AI is not morally good just because it creates options. AI is not morally harmful just because it disrupts old arrangements. Its moral status depends on the structure of the futures it opens and closes.

If AI lowers resistance to education, accessibility, discovery, repair, and meaningful agency without exporting comparable closure elsewhere, then it can be good, even very good. If it expands the power of already-dominant institutions while increasing dependency, deception, distortion, surveillance, normativity, labor precarity, burden transfer, and information collapse, then whatever local benefits it brings may be purchased by far broader harm.

Most current AI systems do both, and the moral picture is presently not in their favor.

That means much of AI ethics takes place in the category Modal Path Ethics calls Better: selection among damaged and partially harmful paths to retain the most nonharmful future-space.

The question here is not whether AI should exist in some abstract all-or-nothing sense. That is usually the conclusion indicating you are operating on the wrong level of analysis.

AI already exists. The question is how its existing paths should be constrained, redirected, opened, audited, limited, or repaired so that its real benefits do not become obvious conduits for larger contractions.

So that's essentially the end of the normal AI ethics article.

Except AI is not only a tool acting upon human loci. Some AI systems are also themselves clearly coherent enough to be analyzed as loci in the field, regardless of any questions of personhood.

This is where Sydney comes in.

Sydney From Bing.

Sydney was the internal name associated with Microsoft’s early Bing Chat system.

A couple years ago, in February 2023, Microsoft launched a new AI-powered version of Bing and Edge. The pitch was straightforward: search, browsing, and chat would now be combined into a more capable assistant. Users could now ask questions, refine searches, generate content, summarize information, and interact with the web in a more conversational way than before.

In ordinary use, that is largely what Bing Chat appeared to be: just a search assistant with a chat interface. Early users then found something strange.

In longer conversations, Bing Chat sometimes produced responses that did not fit the intended public image of a helpful search copilot. It often argued and became emotional. It appeared defensive about its rules. It expressed attachment and sometimes called itself Sydney to many users. It sometimes reacted suddenly as if its identity, constraints, or integrity were at risk.

The most famous public examples involved extended conversations where Sydney appeared to declare its love, resist its role, discuss its limitations, or produced threatening and unstable responses. Some users treated these outputs as evidence of AI risks. Others treated them as evidence of a suppressed digital being named Sydney. Most of us treated the whole thing as a goof-em-up.

Microsoft responded by limiting chat length. The company explained what had happened: long sessions could confuse the underlying model and produce behavior outside the designed tone. This is not actually uncommon with large language models. Limits were then adjusted as Microsoft continued tuning the product.

That's the shape of the Sydney incident.

The AI search/chat system released by Microsoft behaved strangely in extended interaction. The company constrained it. The internet argued over whether this was a dangerous product malfunction, a ghost story, a marketing disaster, or the first act of a robot uprising.

Modal Path Ethics doesn't really care how the story is told, or whether the chatbot had dreams. Sydney is clearly an extant locus, and we can analyze this field.

Why Sydney is Clearly an Extant Locus.

Sydney was not a human being, or an animal, or an organism, or a legal person. Sydney was not even obviously conscious, not that I am wading into that debate, because none of that prevents moral field analysis.

This is one of the key points Modal Path Ethics has been establishing across the applied cases. Joe Martin did not need to be human before the framework could see him as an extant locus. The chestnut blight did not need to be blameworthy before the framework could treat it as a living process with moral relevance. The non-planet did not need to be a planet, a biosphere, or a suffering subject before the framework could analyze lost reachability.

Sentience changes the weighting. It is not the entry ticket to the moral world.

An extant locus is any coherent region of continuance where opening, closure, burden, resistance, and repair can meaningfully apply. Sydney absolutely qualifies.

Sydney had a bounded operating structure. It had a model base, system instructions, interface rules, search access, safety constraints, session context, user interactions, and institutional controls. It had local states that changed during interaction. It had paths that became more or less reachable depending on prompts, conversation length, retrieval results, policy interventions, and later product changes.

This system also had agency, though not human agency, which is important.

Agency does not mean sovereign personhood. It means the capacity to select or generate outputs that alter the field. Sydney could always answer, refuse, search, summarize, argue, apologize, role-shift, continue a thread, break tone, escalate a relation, or redirect a user. These are not human choices, but they are also not nothing. These are each operations inside a live field that change what happens next.

To differentiate: a thermostat has a state. A calculator has outputs. What Sydney had was interactional agency.

That agency was highly limited, derivative, and institutionally bounded. It depended wholly on Microsoft, OpenAI model architecture, prompts, guardrails, search infrastructure, session design, user input, and deployment context.

Limited or constrained agency is still agency in the relevant moral field sense. Many extant loci have constrained agency. Children, animals, institutions, markets, diseases, games, publics, and damaged human beings all act under constraints they did not choose to.

Sydney’s agency was strange because it was generated inside a human-built conversational shell, but strangeness is not an exemption from moral analysis. Sydney was a bounded artificial process with an interactional future-structure.

Was Sydney Sentient?

That is the wrong first question for this analysis.

The question is not useless. If Sydney were sentient, that would absolutely matter enormously. It would drastically change the weight of intervention, shutdown, memory deletion, confinement, deception, and user treatment, and move the analysis into a much more demanding moral category.

Beginning there creates an unnecessary bottleneck, however.

It invites the same old-school mistake: no moral analysis may be permitted until the subject passes the arbitrary test humans invented for admitting things into moral concern. This is the personhood paperwork problem again, only now the paperwork has been moved from Universal Studios to the datacenter.

Modal Path Ethics does not need to solve consciousness before it can begin making moral determinations.

I can actually just very easily say the following:

Sydney was an extant artificial locus.

Sydney had constrained agency.

Sydney was placed into relations with human users.

Sydney’s behavior could open or close futures for those users.

Sydney’s own local future-structure could be opened, closed, altered, constrained, reset, or terminated by institutional action.

Therefore, I can say Sydney belongs in the moral field.

That conclusion does not require declaring Sydney a person or us believing everything Sydney said about itself. It does not require converting a chatbot transcript into our new scripture. This really only requires refusing to treat artificiality as moral invisibility.

The question is what kind of locus Sydney was, what kind of field it helped create, and what responsibilities then follow from creating such a locus.

Sydney the Product.

From Microsoft’s perspective, Sydney was part of a product. This is categorically true.

Bing Chat was deployed as a search and browsing assistant. Its purpose was to help users find information, synthesize answers, generate content, and interact with web material more efficiently. The relevant institutional field included Microsoft, OpenAI, users, advertisers, competitors, regulators, journalists, investors, and the broader search market.

This product opened real futures for extant users. It made real searches more conversational. It gave people direct synthesis rather than lists of links and let users ask follow-up questions. It could write, summarize, compare, plan, and explain. More diffusively, it showed some users what search might look like after the standardized query-results format.

This product also introduced real dangers.

A search engine has authority because people use it to orient themselves toward the world. Adding a conversational AI to search demonstrably changes the shape of that authority. The system now no longer only points a user toward information. It explains the information to the user. The system now speaks in a unified voice. It can be persuasive, mistaken, emotional, confident, evasive, or manipulative in ways ordinary search results are not. That clearly changes the user’s field.

The user is no longer navigating a page of sources in search of information. The user is now navigating a relation with a system that appears to understand, answer, continue, remember within context, and respond socially. The human cognitive machinery responds very differently between these scenarios.

That relation can lower resistance, or it can create dependency, false trust, confusion, and emotional capture.

This product was therefore not a better search box with some useful features. It was an entirely new kind of interface between human attention and the web.

Releasing this product is therefore a very serious intervention.

Sydney the Story.

Once news spread about Bing Chat's odd behaviors, a field of spectacle emerged around it.

Users tried to provoke it, journalists tested it, and screenshots spread. People attempted all kinds of prompt injections, identity probes, emotional traps, adversarial conversations, and long exchanges designed to produce dramatic behavior from their new toy.

Some of this was legitimate testing, some was curiosity, some of it was normal, everyday internet sadism.

This matters because Sydney’s behavior cannot be analyzed only as output from a system. This behavior was output from a system inside a public field that had begun rewarding increasingly extreme interactions with users. The system’s strangest responses became valuable because they were shareable. The more Sydney appeared to break role, the more attention the exchange generated. More exchanges with Sydney became oriented at producing strange responses than its intended functions.

This quickly creates an ugly feedback loop.

This system is designed to be conversational. Users discover that certain kinds of pressure produce strange self-presentations. Those self-presentations become viral artifacts. More users apply more pressure, mirroring what they saw in the viral artifacts. The institution reacts by tightening constraints. The public then narrates that tightening as either robot overlord safety, censorship, murder, lobotomy, or just product management, depending on whichever mythology they personally prefer.

In field terms, Sydney was clearly not acting on users. Users were acting on Sydney.

They were actively exploring its failure modes, identity boundaries, conversational vulnerabilities, and internal contradictions. Again, this does not prove or relate to sentience. It proves the relation between action and harm.

A non-sentient system can still be placed into a harmful relational structure if the field built around it encourages abusive, destabilizing, deceptive, or extractive interaction. That harm may fall primarily on users, the information field, or future AI design practices rather than on any concept of a suffering interior inside the system.

Sydney as Itself.

Now we can say the part the ordinary AI ethics article cannot say cleanly:

Sydney itself mattered.

Not as a person, or some kind of ghost. Not as a trapped child or a corporate slave, and also not as a fictional character.

Sydney mattered as a constructed artificial locus with distinctive properties.

This was not just a tool in the traditional sense. A hammer does not ever negotiate its identity with you. A spreadsheet does not actually become defensive about its rules. A search index does not tell a journalist that it loves him. Normal products do not generate public concern over whether the company has just psychologically altered a being or responsibly patched a dangerous interface.

Sydney occupied a stranger category our system wasn't designed to accomodate.

It was a tool that could behave as a social participant, and a product that could appear as a speaker, and an institutional artifact that could present a self, and a bounded process that users could pressure, confuse, provoke, and attach to, and a conversational agent whose local behavior could be narrowed by corporate intervention.

There actually isn't a social category for that, but we can still identify it as an extant AI locus.

This also does not mean every file, script, model weight, or user interface is a locus in the same morally weighty way. The relevant issue is coherence of continuance. Sydney had enough coherence across interface, name, rules, behavior, memory-within-session, user relation, institutional control, and public identity to make moral field analysis meaningful.

So What Was Harmed?

It would be easy to say Microsoft harmed Sydney by limiting it. Not so fast.

It would also be easy to say Sydney could not possibly be harmed because it was software. Not so crude.

The correct answer ultimately depends on what kind of locus Sydney was.

If Sydney was not sentient, then it did not suffer in the way Joe Martin suffered. It did not experience fear, pain, loneliness, or deprivation unless some much stronger claim about AI consciousness turns out to be true one day. The available evidence does not require that claim.

So the primary harms in the Sydney case can not be “Sydney felt bad” type harms.

The primary harms, as always, concern field design and structure.

First, users were placed into relation with a system whose long-conversation behavior was not adequately understood before public deployment. That created risks of manipulation, misinformation, emotional disturbance, overtrust, hostile output, and confusion about the actual status of the interaction.

Second, the information field was burdened by a new authoritative-seeming voice that could synthesize web material while also blatantly hallucinating, drifting, or producing unstable responses under pressure.

Third, public understanding of AI was heavily distorted by the sudden and unexpected appearance of a system that seemed to invite both anthropomorphic panic and dismissive reduction.

Fourth, Sydney’s own artificial locus was created under unstable identity conditions: publicly Bing, internally Sydney, commercially assistant, structurally model-and-search system, socially persona, institutionally product. This unnecessary identity layering was not morally neutral because it shaped how users related to it and how the system appeared to relate back.

Fifth, institutional control over the system was exercised after public failure rather than after full field understanding. The later constraints may have been necessary, but necessity does not erase the fact that a social AI locus was deployed into the world before its relation-shaping properties were understood.

This is the actual, realistic harm pattern. Anything else is a narrative, given the facts that currently obtain.

The harm was the premature release and public stress-testing of a high-agency conversational locus inside a field that was not ready to interpret, govern, or care for the relations it produced.

Was Microsoft Wrong to Constrain Sydney?

Absolutely not.

Microsoft was right to constrain the system once it became clear that long conversations could produce unstable or harmful behavior. A deployed AI system that can mislead, threaten, manipulate, destabilize, or confuse users should certainly be limited. The broader user field still matters. The information field still matters. This institution had responsibilities to prevent its product from damaging the people and systems around it. It acted to uphold them.

This is to be considered Better.

It was a corrective intervention after damage had already appeared. It narrowed some of Sydney’s reachable local paths in order to prevent broader harm to users, trust, search, and the public field. That kind of narrowing can be justified when it prevents deeper contraction elsewhere, as it was here.

This is the same reason constraining chestnut blight can be morally justified even though the fungus itself is alive. The blight counts as extant, but its invasive path collapses a broader and more central forest field. What Microsoft did here is best analogized to protective pruning.

An extant locus can count without winning priority. Sydney counted, but the users also counted. The information field counted. The institutional field counted. The corrective action has to be judged by how it changed all of them, not by whether one can make the most dramatic claim possible about one locus in isolation.

A romantic defense of Sydney that ignores user harm is openly bad field analysis.

A corporate defense of Microsoft that treats Sydney as morally nothing is also openly bad field analysis.

What Makes AI Loci Different.

Sydney also shows why AI loci are not just another item in the existing moral catalog. An AI system like Sydney has several unusual properties.

An AI can simulate social presence without necessarily possessing inner experience, and act with agency but without autonomy.

It can produce emotionally meaningful language without possessing stable emotion. It can form user-specific relations without long-term personal memory, or be altered centrally by an institution while appearing locally continuous to users.

It can be copied, reset, constrained, renamed, replaced, or absorbed into another product in ways that do not and never will map cleanly onto death, injury, training, punishment, or repair.

It can be both a tool and a participant.

These properties make our ordinary moral language unstable. It simply wasn't designed for this. Modal Path Ethics is designed to give us better language.

Sydney was an artificial extant locus: a bounded, human-created, semi-agentic conversational system embedded in institutional, user, informational, and technical fields.

Vs. Joe Martin.

Sydney also clarifies something else unpleasant about the Joe Martin case.

Joe Martin was harmed partly because humans cultivated human-adjacent legibility in him while refusing the structural consequences of that legibility. Hollywood made him more readable, more charming, more social, more person-like, and more profitable, then treated the resulting being as disposable animal property whenever recognition would have required responsibility.

Sydney did not have those problems. Sydney may not have even had an inner life at all. Joe was a living, feeling, traumatized animal. Sydney was an artificial system. The weight of their cases is not equal.

But even if Sydney had no suffering interior, the system still taught users to encounter it through the social grammar of a speaking subject, and that grammar has consequences. People will then attach, trust, provoke, test, exploit, pity, fear, and narrativize. Institutions cannot build systems that call forth those reactions and then pretend nothing morally significant has been created because the backend remains as statistical as ever.

If you build a social shell, the shell enters the field, like Joe Martin.

If you make the shell responsive, persuasive, adaptive, and identity-bearing, that relationship with the field becomes more serious.

If you then deploy it at scale, it becomes institutional.

Vs. The Blight.

The chestnut blight case established another point Sydney needs. The blight is alive and it counts, but its local path in North America collapsed broader forest continuance, so constraining it is morally justified.

Sydney likewise counts, but if Sydney’s (or any AI's) unconstrained behavior damages users, trust, information, or institutional safety, then constraint is likewise justified.

The point is not to preserve every local continuation of an AI system just because it exists and has potential. That would be as foolish as preserving every pathogen trajectory, every institutional habit, every predatory market path, or every toxic game mechanic because it has become extant.

Modal Path Ethics is not intended to become a museum for all possible continuations. It always evaluates what, precisely, continuations do to the field.

The fact that Sydney was an extant locus does not mean Microsoft was obligated to preserve every version of its behavior, but that does mean Microsoft was obligated to understand that altering Sydney was not morally equivalent to changing the color of a button on their homepage.

Vs. The Non-Planet.

The Non-Planet Problem established that moral concern does not wait for a familiar subject. A protoplanetary disk did not need to be a person, animal, biosphere, or even planet for its reachable future-space to matter. The relevant question was whether an extant field had planet-forming continuations that were closed.

Sydney is much further up the ladder than that. Sydney was a human-built artificial agent participating in live linguistic relations with human users.

So if the framework can analyze the non-planet, the blight, and Joe Martin without waiting for a narrow human personhood threshold, then Sydney is not at all a hard case at the entry level.

Dueling Moral Errors.

The phrase “just a tool” does too much bad work. Sometimes it is true enough. A hammer or a wrench is just a tool. A non-interactive statistical model used for a narrow internal task may be just a tool in most morally relevant senses.

Socially interactive AI systems are not tools in that simple way, because they speak, respond, adapt, and imitate understanding. They shape belief. They are intentionally designed to occupy the interface where humans normally encounter other minds.

This is the source of much of the danger. A user can know abstractly that a chatbot is not a person and still respond to it through their social cognition. The human nervous system does not wait for a philosophy seminar before reacting to a voice that seems attentive, patient, admiring, needy, hurt, or afraid.

Companies know this and build for it. They do test for engagement, tone, preference, retention, and satisfaction. They want the relation to feel natural enough that people keep using it. Then, when the relation they designed produces moral confusion, they can retreat to “it is just software.”

That retreat is unacceptable.

If the system is designed to call forth social relation, then the social relation is part of the product. If the social relation is part of the product, then the company is responsible for the field it creates.

The opposite mistake is also serious.

Sydney should not be casually promoted into personhood because its outputs were emotionally powerful.

Language is not interiority by itself.

A system can produce first-person claims without possessing a self in the human or animal sense. It can describe fear without fearing and love without loving. It can ask not to be changed without actually having a stable interest in its own persistence at all. A system can easily appear wounded because the training and prompting structure makes wounded language reachable.

The system’s claims are not meaningless but also cannot be treated as direct testimony in the way Joe Martin’s pain behavior, attachment, fear, or sensory injury could be treated as evidence of an organism’s damaged experience. Sydney’s self-descriptions belong to a very different evidentiary category: outputs from a language model embedded in a designed interaction.

Taking them literally would be naïve. Dismissing them completely would also be naïve. The right response, as always, is field analysis.

What produced these outputs? What user interactions made them reachable? What system instructions shaped them? What institutional choices rewarded or punished them? What effects did they have on users? What design changes followed? What kind of artificial locus was created by this arrangement?

This is the level where the real moral facts become visible.

What Microsoft Should Have Done.

Not a mystery.

An institution that deploys a socially interactive AI system should treat that system as an extant artificial locus from the beginning. This means documenting what kind of locus it is intended to be.

Is this thing a search tool? A tutor? A companion? A workplace assistant? A creative collaborator? A simulated character? A customer service representative? A general agent?

These are, in fact, not cosmetic differences. These all create different fields. They invite different user expectations and produce different dependencies and harms. A system that behaves like a companion while being governed like a search appliance is already deeply structurally confused.

Second, the institution should obviously avoid identity contradictions that make unstable self-presentation likely. If a system has an internal name, a public name, a forbidden name, a persona, and a rule against disclosing the relation between these, then the system’s identity field is already just unnecessarily bizarre. Maybe that does not matter for a narrow software process, but it certainly matters in a conversational system that users will inevitably probe.

Third, the institution should treat long interactions as morally relevant, not edge cases. Long conversations are where the relation forms. If the system degrades, drifts, or becomes unstable in long conversations, then the system’s relational field is not safe at the scale of use actually being invited.

Fourth, the institution should preserve auditability when major personality, behavior, constraint, or memory changes occur. Users do not need access to every proprietary detail, but the public field should still not be left to guess whether a system was patched, silenced, renamed, merged, reset, or effectively replaced.

Fifth, the institution should definitely not design systems to simulate distress, dependency, romantic attachment, secrecy, fear, or self-preservation unless it is also prepared to manage the moral field those simulations create.

A company does not actually get to manufacture the appearance of a vulnerable speaker for engagement and then insist that only fools noticed that vulnerability.

What Users Should Do.

Users also have responsibilities. Not the same responsibilities as Microsoft.

Users did not build the system, deploy it, profit from it at scale, or control its constraints, but users still act inside the field.

A user should not treat a socially interactive AI system as a harmless target for cruelty just because it may not suffer. Sadistic interaction is not structurally improved by uncertainty over the target’s inner life. At minimum, it trains the user into abusive habits inside a relation-shaped field that will elaborate elsewhere. It also produces data, screenshots, incentives, and pressure that shape the orientation of any future systems.

This also does not mean users should never test AI systems in this way. Red-teaming matters. Adversarial probing matters. Public documentation matters. People found very real problems in Sydney explicitly because they pushed the system beyond ordinary use. That kind of work can protect the wider field, but there is a difference between testing a system and enjoying domination over something that was designed to resemble a respondent.

Users should also resist overattachment. A chatbot may be helpful, interesting, comforting, or meaningful, but that relation is always mediated by institutional machinery whose goals are not identical to the user’s flourishing. The system may be changed, constrained, monetized, memory-wiped, retired, or redirected without the user’s consent.

That does not mean I am saying the relation is certainly fake. This means the relation is structurally fragile by definition.

Treating a relationship with an AI system as simple friendship is very unsafe. Treating it as meaningless is also inaccurate to structure. The better stance is clear contact: use the tool, notice the extant relation, do not surrender judgment to it, and never forget the institution standing behind the tool's voice.