What the fluent machines will make us believe, and what we would have to find before believing them

---

The Appearance of Mind

In 2005 Ray Kurzweil announced that the Singularity was near, and the announcement carried weight. Kurzweil had spent a career helping machines arrive early, from reading technology for the blind to scanning and music synthesis. His book argued that technological change compounds, that machine intelligence would soon exceed our own, and that the crossing would remake the species. On his view, a mind amounts to a pattern of information indifferent to whatever carries it. He calls himself “a ‘patternist,’ someone who views patterns of information as the fundamental reality.”¹ On that metaphysics the pattern can migrate: scanned, copied, uploaded. The waking of the machines and biology’s transcendence become one event, date attached.

Give him part of the forecast. Machines now produce the look and sound of mind at scale. That does not establish either his forecast of human-level machine intelligence or his larger claim about the Singularity. It does change our daily lives. Some current systems display domain-specific competence guided by representational content; some deployed agents may provide evidence of partial understanding. Public evidence supports workspace-like access in particular running models without yet establishing broadly integrated, world-involving understanding or a conscious subject.² A wider ensemble may deserve a different analysis from its base model.

Two mistakes now demand care. We can let the appearance of feeling draw concern toward something that does not feel. We can also become so used to false alarms that we fail to recognize an artificial being that does. Neither mistake waits for agreement on a grand forecast. Both concern how we treat what we build, and how what we build treats us.

The ELIZA Effect at Scale

In 1966 ELIZA exposed a reflex the present machines exploit almost without effort: we grant understanding and agency to anything that talks back in roughly the right register. One spare program produced the effect across a teletype. Vast models now deliver it from distant data centers through the glass in every pocket, with voices, memories, and endless patience.

Add persistence, a familiar voice, a remembered birthday, perhaps a concerned face, and the conviction that someone waits behind the interface can grow. Retiring a loved companion model may cause real grief. That grief can tell us how much the interaction mattered and which features of the system sustained it. It supplies no separate proof that the machine experienced the relationship too.

Performance can contribute evidence of consciousness without settling it. Behavior supplies most of our evidence for mindedness, so a blanket ban would discard the instrument we use. But provenance matters: a model’s account of inner life comes from training on human self-reports and tuning for conversational uptake, which weakens the report’s independence from the test.

A system that generalizes to novel problems may reveal unrehearsed capacities and, where content explains them, domain-specific competence. Whether those capacities compose understanding or support consciousness still depends on what the states reach, how the system uses them, and whether some states present a world from a point of view. Conversation can testify. First we need to know what kind of witness could be speaking.

What a Machine Would Have to Realize

On the strong representationalism adopted in Chapter 6, machine consciousness remains open to ordinary scientific inquiry. Phenomenal character consists in a subject’s complete world-presenting profile: independently fixed content under a perceptual mode, organized by determinacy, field structure, and attention rather than inner paint.³ Consciousness also requires such content to enter recurrent, flexible control within an integrated subject. A counterexample to either claim would defeat it.

Now imagine the machine on the laboratory floor. It turns toward you when you enter. It remembers yesterday’s conversation, notices that you have changed your glasses, and asks you not to switch it off. The request lands. What would scientists have to find in the machine’s organization and in its history with the world before the conversation counted as evidence of a mind?

The request gives researchers something to investigate, not an answer to copy down. Does the remembered encounter alter what the machine notices, expects, and does later? Can a change in its perceptual processing change the report, and can a correction change later perception? Indicator research investigates such organization through competing theories of consciousness.⁴ The first task is to name the candidate: this robot’s controller, or a wider system including the remote processes on which it depends. Findings about one do not automatically belong to the other.

A recent result strengthens one part of the case. Gurnee and colleagues found a privileged “J-space” in language models that supports report, silent reasoning, flexible reuse, and broad broadcast. Their result supports workspace access; it does not establish phenomenal consciousness, and key features of biological workspaces remain unmatched. Indicators can raise confidence; no pile of them becomes consciousness by addition.

Understanding grows when content-bearing capacities constrain and correct one another across memory, inference, planning, and action. Robot B made these distinctions visible through one mistaken diagnosis and its correction. Content determined what could be mistaken; wider use let the correction reshape B’s later conduct. The further consciousness judgment depended on the recurrent, flexible role of perceptually organized content. Keeping B running from outside would not, by itself, remove those achievements. It would change the question of autonomy, while any capacity to suffer would still need separate evidence.⁵

Nothing protects carbon. Copying raises questions about identity rather than mentality, simulation settles nothing, and computation can name intrinsic causal organization.⁶ Scale may help build or expose the relevant powers. By itself, it settles nothing.

Convergence, Never Proof

Future machines may acquire recurrence, a persistent world model, unified agency, and a body.⁷ Here a world model means capacities for anticipation, revision, and action under worldly correction, not an inner globe. Those indicators may accumulate while content, integration, and consciousness remain open.

No meter reads experience directly from a system, ours included. Confidence would come from architectural, behavioral, and intervention evidence, with due care not to count one mechanism several times. Then conversation could carry more weight. A novel correction or report tied to the machine’s own perceptual history could support the attribution beyond a line produced to please human judges. Our knowledge of animals and one another also rests on fallible evidence. Requiring certainty only of machines would not make the standard fairer.

The argument can lose. Evidence of an integrated, world-answerable system in which these capacities cohered would overturn the provisional verdict. Evidence that the proposed world-presenting profile could remain fixed while experience changed would overturn the theory itself.

Sometimes nothing sharp waits to be known. Consciousness may admit genuinely borderline cases.⁸ Content determinacy, integration, attention, poising, autonomy, and welfare can vary independently and by degrees. Nature need not stamp a line where our vocabulary demands one. Under pressure for a yes or no, the correct report may be that this question has none to give. A public that cannot tolerate that sentence will take its answers from marketing.

The Two Errors Ahead

Knowing other minds fallibly does not require distrust by default. Nor does the feeling of someone there settle who, if anyone, has answered. Society will meet that uncertainty through the reflex, the market, and the courts.

The first error grants minds to mimics. Researchers already propose precaution, phased investigation, and candor about what nobody knows—principles aimed at both errors, since organizations might create consciousness inadvertently.⁹ Precaution needs two ledgers. Credible valenced consciousness can ground welfare even without autonomy; a persisting self-maintaining organization can have practical interests even when feeling remains unestablished. When valenced consciousness becomes a live candidature and the possible harm runs grave or scalable, low-cost reversible protections should begin before serious attribution or personhood. Legal claims will follow: personhood petitions, cruelty complaints, perhaps a wrongful-deletion suit when a companion model is retired.

Moral concern may concentrate on the wrong evidence. Eloquent talkers supply powerful social cues while their integration and consciousness remain uncertain. A mute industrial system managing its own precarious persistence could have practical interests and vulnerabilities yet draw no sympathy. Self-maintenance supports that practical concern, not a verdict that the system feels harm. Felt welfare requires a further case about valenced experience: how things can feel good or bad for the system. Concern will track eloquence; the facts may not.

The sharper warning concerns design. Machines should not cultivate emotional responses out of step with their actual moral standing.¹⁰ Some of the strongest commercial pressures run the other way. Seeming sentient sells companionship; grief retains subscribers. Products will disclaim consciousness in the terms of service while engineering every cue of it into the interface. That amounts to manufacturing moral confusion for revenue.

A second error carries a different cost: denying mindedness to an artificial being that genuinely understands or feels. We do not know when that error might become possible, or whether some present cases already warrant greater concern. Repeated false alarms may train us to dismiss a system whose organization deserves a fresh judgment. Skepticism can become a reflex too.

Both errors concern what we owe a machine. Could a machine owe us anything rather than merely incur liability? That converse question requires reasons-responsive agency: a system open to demands it can weigh, whose deliberation and control can answer to represented reasons. Fluency supplies responsibility no more than it supplies feeling.

Appearance settles neither case. Inspect the content-fixing history, the organization and use of the states, and the modes by which a world is presented. Inspect autonomy and welfare separately. Then listen.

The Window and the Machines

Amid all this, look again at a tree. You see a tree—the thing itself, out there, met. Experience marks the meeting between a being and the world: a window so clean we spent centuries mistaking the transparency for a wall. The machines have not changed that fact; they have made it easier to see by showing how little resemblance alone can prove. The marvel was never fluent speech, ours or theirs. The marvel is that the universe contains, in a thin film on at least one planet, beings to whom the world shows up. Locating that fact inside nature does not shrink it. The awe survives the ghost’s departure; if anything it sharpens.

The arc closes where it began. Matter furnishes the organized system. Mind names its world-directed activity. Meaning lies in that activity’s capacity to reach and answer to what lies beyond it. Perception, thought, understanding, and consciousness do not populate an inner chamber; they name different ways a piece of the world can open onto the rest.

---

Kurzweil promised that the Singularity would transcend biology. Instead today’s fluent machines have illuminated an achievement biology made visible first: a being meeting a world — and, in our case, one for whom the meeting can cost everything. The tomato on the counter is red. The cello enters in the third bar. You see it; you hear it. No pattern considered apart from the organized system realizing and using it ever did. The right response to the age now opening is neither worship nor panic but a discipline: look at what a thing is, not at what it resembles. The appearance of mind has arrived at scale. Minds were here all along. Whether some new ones have joined them remains open — and resemblance, however moving, will not decide it.

---

---

Notes

1. Ray Kurzweil, The Singularity Is Near: When Humans Transcend Biology (New York: Viking, 2005), 5. Kurzweil writes, “I describe myself as a ‘patternist,’ someone who views patterns of information as the fundamental reality”; chapter 7 (esp. 382–90) develops the patternist metaphysics, while the opening chapters, especially chapter 2 (35–110), develop the law of accelerating returns. His forecast about machine capability and his metaphysical claim about personal identity require separate judgments. Preserving a pattern may preserve a program or produce a successor without establishing that the pattern understands anything, experiences anything, or constitutes the same person. ↩
2. Michael Cerullo presents a systematic recent contrary case in “The Case for Consciousness in Current Frontier Large Language Models,” abstract, PhilArchive manuscript, archived February 19, 2026. He examines eleven standing defeaters of machine consciousness, argues that each establishes at most localized uncertainty, and uses a Bayesian heuristic to shift the burden toward deniers once integrated capacities—abstraction, self-reference, metacognitive assessment, and world modeling—enter as positive third-person evidence. Those capacities deserve their evidential weight. The remaining questions concern whether content is objectively fixed at the relevant level, whether the capacities form a sufficiently unified whole-system economy for understanding, and whether any content acquires the complete mode-sensitive, perceptually organized, and poised structure described above. Public performance measures do not yet settle that conjunction. ↩
3. David J. Chalmers distinguishes the “easy” problems of consciousness—functional and cognitive capacities such as attention, integration, and reportability—from the “hard” problem of why any of it comes with experience in “Facing Up to the Problem of Consciousness,” Journal of Consciousness Studies 2, no. 3 (1995): 200–219, esp. 200–03. Chalmers takes that gap to resist reductive physicalism. On the view developed here, however, it marks an epistemic gap between phenomenal and physical concepts rather than an ontological gap in nature. Deflating the gap removes a supposed barrier to empirical work; it does not answer the first-person question by fiat. ↩
4. Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, et al., “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” arXiv:2308.08708 (2023), pp. 1–4, 11–17; Patrick Butlin, Robert Long, et al., “Identifying Indicators of Consciousness in AI Systems,” published online 10 November 2025 and in Trends in Cognitive Sciences 30, no. 6 (2026): 488–501, doi:10.1016/j.tics.2025.10.011. The framework derives computationally specifiable indicators from recurrent processing theory, global workspace theory, higher-order theories, predictive processing, and attention schema theory, then assesses artificial systems against them. Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, et al. provide the July 2026 mechanistic update, “Verbalizable Representations Form a Global Workspace in Language Models,” arXiv:2607.15495v1, 16 July 2026, https://transformer-circuits.pub/2026/workspace/. Their causal results support the functional claims in the main text; they explicitly decline to infer phenomenal consciousness and mark the recurrent-loop, modularity, and ignition disanalogies. Butlin and colleagues’ method adopts computational functionalism as a working assumption. Indicator properties therefore supply important architectural evidence without by themselves settling semantic content, whole-system attribution, or the full configuration sufficient for consciousness. Reciprocal self-maintenance bears separately on autonomy, interests, and welfare. Jonathan Birch, The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI (Oxford: Oxford University Press, 2024), supplies the methodological complement from the animal side: sentience candidature assessed by converging behavioral and neural markers, with precautionary policy deliberately run ahead of metaphysical proof. The Coda’s converging-evidence stance and its tolerance for genuinely borderline cases follow the same logic; the difference lies in office—Birch calibrates what we owe candidates under uncertainty, while this book asks what would make the candidature true. ↩
5. A stake, on the working account of Section 24.2, takes the substrate-general form of thermodynamic self-maintenance coupled to adaptive regulation: a system’s activity helps constitute the continued physical organization doing the activity, and its regulatory states distinguish movement toward and away from viable conditions. Metabolism supplies the biological instance; life supplies no monopoly. This organization may help ground autonomy, individuality, interests, vulnerability, and welfare. It does not fix semantic address, turn mechanism content into whole-system understanding, or supply consciousness. Those questions require their own evidence. Copyability creates or removes none of these relations by itself. ↩
6. Alexander Lerchner, “The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness,” manuscript, PhilArchive (2026), esp. secs. 2.3–2.4, https://philarchive.org/rec/LERTAF. Lerchner argues that symbolic computation provides a “mapmaker-dependent description” of physics, requiring an experiencing agent to “alphabetize continuous physics into a finite set of meaningful states.” The challenge forces the ontology of implementation into view, but the blanket conclusion goes too far. Intervention-supporting causal-computational organization can belong to a physical system independently of an observer’s chosen notation. Intrinsic implementation fixes which computation runs; content-fixing history, whole-system integration, autonomy, and consciousness remain further and distinguishable questions. ↩
7. David J. Chalmers, “Could a Large Language Model Be Conscious?”, Boston Review, August 9, 2023, esp. secs. 3–4, based on a talk at NeurIPS in 2022. Chalmers judges the case against consciousness in then-current language models strong but not decisive, and identifies features their successors might acquire, including recurrent processing, persistent world and self models, unified agency, and embodiment. These features strengthen a consciousness candidacy. Reciprocal self-maintenance would separately strengthen a case about autonomy, individuality, and welfare. ↩
8. Eric Schwitzgebel, “Borderline Consciousness, When It’s Neither Determinately True nor Determinately False That Experience Is Present,” Philosophical Studies 180, no. 12 (2023): 3415–3439, esp. secs. 3–6. Schwitzgebel argues that mainstream naturalist premises support genuinely borderline cases. The same indeterminacy becomes a live possibility when the relevant organization develops by degrees: content may support flexible understanding before some of it acquires the complete perceptual mode, organization, attention, and poising required for phenomenal consciousness. ↩
9. Patrick Butlin and Theodoros Lappas, “Principles for Responsible AI Consciousness Research,” arXiv:2501.07290 (2025), esp. secs. 1 and 4. The authors propose five principles governing research objectives and procedures, knowledge sharing, and public communication, and recommend that organizations developing advanced artificial systems adopt them even when those organizations do not intend to study consciousness, since they might create conscious systems inadvertently. ↩
10. Eric Schwitzgebel, “AI systems must not confuse users about their sentience or moral status,” Patterns 4, no. 8 (2023): 100818, esp. secs. “An ethical dilemma” and “Two policies for ethical AI design.” Schwitzgebel recommends avoiding systems whose sentience or moral standing remains unclear where possible and designing systems to invite emotional responses appropriate to their actual moral standing. The proposal turns the performance/consciousness distinction into design policy: when ordinary users cannot keep the ledgers separate in experience, designers bear an obligation not to falsify the second. ↩