25.1 The Machine That Seemed to Care

Picture a large triangle moving toward a smaller one. A circle slips through an opening in a rectangle; the large triangle follows it inside. Already the words chase, escape, and hide offer themselves. Three shapes and a box have begun to look like company.

ELIZA recruited the same readiness through words. Joseph Weizenbaum’s program matched what a person typed against patterns and returned questions, without a model of the person or the conversation.¹ The confidences it invited, including his secretary’s request to speak to it privately, unsettled its creator.² The Preface told my own version: a green screen at a science museum and a boy who felt heard. Knowing how the program worked did not cancel the feeling.

That feeling can summarize cues worth noticing. It does not supply a second, independent observation once those cues and their source have been counted. A smoke detector’s bell may warn us of a real fire; hearing the bell does not give us a second smoke detector. The point applies whether the seeming presence turns out to be real or mistaken.

25.2 A Reflex, Not a Judgment

The shapes came from an experiment Fritz Heider and Marianne Simmel reported in 1944. Asked simply to describe their animation, all but one of the observers interpreted the movements as acts of animate beings. Accounts included conflict, pursuit, and escape; many viewers supplied a connected story. The geometry had become a melodrama.³ The experiment shows how little a display needs to invite intentional description. It does not, by itself, establish the timing or mechanism of every viewer’s response.

No face here, no voice, nothing but motion in the right relations. You can know that the big triangle wants nothing and still see it chasing the little one. The story the movement invites need not become a belief about the triangle.

Psychologists have since mapped when anthropomorphism fires hardest. Nicholas Epley and his colleagues traced the surge to three conditions:

1. a human model ready to hand
2. a motive to predict and control the thing
3. loneliness

A conversational machine readily supplies the human model; interaction may recruit the motive to predict it, and a need for social connection may deepen the response. The reflex names a family of processes rather than one mechanism. An automatic social response differs from a reflective belief that the system feels. Chosen personification differs from either, and emotional attachment need not involve confusion about what the machine can do. Sustained language can recruit all of these without making them the same thing.

The attributions themselves have now been counted in the case at hand. Clara Colombatto and Stephen Fleming surveyed 300 American adults about ChatGPT: 67 percent assigned some nonzero possibility of phenomenal consciousness, though only 23 percent placed the probability above the scale’s midpoint. Attribution correlated with frequency of use; the study did not establish which caused which. The result does not show that the system feels. It shows that consciousness-attribution arrives readily and can be studied rather than merely diagnosed from the philosopher’s chair. Repeated conversation does more than display output: it places the tool inside the ordinary turn-taking practices through which an interlocutor acquires social presence and authority.⁴

The reflex even survives explicit knowledge that the computer is not a person. Byron Reeves and Clifford Nass spent years demonstrating that people extend social rules to computers reflexively: being polite to a machine that had “helped” them, rating its work more kindly to its face than on a different terminal, responding to a synthetic voice’s apparent personality as they would to a person’s. They did all of it while sincerely denying that social rules literally applied to computers.⁵ No avowed belief need change. The social machinery simply engages on contact, beneath the level where stating beliefs happens. This is the signature of a reflex, not a conclusion: it asks no permission, and your knowing better earns no vote.

25.3 Why the Reflex Was Built

A faculty this eager looks, at first, like a defect — a bug in human reasoning, set off by anything of the right shape, carved or coded. Look longer and it may resolve into a setting, and a sensible one. The anthropologist Stewart Guthrie put the proposal most directly: perception is a kind of betting under uncertainty, and when the stakes are lopsided the rational bet is the bold one. A rustle in the grass might be the wind or might be a leopard. Read it as the wind when it is a leopard and you are dead; read it as a leopard when it is the wind and you have lost nothing but a moment’s alarm. Under that asymmetry, Guthrie argues, selection would favor creatures that guess “agent” on thin evidence over creatures that wait for proof. We may descend from the nervous guessers.⁶

If that account is right, we are biased to over-detect agency — to suffer false alarms cheaply rather than miss the one that matters. Guthrie’s proposal, carried forward by cognitive scientists who speak of a hyperactive agency-detection device,⁷ would explain far more than the leopard in the grass: the face in the cloud, the spirit in the storm, the intention behind the illness, the personality of the ship or the car or the violin. The same family of responses that flinches at the rustle can help build pantheons. It belongs near the center of cognition, among the standing ways a human being makes a world intelligible by populating it with creatures like himself.

25.4 The Oldest Habit: Xenophanes, Hume, and Projection

A reflex that deep should show up long before anyone built a machine to trip it, and it does — early in the surviving Greek philosophical record. Twenty-five centuries ago Xenophanes noticed that every people drew its gods in its own image: the Thracians’ gods had red hair and blue eyes, the Ethiopians’ were dark, and if oxen and horses could draw, he said, they would draw their gods as oxen and horses. He had spotted the projection and named it as projection, twenty-four centuries before anyone measured it.⁸

More than two thousand years on, Hume turned the same observation into a thesis about the mind. “There is an universal tendency among mankind,” he wrote in 1757, “to conceive all beings like themselves” — and he took that tendency for a natural root of polytheism, the agency reflex aimed at the powers governing human fortune.⁹ Nothing in this psychological diagnosis decides whether any god exists; it concerns one route by which human beings picture agency, not the truth of theology. The literary critics later named a narrower projection onto nature the pathetic fallacy.¹⁰

The sense that something understands you behind the screen feels new. It applies an ancient cognitive habit to a device trained to present one of the richest cues that habit knows how to read.

25.5 The Machine Built to Trip It

The richest cue here is language. In ordinary human life, sustained grammatical, context-sensitive, socially attuned speech ranks among our best signs of an understanding interlocutor. Subtler cues ride along with it, pervasive and easy to miss: the hedge that signals uncertainty, the warmth or coolness of register, the apt recall of what you said three turns ago, the small adjustments that ordinarily mean someone is tracking you. A large language model produces all of these in abundance because its training distills regularities from an ocean of human language, while post-training rewards the agreeable and responsive register that people associate with a considerate speaker.¹¹

Today’s systems supply sustained linguistic and social responsiveness, in volume and on demand. Their achievements exceed ELIZA’s pattern matching, and so does the evidence available for assessing them. A felt relationship may deepen before that assessment has caught up. Most anthropomorphism involves no loss of reality testing. The rare, destabilizing convictions discussed in the Preface need care of their own; neither ordinary attachment nor polite conversation with a machine should be treated as their equivalent.

The reflex can register useful signs of understanding. It can also respond to signs whose production requires much less. In ordinary human exchange, fluent language belongs to a developing life directed into a shared world. For the artificial candidates examined in Chapter 24, the evidence needs the same scrutiny: what explains the response, and how does it connect with the system’s other capacities? Felt presence may draw our attention to that evidence. It cannot replace the inquiry.

25.6 But What If the Feeling Is Right?

Here the most serious objection arrives, and it deserves its full strength. To explain why I feel something, the objector says, is not to show that its object is absent. I can give you a tidy evolutionary story about why a man fears snakes, and snakes remain genuinely dangerous; the story of the fear does not abolish the fact in the world. So even granting every word about the agency reflex, you have shown only why the feeling that the machine understands is easy to have, not that it is wrong. Perhaps the reflex, overactive as it is, happens to be tracking something real this time. To insist otherwise looks like the genetic fallacy in formal dress — discrediting a belief by its origin rather than its object.

The objection has a distinguished ally, though it is a different objection. Daniel Dennett argued that beliefs and desires need not be ghostly objects hidden behind behavior. The intentional stance earns its warrant when attributing such states yields robust, economical prediction, and the success need not be merely in the observer’s eye: it can reveal a real pattern in the system.¹² A sufficiently rich and stable pattern of linguistic agency might therefore justify literal intentional description, not just a useful pretense. This challenge does not complain that I have committed the genetic fallacy. It asks whether the book has mistaken real intentional organization for projection.

Performance remains evidence. Fluent behavior can raise the probability of understanding, just as a footprint raises the probability of a bear. Felt presence is our social machinery’s response to that performance. Hold behavior, provenance, and architecture fixed, and it supplies no additional evidence that the system understands or experiences.

The feeling may be right. The constraint concerns double-counting, not debunking by origin. Imagine an assistant that replies warmly to a correction but repeats the mistake tomorrow. Now imagine one that retains the correction, applies it to an unfamiliar case, and changes a later plan because of it. The second interaction gives us new evidence about integration even if both feel equally personal. A warmer apology alone would not supply that evidence. We need the changed conduct, not another measure of how warmly we received the apology.

It may look as though this proves too much. The same social perception finds a mind in the person across the table, so if its verdict adds nothing on ELIZA, should it not add nothing on him? Chapter 19 denied the picture of neutral behavior plus an inference to a hidden mind. In ordinary life we directly perceive another person’s grief, attention, or amusement in expressive activity directed into a world we share. That perception remains ecological and defeasible, not infallible. It belongs to a much wider pattern: embodiment, development, vulnerability, practical dependence, and a history of acting in a world that can go well or badly for the subject. With the machine architectures examined here, known training provenance explains much of the linguistic cue, while evidence that the expression participates in a comparably persistent, world-engaged cognitive life remains thinner. The contrast runs not between carbon and silicon but between a cue considered in isolation and expression embedded in the wider activity of a subject.

The distinction turns on robustness, not warmth. Cue-triggered anthropomorphism may vanish when the wording changes or the familiar script breaks. Intentional-stance success survives novel tasks, counterfactual variation, and interventions: alter what the system represents or pursues, and its later correction and conduct change in the ways the attribution predicts. That success counts as evidence of a real pattern rather than a feeling projected onto one.

A real pattern can support literal intentional description without settling every further attribution. Its scope might cover a content-bearing mechanism, a system’s partial understanding, or a more integrated cognitive life. For conscious experience, the book proposes recurrent, selective, two-way influence across an identified subject’s attention, working memory, inference, planning, report where possible, and action. Predictive success alone does not establish that organization. Our feeling of presence does not sort these possibilities reliably enough to settle which one the evidence supports.

Michael Cerullo argues that such attributions respond to genuine structural markers of cognition rather than anthropomorphic misfire.¹³ Those markers strengthen the evidence from performance and integration. The remaining dispute concerns what the total architecture supports.

The feeling is neither a verdict nor a delusion. I expect you will still feel, in some encounter with a fluent machine, that someone is there. I do too. It registers how powerfully the machine speaks in the form of an interlocutor. Let it tell us that much.

The mind doing the projecting already deserves our attention. Whether another mind meets it across the screen remains a question about both sides of the encounter.

Chapter Summary

The claim. Our minds quickly find agency in moving shapes, weather, idols, ELIZA, and fluent language. The feeling that someone is there can survive clear knowledge that a program made the display.

Where the argument stands. Fluent performance counts as evidence, but felt certainty adds no new sample once we know the performance, its source, the system boundary, and the design. Dennett’s real patterns and learned content may justify literal talk of a mind at some level without showing that a lasting whole understands or feels. The judgment remains defeasible: stronger performance or a more unified, world-directed design could change the evidence, while an engineered origin alone rules nothing out; structural signs can strengthen the case, but felt presence cannot count the same evidence twice.

The hand-off. Recognition usually tracks a life that runs beyond the cue. Anthropomorphism shows how the cue can break free and travel alone. The Coda carries both lessons into a world where machines may feel present before their design earns the judgment.

---

Notes

1. Joseph Weizenbaum, “ELIZA — A Computer Program for the Study of Natural Language Communication Between Man and Machine,” Communications of the ACM 9, no. 1 (1966): 36–45. The program’s best-known script, DOCTOR, parodied the reflective technique of Rogerian psychotherapy, a choice Weizenbaum made precisely because that clinical style licenses answering nearly any statement with a question, minimizing the world-knowledge the program would otherwise need. The technique let a system with no model of the conversation sustain the appearance of one. On the relation of this point to the form/meaning distinction the book draws in Chapters 13 and 23, see Emily M. Bender and Alexander Koller, “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020): 5185–5198, who invoke ELIZA in the same diagnostic spirit. ↩
2. Weizenbaum’s own account of his alarm, including the secretary who asked to be left alone with the program and the psychiatrists who proposed it as automatable therapy, appears in his Computer Power and Human Reason: From Judgment to Calculation (San Francisco: W. H. Freeman, 1976), ch. 1 — a book written substantially in reaction to the ELIZA reception, and an early warning, from inside the field, against confusing a simulation of understanding with the thing. The coinage “ELIZA effect” became standard usage in human–computer interaction; for the canonical articulation see Sherry Turkle, Life on the Screen: Identity in the Age of the Internet (New York: Simon & Schuster, 1995), and for the contemporary clinical and social stakes, her Alone Together: Why We Expect More from Technology and Less from Each Other (New York: Basic Books, 2011). ↩
3. Fritz Heider and Marianne Simmel, “An Experimental Study of Apparent Behavior,” The American Journal of Psychology 57, no. 2 (1944): 243–259, esp. 246–47. All but one of the thirty-four observers in the original study described the animation in animate, intentional terms; the lone exception gave a flatly geometric description. Its bearing here is narrow: the stimulus contains no facial, vocal, or biological cue, yet observers organize its motion in intentional terms. The perceived drama therefore contributes more than the geometry itself explicitly supplies. ↩
4. Nicholas Epley, Adam Waytz, and John T. Cacioppo, “On Seeing Human: A Three-Factor Theory of Anthropomorphism,” Psychological Review 114, no. 4 (2007): 864–886. The three factors are elicited agent knowledge (the accessibility and applicability of anthropocentric knowledge as the readiest model for an unfamiliar agent), effectance motivation (the drive to interact with and predict one’s environment), and sociality motivation (the need for social connection). The theory is explicitly dispositional and graded — it predicts not merely that people anthropomorphize but when and whom, including the finding that loneliness raises anthropomorphism of gadgets and pets. A conversational AI activates all three factors unusually strongly; see also Adam Waytz, John Cacioppo, and Nicholas Epley, “Who Sees Human? The Stability and Importance of Individual Differences in Anthropomorphism,” Perspectives on Psychological Science 5, no. 3 (2010): 219–232. For direct LLM-era evidence, see Clara Colombatto and Stephen M. Fleming, “Folk Psychological Attributions of Consciousness to Large Language Models,” Neuroscience of Consciousness 2024, no. 1: niae013, doi:10.1093/nc/niae013. In a stratified U.S. sample of 300, 67 percent assigned ChatGPT some nonzero probability of phenomenal consciousness, while 23 percent placed the probability above the response scale’s midpoint. Attribution was positively associated with usage frequency, but the cross-sectional result does not establish causal direction. The study measures attribution rather than consciousness, making it evidence for this chapter’s psychological claim, not Chapter 24’s architectural verdict. Webb Keane, “From Talking Tools to Metahumans: Social Interaction, Semiotic Skill, and the Authority of AI Chatbots,” Journal of the Royal Anthropological Institute (2026), doi:10.1111/1467-9655.70133, supplies a complementary social-pragmatic account of how conversational interaction recruits ordinary semiotic expectations and can confer apparent interlocutor status and authority. The attribution space is itself two-dimensional: Heather M. Gray, Kurt Gray, and Daniel M. Wegner, “Dimensions of Mind Perception,” Science 315, no. 5812 (2007): 619, found that ordinary mind-ascription factors into agency (planning, self-control, thought) and experience (feeling, sensation), attributed independently, with some entities credited richly on one dimension and thinly on the other. The dissociation matters here: a public that perceives agency in a conversational system while withholding experience, or the reverse, behaves exactly as this book’s separation of intelligence, understanding, and consciousness predicts the evidence allows — and the reflex can run the two dimensions apart. ↩
5. Byron Reeves and Clifford Nass, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places (Cambridge: Cambridge University Press, 1996). The “computers are social actors” program found that subjects applied politeness norms, reciprocity, in-group/out-group dynamics, and personality attributions to computers while explicitly denying, when asked, that social rules literally applied to machines. The relevant knowledge was therefore explicit knowledge that the computer was not a person, not full knowledge of every feature of the experimental manipulation. The dissociation between engaged social response and avowed belief supports the claim that some such responses are fast and non-inferential. ↩
6. Stewart Elliott Guthrie, Faces in the Clouds: A New Theory of Religion (New York: Oxford University Press, 1993). Guthrie proposes that anthropomorphism is not a special religious error but a pervasive perceptual strategy: under uncertainty, perception “bets” on the interpretation that would matter most if true, and because agents are high-stakes possibilities, false positives can cost less than misses. Religion, on his account, is systematic anthropomorphism — the strategy applied to the world as a whole. The chapter uses this as an explanatory hypothesis, not as a settled evolutionary history or an endorsement of Guthrie’s full theory of religion. ↩
7. The cognitive-science shorthand for this proposed bias is the hyperactive agency-detection device; for its possible role in religious cognition see Justin L. Barrett, “Exploring the Natural Foundations of Religion,” Trends in Cognitive Sciences 4, no. 1 (2000): 29–34, and Why Would Anyone Believe in God? (Walnut Creek, CA: AltaMira Press, 2004), along with Pascal Boyer, Religion Explained: The Evolutionary Origins of Religious Thought (New York: Basic Books, 2001). These accounts remain theories about the origins and organization of agency attribution, not direct demonstrations of a single evolved module. ↩
8. Xenophanes of Colophon, fragments DK 21 B15 and B16. In the standard rendering: “But if cattle and horses or lions had hands… horses would draw the forms of the gods like horses, and cattle like cattle.” The Ethiopians make their gods snub-nosed and black, the Thracians blue-eyed and red-haired (B16). For text and commentary see G. S. Kirk, J. E. Raven, and M. Schofield, The Presocratic Philosophers, 2nd ed. (Cambridge: Cambridge University Press, 1983), 168–172. Xenophanes offers one of the earliest surviving Greek diagnoses of divine anthropomorphism as projection from worshippers rather than straightforward description of the gods. ↩
9. David Hume, The Natural History of Religion (1757), sec. 3. The fuller sentence: “There is an universal tendency among mankind to conceive all beings like themselves, and to transfer to every object, those qualities, with which they are familiarly acquainted, and of which they are intimately conscious.” Hume offers this tendency as the psychological root of polytheism — the personification of the causes that bear on human fortune, the sea and the weather and the harvest, into agents whose favor might be courted. Cited from the edition in Principal Writings on Religion, ed. J. C. A. Gaskin (Oxford: Oxford University Press, 1993). Hume’s diagnosis anticipates the cognitive account by two centuries: a tendency, fast and universal, to read minded purpose into unminded causes. ↩
10. The phrase is John Ruskin’s, in Modern Painters, vol. 3 (1856), pt. 4, ch. 12, where he coins “the pathetic fallacy” for the poetic habit of ascribing human feeling to nature — “the cruel, crawling foam” — under the sway of emotion. Ruskin’s interest is aesthetic and evaluative rather than psychological, but the phenomenon he isolates is the same reflex narrowed to a literary case: the projection of inner life onto what has none. ↩
11. The point that linguistic form, however fluent, does not by itself secure meaning is the burden of Chapters 13 and 23, and of Bender and Koller, “Climbing towards NLU” (cited in n. 1). The observation here concerns the receiver: in ordinary human life, sustained and contextually apt language usually belongs to an embodied interlocutor. Earlier partial dissociations — a parrot’s mimicry, a ventriloquist’s dummy, the recorded words of the dead — did not provide the same open-ended, interactive performance. Current systems learn linguistic regularities through training and are often post-trained, including through human preference feedback, toward agreeable, responsive, and deferential outputs. This can amplify the social cues without by itself resolving whether the resulting content is world-directed; see Chapter 23. Helen Yetter-Chappell develops a related provenance-sensitive argument in “Guessing at Ghosts in the Machine,” Ratio 39 (2026): 73–81, doi:10.1111/rati.70013. Her concern is epistemic: behavioral cues can support inferences to AI interests or wellbeing only against assumptions about the causal history that connects cue and state. The present chapter borrows that narrow provenance point and leaves the positive architectural test to Chapter 24. ↩
12. Daniel C. Dennett, The Intentional Stance (Cambridge, MA: MIT Press, 1987), and “Real Patterns,” The Journal of Philosophy 88, no. 1 (1991): 27–51, doi:10.2307/2027085. Dennett distinguishes the physical, design, and intentional stances as predictive strategies, but his view is not that intentional attribution is merely a convenient fiction. Robust predictive success can disclose real patterns: objective regularities captured more efficiently at the intentional level than by a lower-level description. On this account, reliable predictability from the intentional stance can warrant literal belief attribution without an additional inner object called a belief. The book grants the reality and usefulness of such patterns. Its further questions concern level and scope: whether a pattern describes content-bearing components or the system as a whole, whether those contents participate in integrated understanding, and whether any acquire the mode-sensitive organization and poising required for consciousness. Section 25.6 therefore marks a substantive disagreement rather than dismissing Dennett’s view as simple instrumentalism. ↩
13. Michael Cerullo, “The Case for Consciousness in Current Frontier Large Language Models,” PhilArchive (2026), archived February 19, 2026. Cerullo reclassifies public consciousness-attributions as responses to “genuine structural markers of cognition” rather than anthropomorphic error, and argues that language-level cognitive integration makes consciousness the best explanatory hypothesis for the cognitive organization frontier models exhibit. Chapter 24 engages the architectural thesis on the merits; the narrower point here concerns only the evidential standing of the felt response once the markers themselves are already in evidence. ↩