Mind, Matter, and Meaning

Chapter 20: The Chinese Room

Why syntax never yields semantics — formal symbol manipulation, at any scale or speed, is not understanding

What this chapter does. Everything this book has to say about machines turns on one claim: shuffling symbols by their shapes, however fast and however well, never adds up to meaning. The Chinese Room is the famous vehicle for it, but the argument was never really about rooms or computers — it is about syntax and semantics, and it survives its standard rebuttals largely because so many people misremember what it actually claimed. This chapter states the argument at full strength, answers the four canonical replies in order, marks the deeper worry that the next chapter takes up, and then looks carefully at what large language models actually do. By the end you should be able to say exactly what the Chinese Room shows, what it does not show, and why fluency is not understanding.

The chapter argues in five steps:

  1. The argument in its strongest form (§I). Syntax is not sufficient for semantics — the claim, stripped to one sentence.
  2. Four replies, each at full strength (§II). The systems reply, the robot reply, the brain-simulator reply, and the other-minds reply — named, numbered, and answered.
  3. A deeper worry, deferred (§III). Behind the syntax-semantics gap lies a more radical question — whether computation is even an intrinsic fact about a physical system — taken up in full in Chapter 21.
  4. What large language models actually do (§IV). Next-token prediction described accurately — neither dismissed as “autocomplete” nor inflated into comprehension.
  5. What the Chinese Room does not show (§V). The careful limit: it refutes sufficiency, not the possibility of machine understanding — which Chapter 24 takes up.

Let me describe a system to you. It receives inputs in Chinese, applies an elaborate set of rules for manipulating Chinese symbols, and produces outputs in Chinese. To a native Chinese speaker, the outputs look indistinguishable from those of someone who understands Chinese — the answers are correct, the conversation flows, the responses are contextually apt. From the outside, the system passes.

Now I’ll tell you what’s inside. There is a person — call him the operator — sitting in a room with a large collection of buckets containing Chinese symbols and a rulebook written in English. When a string of Chinese symbols comes in through the slot in the door, the operator consults the rulebook, which tells him which symbols from the buckets to combine and send back out. He follows the rules meticulously. He produces outputs that pass every behavioral test. And he understands not one word of Chinese.

This is John Searle’s Chinese Room, and in its original 1980 form it generated one of the most sustained debates in the philosophy of mind.1

The argument has a complicated reception history. Many people, on first hearing it, find it convincing. Then they encounter a philosophy seminar or a popular AI article, absorb the standard objections, and conclude that Searle must have missed something — that the systems reply, or the robot reply, or some other clever response disposes of it. That trajectory, while understandable, is a mistake. The core of the argument survives the standard objections, because it is not primarily an argument about computation. It is an argument about the relationship between syntax and semantics.

§I. The Argument in Its Strongest Form

Searle’s core claim: syntax is not sufficient for semantics. No amount of symbol manipulation, however sophisticated, generates genuine meaning. The claim has two parts. The negative one: pure symbol manipulation doesn’t yield understanding. The positive one: understanding requires the right kind of causal connection to the world.

The Chinese Room demonstrates the negative claim. The operator manipulates symbols correctly and produces correct outputs, and he does not understand Chinese. Nothing in the room understands Chinese — not the operator, not the rulebook, not the buckets of symbols. Understanding Chinese requires knowing what the symbols mean, and meaning is not a property of symbol shapes or of the rules for transforming them. It is a property of the relationship between the symbols and what they are about.

The positive claim connects to everything developed in the previous chapters. Genuine understanding requires semantic grounding — the right kind of causal engagement with the world the symbols represent. Chapters 14 and 15 showed that representation requires more than information-carrying and that meaning extends outward through embodied causal history. The Chinese Room makes those abstract claims vivid: here is a system that does everything a formal symbol manipulator can do and conspicuously lacks understanding, for the same reason the thermostat lacks it — no genuine connection to what the symbols are about.

Name the missing thing and the whole book is standing in the room with you. What the operator lacks — what no rulebook can supply — is aboutness: the directedness toward the world that Brentano put at the mark of the mental in Chapter 1, that Part Two found in seeing and Part Three grounded in a body’s dealings with its surroundings. Syntax is the shuffling of shapes; semantics is shapes that are about something; and the Chinese Room is one more place the thread runs — this time marked by its absence.

§II. Four Replies, Each in Its Strongest Form

Searle’s 1980 paper appeared with a substantial set of peer commentaries, and the standard objections were laid out then. They have remained the standard objections ever since. Four deserve direct engagement.

§II.1. The Systems Reply

The objection: of course the operator doesn’t understand Chinese. But the system as a whole — operator plus rulebook plus buckets plus room — might. You are pointing at the wrong level of organization.

This has enough surface plausibility to have kept the debate alive for forty years. Searle’s response is incisive: imagine the operator memorizes the entire rulebook and carries it all in his head. Now there is no room, no rulebook, no buckets — just a person with the rules internalized. He walks out into the world, answers questions in Chinese, passes every test. Does he now understand Chinese?

No. He’s still just manipulating symbols according to rules. The fact that they are now “in his head” rather than in a room creates no understanding. What matters is not the organizational level at which the symbols are processed but what connects the processing to meaning — and symbol manipulation at any level doesn’t make that connection. The systems reply confuses the location of the processing with its nature. It assumes that scaling up or relocating the system makes something new emerge; but understanding doesn’t emerge from symbol manipulation by virtue of scale or location. It requires the right kind of semantic grounding, and that is precisely what the formal system lacks at every level.2

§II.2. The Robot Reply

The objection: put the Chinese Room inside a robot. Give it cameras for eyes, microphones for ears, motors for limbs. Now the symbol manipulation is hooked up to the world through perceptual and motor causal chains. Now the system understands.

This reply concedes more than its proponents notice. It concedes that pure symbol manipulation does not suffice — that something more is needed, namely causal contact with the world. That concession is most of Searle’s point. The remaining question is whether bolting cameras and motors onto a Chinese Room delivers the right kind of causal contact.

It does not, on Searle’s analysis, for the same reason the original room does not. The operator inside the robot still receives only symbols. Whether they originated from a camera or from a slot in the door makes no difference to him, who has no way to tell. The symbols come in; the rules are applied; symbols go out. That they correlate with the robot’s environment gives the operator no grasp of what they refer to. Embodiment in the relevant sense — the embodiment Chapter 17 developed — requires that the system’s processing be constitutively shaped by environmental engagement, not merely receive symbolic reports of it. A robot whose only access to the world runs through a Chinese Room has no such shaping; it just has more elaborate input.3

The robot reply is also revealing about what it would take to deliver genuine understanding. The system must be the kind of thing whose internal structure has been formed by, and continues to be answerable to, an environment it acts within. Bolting peripherals onto a symbol-shuffler does not produce that. Chapter 24 takes up the question of what would.

§II.3. The Brain Simulator Reply

The objection: imagine the room contains not a rulebook but a complete simulation of a Chinese speaker’s brain — every neuron, every synapse, every firing pattern. The operator follows rules that mirror exactly the causal structure of an actual Chinese-speaking brain. Surely that simulation understands Chinese.

This presses on a real intuition: if neural activity grounds understanding, and the simulation reproduces neural activity, the simulation should ground understanding too. But the response brings out Searle’s deeper point. The simulation reproduces the formal structure of the neural activity — the pattern of which symbol gets manipulated when. It does not reproduce the causal-physical events themselves. A simulation of digestion does not digest; a simulation of combustion does not burn. The formal pattern of neural firing, manipulated symbolically, is not neural firing — it is a description of neural firing being processed by a different physical substrate.

This is the point the 1990 companion argument extends, and it opens onto two distinct further questions taken up by the next two chapters: whether running a program is even an intrinsic fact about a system (Chapter 21), and whether simulating a process — even perfectly — ever amounts to performing it (Chapter 22). The brain simulator reply mistakes the observer-relative description for the intrinsic process being described, and a model of a causal process for an instance of it.

§II.4. The Other Minds Reply

The objection: how do you know any other person understands Chinese? You judge from behavior. The Chinese Room produces the right behavior. By your own standards, you should attribute understanding to the room.

This is the most rhetorically powerful of the standard replies, and the most quickly diagnosed. The point about other minds is epistemological — it concerns how we know whether something understands. The Chinese Room argument is metaphysical — it concerns what understanding consists in. Conflating the two confuses our justification for attributing a mental state with what makes the attribution correct.

Searle’s argument does not deny that we attribute understanding to other people on the basis of behavior. It denies that behavior is what understanding is. We attribute it to other people partly because we have independent grounds — biological similarity, evolutionary history, causal-environmental embedding — for thinking they have the kind of system in which understanding can occur. The Chinese Room lacks those grounds. Behavior alone, in either case, was never what made the attribution correct.

§III. A Deeper Worry, Deferred

Behind the Chinese Room sits a question more radical than whether syntax suffices for semantics: whether a physical system is running a program at all, except relative to someone’s interpretation. Searle pressed it in a 1990 companion paper — computation, he argued, is observer-relative; syntax is not intrinsic to physics; whether a lump of matter counts as implementing one program, another, or none is something an observer reads onto it, not a fact found in it.4 If that holds, then “the brain is a digital computer” was never the kind of fact a mind could be built on, whatever the Chinese Room shows about symbols. The point is large enough to need its own chapter, and it gets one: Chapter 21 takes up observer-relativity and the triviality results (Putnam, Searle, Chalmers) in full, and Chapter 22 the companion question of whether simulating a process ever amounts to performing it. Here it is enough to mark that the syntax-semantics gap is not the only thing wrong with the computational picture of mind — only the most famous.

§IV. What Large Language Models Actually Do

Contemporary large language models are, at the level of their fundamental architecture, sophisticated pattern-completion systems. They are trained on enormous quantities of text to predict the next token in a sequence — to learn, with extraordinary precision, the statistical regularities in how human beings use language. Their responses are then refined through feedback to align with human preferences.

The results are remarkable. The models produce fluent, contextually sensitive, topically appropriate text across an enormous range of domains. They explain scientific concepts, discuss philosophy, write code, carry on extended conversations, and adapt to context in ways that repeatedly surprise even experienced users.

None of this constitutes understanding in the sense at issue in this book.

The models do not stand in the right kind of causal relationship to what their words refer to. “Water” in a language model’s vocabulary refers, in the relevant sense, to a pattern in text — to the statistical context in which the token “water” tends to appear. This is a real kind of knowledge: knowledge of how language works, of what typically follows what, of the textual neighborhoods in which concepts reside. It is not the same as knowing what water is to someone who has been thirsty, who has been caught in rain, who has watched a river and felt the force of it.

This is Searle’s point, updated for the current technology. The language model is an extraordinarily sophisticated Chinese Room. Where the original room had a person with a rulebook, the model has neural network weights encoding statistical regularities. The architecture differs; the limitation is the same. Formal processing of linguistic symbols, however sophisticated, does not generate genuine semantic content. The connection to the world runs through text about the world, not through the world itself. A book about swimming encodes information produced by swimmers. Reading the book doesn’t make you a swimmer. The language model has read an enormous library. It has not thereby swum.5

Whether it might nonetheless inherit an indirect grip on the world through the statistical residue of all that writing — and whether the multimodal, tool-using, feedback-trained systems on our desks escape the confinement to form at all — is the live dispute, and it has earned its own chapter. Chapter 23 takes the live-systems case up in full: the borrowed-meaning diagnosis, Harnad’s symbol-grounding problem, the multimodal rejoinder, and Bender and Koller’s eavesdropping octopus — the Chinese Room recast for the age of the language model. Here the 1980 point is enough: a system trained on form, at any scale, does not thereby acquire the world the form is about.

§V. What the Chinese Room Does Not Show

Having argued that the Chinese Room argument is substantially correct, I want to be careful about what it does not establish.

It does not show that machines cannot think. It shows that formal symbol manipulation — computation in the syntactic, substrate-neutral sense — is not sufficient for thinking. Whether there could be a machine that thinks by virtue of having the right kind of causal, embodied engagement with the world is a separate question, and nothing in Searle’s argument settles it negatively. Chapter 24 takes that question up.

It does not show that language models are useless or their outputs worthless. Their outputs often encapsulate human understanding in genuinely valuable ways — the understanding is in the training data, and the model distributes it. What the argument shows is that this encapsulation should not be mistaken for the model’s own understanding.

And it does not show that the hard questions about AI and consciousness are settled. They are not. What the Chinese Room provides is a diagnostic: when we feel the pull of attributing understanding to a language model, we should ask whether we are responding to the content and fluency of the output (which can be very high) or to evidence of genuine semantic grounding (which, on current architectures, is absent). These are different things, and keeping them distinct is the beginning of clear thinking about artificial minds. The next three chapters sharpen the diagnostic by answering the three questions it raises — observer-relativity (Chapter 21), simulation versus realization (Chapter 22), and whether mastery of linguistic form ever reaches the world (Chapter 23) — before Chapter 24 turns the picture over and asks what genuine artificial understanding would actually require.

Chapter Summary

The Chinese Room defends one claim: syntax is not sufficient for semantics, because understanding consists in semantic grounding — the right causal engagement with what the symbols represent — and the four standard replies each miss this. Today’s large language models are exactly such formal systems, their outputs valuable but their understanding absent. The chapter hands observer-relativity to Chapter 21 and the octopus to Chapter 23, and leaves open the constructive possibility Chapter 24 pursues.

Notes

  1. John Searle, “Minds, Brains, and Programs,” Behavioral and Brain Sciences 3 (1980): 417–424. The paper was published with a substantial collection of peer commentaries and Searle’s replies, making it one of the most thoroughly debated papers in the philosophy of mind. The standard objections — the systems reply, the robot reply, the brain simulator reply, the other minds reply — are all addressed in the original exchange. For a useful overview of the subsequent debate, see Ned Block, “The Mind as the Software of the Brain,” in Thinking: An Invitation to Cognitive Science, vol. 3, ed. Daniel Osherson (Cambridge, MA: MIT Press, 1995). ↩
  2. Searle’s response to the systems reply is in his “Minds, Brains, and Programs,” 419–420. The key move — asking us to imagine the operator internalizing the rules — is sometimes dismissed on the grounds that internalization changes the relevant organizational facts. But Searle’s point is that the internalized version is stipulatively identical in computational terms to the externalized version; if organizational facts at the systems level are what matter, they should be equally present when internalized. That the internalized version equally fails to understand Chinese shows that organizational level is not what is missing. ↩
  3. Searle’s response to the robot reply appears in “Minds, Brains, and Programs,” 420. The connection between the robot reply and the embodiment requirement developed in Chapter 17 is mine, not Searle’s; Searle himself rests on the narrower point that perceptual symbols remain symbols. The richer notion of embodiment as constitutive shaping by environmental engagement comes from the embodied cognition tradition — Varela, Thompson, and Rosch’s The Embodied Mind (Cambridge, MA: MIT Press, 1991) provides the canonical statement, and Andy Clark’s work on extended cognition develops complementary themes. The two perspectives converge on the failure of the robot reply: a robot driven by an internal Chinese Room has the wrong kind of relationship to its environment, regardless of what peripherals it carries. ↩
  4. John Searle, “Is the Brain a Digital Computer?”, Proceedings of the American Philosophical Association 64 (1990): 21–37. The argument that computation is observer-relative while the brain’s causal processes are intrinsic is the basis of Searle’s claim that “the brain is a causal machine and not a computational machine.” This is a strong claim and has been contested; for a critical response see Daniel Dennett, “Granny’s Campaign for Safe Science,” in The Intentional Stance (Cambridge, MA: MIT Press, 1987), and for a defense of the observer-relativity claim see Searle, The Rediscovery of the Mind (Cambridge, MA: MIT Press, 1992), ch. 11. Chapter 21 takes up the argument, and the triviality results it connects to, in full. ↩
  5. The specific point about language models inheriting the residue of human understanding without acquiring that understanding themselves connects to Luciano Floridi’s work on the distinction between information and intelligence, and to discussions in the philosophy of language about the difference between testimonial knowledge and first-person understanding. For recent philosophical work directly on LLMs and intentionality, see the growing literature on whether statistical language models achieve genuine semantic content; representative positions include Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜,” Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 610–623 — the source of the “stochastic parrot” characterization of an LM as “a system for haphazardly stitching together sequences of linguistic forms… according to probabilistic information about how they combine, but without any reference to meaning” (617) — and the more sympathetic assessment in Murray Shanahan, Talking About Large Language Models (2023). ↩
← Chapter 19: Functionalism
Contents
Chapter 21: Observer-Relativity →