This is a first, rough draft of what is intended to be the second chapter of a work-in-progress on consciousness.
This chapter touches on themes that have also been discussed with GPT4:
If you prefer the less efficient but more interactive style of a discussion with GPT4, check the links above.
Otherwise read on.
Comments and feedback are welcome. I am already aware it needs more rewriting to be accessible to a more general audience. That’s on my to-do list. In the meantime, please let me know if something is unclear, preferably by posting comments to the first maralia post.
The chapter assumes a basic knowledge of The Hard Problem, physicalism, and qualia. If these notions are unfamiliar to you, start with the preceding chapter.
In view of the complexity of the issues involved, I have split this chapter into two, and another qualia chapter will be posted soon. This installment primarily deals with flaws in what is known as the Knowledge Argument, also known as the story of Mary the Colour Scientist. It only provides a brief survey of the major flaws in the argument; it’s not an exhaustive account of all the issues Mary raises, which will require a whole book to explore. (A later chapter will offer a preliminary definition of qualia, via the notion of maralia, and it will consider some of the related philosophical issues, such as the problem of epiphenomenalism, and the interaction between epistemology and ontology. Don’t worry if these terms are currently unfamiliar.)
Not so simply red
Most serious discussions about consciousness eventually lead to Mary the Colour Scientist and then follow her through the laboratory door that divides objective neuroscience from lived experience.
She is the protagonist of a famous thought experiment, often known as the Knowledge Argument, which was created by the philosopher Frank Jackson (1982, 1986).
Mary is a brilliant scientist who is, for whatever reason, forced to investigate the world from a black and white room via a black and white television monitor. She specializes in the neurophysiology of vision and acquires, let us suppose, all the physical information there is to obtain about what goes on when we see ripe tomatoes, or the sky, and use terms like “red”, “blue”, and so on. She discovers, for example, just which wavelength combinations from the sky stimulate the retina, and exactly how this produces via the central nervous system the contraction of the vocal cords and expulsion of air from the lungs that results in the uttering of the sentence “The sky is blue”. … What will happen when Mary is released from her black and white room or is given a color television monitor? Will she learn anything or not? It seems just obvious that she will learn something about the world and our visual experience of it. But then it is inescapable that her previous knowledge was incomplete. But she had all the physical information. Ergo there is more to have than that, and Physicalism is false. (Jackson, 1982)
The intuitive force of this argument can’t be denied: first everything is black-and-white-and-gray and then, lo and behold, Mary steps into the world of colour, seeing the inadequacy of the physicalist world view. The door of her laboratory provides a handy metaphor for the brain-mind divide, and her discovery of real colour on her release is one of the starkest possible demonstrations of what might be called the Qualia Problem.
We could quibble about the set-up – she couldn’t really know everything involved in colour vision, and her colour-processing abilities would probably be irretrievably damaged by her isolation – but I think Jackson’s core intuition here is essentially correct. His argument exposes our innate belief that objective, physicalist accounts of reality seem totally incapable of capturing colours. By extension, we can expect physicalist neuroscience to fail just as profoundly for most other qualitative aspects of experience: pains and sounds and scents and tastes. If physicalism tries to meet the challenge implied by Mary’s story – if it tries to derive qualia from physical reality – then I believe it is destined to fail, and it is fair to ask physicalists to account for this failure.
For all that, I also think the Knowledge Argument is very weak, and Jackson’s logic falls apart as soon as it is examined more closely. In this chapter, I intend to sketch some of the major flaws in his reasoning, introducing themes that will be explored in more detail throughout this book. Later chapters will reconsider Mary’s situation much more carefully.
To some extent, this pre-emptive rebuttal is necessary because I suspect that, for many readers, it will be difficult to consider any physicalist theory of consciousness while the impossibility of deriving qualia is lurking in the background as an ever-present distraction. But the Knowledge Argument is not Mary’s main point of interest, and not my main reason for introducing her so early in this book. I believe that a rich understanding of her situation is a useful stepping stone on the path to understanding consciousness. Jackson’s argument fails – and I believe it fails badly – but he should be congratulated for providing as with a very clear metaphor to work with as we explore issues of fundamental importance. In particular, Mary’s journey provides us with a way of operationalising the definition of qualia. To a first approximation, we can think of qualia as the component of perceptual experiences that seem to be left out of a standard physicalist account of the brain, as might be found in a neuroscience textbook. Her story forces us to confront the question of how the physical world could come to have entities that seem to defy scientific explanation – but it also provides us with the tools for finding the answer. It might not be the answer we wanted, or expected, but that’s okay. Understanding her plight will help us set realistic limits on what sort of explanations we can expect from a completed theory of consciousness.
So what’s wrong with the argument? If colour experiences and other mental phenomena are ultimately physical, how could Mary know everything about them – and then learn more? It seems like a contradiction. If we assume (for now) that she learns something on her release, surely her capacity for learning implies there must have been something wrong with her original information… What’s more, the nature of the missing ingredient seems obvious. Prior to her release, she has a bland account of the physics of photons outside the head and another bland conception of the computational activities of neural circuitry inside the head – and in both settings, everything is all black and white and spatial. After her release, she has a vivid experience of colours. She learns what red looks like. (And blue, and yellow, and so on, but usually the discussion circles around redness.)
The obvious conclusion is that the neuroscientific story in her black-and-white textbooks has failed to account for the vivid redness of red.
One reason this situation is disconcerting is that we have expectations of what a successful scientific theory should deliver. We have come to expect that a thorough scientific analysis of an entity should yield all the valid insights that are available. If someone presented a theory of black holes, claiming it was a comprehensive account, and we discovered that an entire class of black-hole phenomena had escaped analysis, we would justifiably conclude that there was something wrong with the theory. We would declare that it was not comprehensive after all. And we would expect that, once the theory was fixed, the relevant phenomena would no longer evade analytical capture, because a complete theory should account for all relevant phenomena. That’s how science works.
Jackson’s main point is that, when it comes to qualia, this standard scientific expectation seems destined to remain frustrated. (The same point had been made earlier, by Levine in 1983, and by others.) And in Mary’s case, the phenomenon that gets left out of her textbook account of redness is not an obscure, marginal one but the very essence of what she’s trying to understand.
Some physicalists might counter that any judgement about what Mary might learn in her laboratory is premature. Obviously, we can’t reliably know everything that future neuroscience might discover, so it is somewhat hazardous to speculate on where complete knowledge in this field might lead us. Perhaps, one might think, a future neuroscience discovery will fill this gap, allowing Mary to derive redness after all. I don’t think so. For neuroanatomical reasons that I’ll discuss below, I think Jackson is almost certainly right in assuming that we’ll never be able to get from neural circuit diagrams to subjective redness no matter how much neuroscience advances. Or rather, redness will not be derived in the way Mary attempts it in the story as told: by reading about neuroscience and thinking very hard. If we relaxed the constraints of the thought experiment and allowed Mary to employ futuristic brain-manipulating technology within her laboratory, she might indeed be able to produce a redness experiences in a black-and-white world, but this sort of neural-engineering approach wouldn’t resemble the normal process of understanding a theory.
This, then, is the very essence of qualia: that they are unreachable through objective scientific analysis. If Mary could sit in her laboratory and derive redness from first principles, then there wouldn’t be much of a mystery, and there would barely be a need for the term “qualia”. If she could derive redness from a computational approach, we could notionally look over her shoulder and follow that derivation. The relation of subjective redness to its neural substrate would finally make sense to us. Perhaps we could even take whatever principles we had just used to derive redness in humans and apply those principles to AIs, pinning down whether AIs experienced colour in anything like the way we do. Analogously, we might be able to work out whether a human – or an animal, or an AI – was really in pain. And so on for other sensations. If only we could derive all of these things from textbooks, we could justifiably claim that we understood them. And if we could reach this understanding from a mere description of the physical brain, without invoking extras, we would know that special non-physical ingredients were not needed to account for qualia. There might be residual difficulties explaining awareness itself (consciousness-the-container), but the mystery of qualia would be tamed. The contents of the human mind would be on a sound scientific footing.
Instead, as Jackson notes, this sort of direct scientific attack on qualia seems hopeless. Intuition suggests that no insights gained from studying neural circuit diagrams in a black-and-white textbook are likely to come anywhere near the essential nature of redness, or pain, or any other qualia. Scientific accounts of these types of experiences seem destined to leave out their very nature. But if we can’t derive those things from textbooks, how can we ever claim to understand them? How can we argue that subjective redness is merely a property of the physical brain when the set of properties derivable from physical accounts of the brain seems to exclude redness?
Following the original publication of Jackson’s argument, Mary’s situation has been thrashed to death in countless first-year university papers and extensively debated in the literature and online, to the point that mere mention of Mary can elicit a groan from seasoned debaters in this field, who are long past being ready to move on.
Only, we can’t move on, because opinions about Mary are as divided as ever.
And I don’t think we should move on, because we can be much more specific about what it is that Mary gains on her release, and why, and the issues turn out to be rather important. There is more loot to be dug up from this well-trampled battlefield.
The early consensus among physicalists was that the argument slides between various sloppy definitions of knowing, and information, and facts, proving nothing. Mary learns, but only by gaining a new format for information that was already available to her. Her frustration in trying to derive subjective colour was not because she lacked information, but because she lacked a particular mode of access to that information. I agree with this diagnosis, and I’ll consider the relevant formatting issues below, but it is helpful to consider the overall context, first. It would be wrong to think of Mary’s frustration as analogous to an everyday file-formatting issue – like having a jpg file when you need a png file, or having a doc file when you need a txt file. That sort of formatting barrier would be arbitrary, and it would be just the sort of obstruction that we would expect an expert armed with full information and an adequate theory to be able to work around. The issue runs much deeper than that, and computational rearrangement of what she knows would not help her.
Some physicalists rejected Jackson’s conclusion because it would never be possible for Mary to learn everything there is to know about the physical basis of colour perception. The brain does not have the cognitive capacity to model a significant proportion of itself, so the entire argument is built on an implausible premise. That’s a fair complaint. A human can typically hold seven to ten things in working memory, and Mary’s complete understanding of the physical basis of redness would require her to track billions of neurons and all of their connections. There is a quantitative mismatch between what she must know to follow the physical story and the cognitive machinery available for understanding that story; within that vast mismatch, there is ample scope for Mary to fail to derive redness without that failure representing any flaw in physicalism. But I don’t think this is the most interesting point of failure in Jackson’s argument. We can gain greater understanding by overlooking this implausibility in the original scenario and granting that Jackson’s original intuition was qualitatively correct. No amount of study in a black-and-white environment would enable Mary to entertain the concept of redness in anything like the same way that she can understand redness when she finally experiences it for herself. Furthermore, what she is missing, prior to her release, is a concept of colour that seems very simple and elemental. Redness doesn’t seem like something that is too complicated for us to understand; it seems like something we can easily conjure up in our own minds. That redness is the way it is seems like a very simple, elementary fact – something we learned in kindergarten, or earlier. And it doesn’t seem as though Mary’s pre-release conception of redness gets part of the way towards knowing what red looks like, but then falters because she runs out of cognitive space; her understanding of redness never really gets started – it never heads in the right direction at all. Black-and-white neural diagrams are simply not red. Not even a tiny bit.
Could this be a case of a quantitative mismatch being so vast it seems like a qualitative one? Well, yes. I think that is a large part of the issue – but still not the most important part.
A more sophisticated response to Jackson’s argument has been to point out – in addition to the flaws already mentioned – that it involves a basic conceptual error, loosely related to what a linguist (or a programmer) might call a use-mention conflation.
For those of you who are not familiar with this concept, consider the difference between these two statements, which differ in their use of quotation marks: “The word on the sign was green,” and “The word on the sign was … ‘green’ ”.
In the first case, the word “green” is actually being used; its meaning is active within the sentence, and so we know the colour of the word on the sign – we just don’t know what the word says. In the second case, the word is being mentioned; its meaning has been de-activated by the quotation marks to show that the word on the sign is the one spelled “g-r-e-e-n” – but now we have no information about the word’s colour.
A use-mention conflation consists of failing to keep track of whether a representation has been used or mentioned. Someone listening to this chapter in an audiobook format, for instance, might suffer from use-mention confusion if they merely heard the green-word sentence and did not have the punctuation conveyed with a suitable pause.
Or take this sentence: “His answer was ‘monosyllabic’”. How many syllables were in his answer? One or five? To answer, we need to know whether the word “monosyllabic” is being used or mentioned.
Several common programming traps are of this nature. Consider the difference between assigning the numerical value of 10 to an integer variable and assigning the alphanumeric sequence “10” to a string variable. If Z is the array [10, 1, 2], then the instruction Z.sort() can produce different results in different programming languages according to whether those languages default to treating the digits as being used with their numerical meanings intact (producing the result [1, 2, 10]) or mentioned as alphanumeric strings (producing [1, 10, 2]). The difference becomes important for such simple tasks as listing files, where it is common to see “file10” ranked between “file1” and “file2”, because the digits in the filename are treated as though they have been mentioned, which strips the digits of their numerical significance.
These issues might seem obscure, but there is an analogous risk of use-mention conflation whenever we think about qualia (or any other aspect of mentality), and much of the confusion in this philosophical field relates directly to confusion about when brain states are being mentioned and when they are being used. For instance, the question that opened the previous chapter, “What is a thought bubble made of?” straddles two different interpretations, depending on whether the cartoonist’s thought bubble is envisaged in a mention capacity or a use capacity. Do we accept the meaning as being intact, or do we concentrate on the vehicle of representation?
Take the seemingly innocuous thought, “The brain state of imagining redness is like this.”
As you entertain this thought, you probably imagine redness.
Now, I’m aware that not every reader can simply imagine redness on demand. Not only are some people colour-blind, but not everyone with normal colour vision can imagine colours with the same ease. Aphantasia, the inability to form vivid mental images, is much more common than many people realise. Abilities in this domain lie on a spectrum. Some people with aphantasia don’t realise that other people can effectively produce internal mental paintings whenever they want. But hopefully you can at least relate to the idea of imagining redness and conjuring up a vivid example in your mind’s eye.
Let’s assume, for the sake of the argument, that you can imagine vivid redness whenever you want. So, there you are, thinking: “The brain state of imagining redness is like this.”
Is the neural representation of redness being mentioned, here, or being used? Are you activating a brain state and saying that imagining redness is like this example of a brain state? Such a thought is mentioning the neural state that usually means “red” within your cognition; it is like a form of quotation, and it doesn’t attribute a colour to that brain state. In that sense, it is like the sentence “The word on the sign was ‘green’ ”, which does not attribute a colour to any word on the sign, despite being about a word that is in turn about colour.
Conversely, are you entertaining the thought that your brain state is red, like this redness I’m imagining right now? Such a thought would be using the redness of the activated concept.
These two approaches might feel the same, subjectively, and it is quite possible you have never paused to differentiate the two, but they have radically different meanings.
If the thought being entertained is that the brain state of imagining redness is like an example of such a brain state, then this thought is necessarily true, but it doesn’t mean much because it is close to being tautological: all brain states are like themselves. If the thought being entertained is that the brain state has the property of being red (in some mysterious and poorly defined way that avoids simple falsehood), then this is a controversial conjecture for which there is no evidence.
Some philosophers might argue that some brain states do involve the property of being red in some inscrutable non-physical domain; that’s the whole point of the idea of phenomenal redness, it’s what Jackson set out to prove, and it’s a large part of what anti-physicalists mean when they talk of the redness quale. I won’t dissect this view just yet, though I will explore it at length throughout this book. My point, for now, is that we need to know when we are making this claim, even within the privacy of our own thoughts, and we need to distinguish it from merely holding up a brain state as an example. That is, when talking about what brain states are like, it’s important to distinguish between necessarily true mention-claims and controversial use-claims. Sliding unwittingly from one to the other can lead to false confidence in the unsupported notion of mysterious phenomenal redness – because the truth of the mention-claim (imagining redness is necessarily like imagining redness) is assumed to apply to the controversial use-claim (a brain-state is red).
A similar use-mention conflation is subtly woven through Jackson’s argument, affecting it at multiple levels. In Mary’s situation, these issues are particularly complex because she is studying a textbook representation of neurons that are themselves involved in representing redness, and we are in turn representing this nested series of representations (and I am now discussing your representation of this chain of relationships, representing your understanding in text).
Some degree of confusion is, perhaps, understandable.
But let’s approach this slowly. Mary’s textbooks describe the generic neural circuits that encode redness, and she reads this description to create her own personal neural representation of the generic neural circuits that in turn encode redness. To simplify the situation enormously, we might say that the red-encoding neurons she reads about are in the primary colour cortex of a generic human brain – in the V4 region of the occipital lobes. But Mary’s understanding of those V4 neurons is second-hand and involves different neurons in a different part of the brain. The relevant V4 neural patterns are merely being mentioned in her mind; they are not being used with their meaning intact. They don’t synapse onto the rest of her cognition or have their usual cognitive effects because they are mere abstractions – and they’re not even in her V4 cortex. Her frustration in not being able to form a standard redness concept can be described as being stuck in mention mode – it involves an inability to use the generic red-encoding V4 circuits within her own cognition, as well as her inability to get her own under-stimulated V4 neurons to copy the activity of the generic textbook V4 neurons. She can’t, in effect, remove the quotation marks from the circuits her cognition is quoting.
One irony that is often missed in this discussion is that the failure of her textbook neurons to encode redness stems directly from their abstract, non-physical nature.
Because there are multiple layers of representation involved, we can considering this representational hierarchy at another level. The generic V4 neurons in Mary’s textbook (and later in her understanding of the textbook) merely represent redness, so when Mary attempts to understand a generic neural circuit and concludes that she doesn’t find redness in the textbook brain, this is actually a correct conclusion; a V4 neural circuit in a human brain is not actually red, but merely represents redness. Her failure to derive redness in those circuits is, in that sense, entirely appropriate. (And photons outside the head are indifferent to the human colour spectrum, so it is appropriate for her to find no redness there, too).
When we read Mary’s story, or discuss her situation, additional neural systems come into play – in your brain, and mine. We entertain a thought of the following form: Mary fails to discover that the relevant neural state is like this. Unless we have aphantasia, we probably imagine redness while we mentally voice the pronoun. What she misses out on is this. In doing so, we need to be aware that we are holding up an example of what she is missing; we are not necessarily imagining a true property of the brain state that she lacks.
Another way of looking at this situation is to note that there is a theory-example conflation involved in Jackson’s argument. Theory-example conflations are almost exclusively confined to the philosophy of the mind; they are, essentially, an unusual variant of a use-mention conflation. Think of the relation between the quoted word, “green” and an actual example of green lettering: that’s the simplest form of use-mention conflation. If the quoted word grew in complexity to become an entire theory of colour in a textbook, and the example grew to become a library of colour images, then a theory-example conflation would be to mistake the theory of colour with the library of examples. Such a mistake would be very unlikely – no one would ever mistake a text-based theory with a library of images – but in Mary’s case, the theory is embodied in neurons, and so is the library of examples, and usually they are both within conceptual reach. The fact that, ordinarily, theory and example are both reachable concepts within our heads makes it easy to slide from one to the other, and gives us unrealistic expectations of what a theory should be able to achieve.
In any other domain, this sort of theory-example conflation would not be possible. For instance, we would not expect Sally the Sun Scientist to be able to produce a thermonuclear fusion reaction in her laboratory, just because she is reading about it. Even when the theory and example are both neural, we would normally reject the idea that an example should be derivable from a theory. Suppose that Emily the Epilepsy Scientist is a world expert on epilepsy who has never had a seizure. She reads that such-and-such a combination of neural firings produces a seizure. Can she concentrate very hard and produce the brain-state that she is reading about? Or suppose that Agnes the Ignorance Scientist is a world expert on the neurobiological basis of ignorance, with a particular interest in the neurobiological basis of ignorance about neurobiology. Could she achieve a state of complete ignorance about neurobiology by studying that state? Why not? Does she lack some vital information about ignorance? Is ignorance mysteriously non-physical?
Reading about specific entities does not allow us to produce those entities at will, not even when the entities under consideration are themselves brain states that involve neurons that are, under the right circumstances, under our control.
What is different about MAry’s situation is that the brain-state that is denied to her seems tantalisingly close, because we can reach it through a non-deductive process. This is not true of seizures or ignorance or any of the non-neural examples.
So why would anyone expect Mary to be able to derive redness? Why would anyone draw any conclusion from her failure to generate an example? Why does the Knowledge Argument seem plausible at all?
One way we might slip into a theory-example conflation in Mary’s case without noticing it is if we have mistaken an example of a perceptual state for a fact about a perceptual state. For instance, we might think that our ability to conjure up an example of imagining redness is a fact about the world – in our world, redness looks like that. Chalmers, for instance, explicitly cites this sort of statement as a fact (as we will see below). But this “fact” entails the same use-mention conflation considered earlier.
This will be a recurring theme. Many of the most puzzling questions in relation to the mind are primarily puzzling because they ignore representational issues of this nature and, in particular, because they ignore the representational twist that appears whenever we turn a representational system upon itself.
The physicalist philosopher, David Papineau, has drawn similar conclusions about use-mention conflations affecting our view of qualia, as discussed in his 2002 book, “Thinking About Consciousness.” The book explored our intuitive distrust of physicalism (which he called “materialism”), relating this distrust to conceptual confusion surrounding a class of concepts that he called phenomenal concepts. These are, essentially, concepts about qualia; that is, they are concepts about perceptual states that we usually entertain by holding up an example of themselves. We might think of them as quotational concepts or demonstrative concepts – concepts we activate and point to when we want to think about them, so that they become self-referential.
Consider the following sentence. Italics are like this. The linguistic content of the sentence tells us nothing about italics; it relies on a self-referential example.
Our standard concept of redness has this same quotational nature. When we think of redness, we usually re-activate many of the circuits involved in our normal perception of redness; when we think of pain, we reactivate many of the same circuits that are activated in states of pain, and so on. This is more than a speculative theory; it has been confirmed with imaging studies.
Reading about these same brain states objectively, without actually activating the corresponding circuits, provides different cognitive modes of referring to those same brain states, producing what Papineau refers to as material concepts. In Papineau’s terminology, pre-release Mary has a material concept of redness, and on her release she forms a phenomenal concept of redness. One reason she shouldn’t be considered to have acquired any new facts upon her release is that she is able to refer to the redness experience and all the facts about it with either concept. Prior to her release, she can’t think about the experience by the convenient method of holding up a self-referential example, but she can nonetheless think about the experience via a bland, cumbersome alternative.
Our intuitive rejection of physicalism is related, in part, to the fact that our material concepts are disappointingly bland, but our material concepts nonetheless point to the same facts.
Essentially, what Papineau is conceding is that there is a genuine sense in which material concepts “leave out” the experience at issue. They “leave out” the pain and the technicolour phenomenology, in the sense that they don’t activate or involve these experiences. Unfortunately, it is all too easy to slide from this unremarkable truth to the conclusion that, in exercising material concepts, we are not thinking about the experiences themselves.
But it simply does not follow that material concepts ‘leave out’ the feelings in the sense of failing to mention them. They can still refer to the feelings, even though they don’t activate them. After all, most concepts don’t use or involve the things they refer to. When I think of being rich, say, or having measles, this doesn’t in any sense make me rich or give me measles. In using the states they mention, phenomenal concepts are very much the exception. So we shouldn’t conclude on this account that material concepts, which work in the normal way of most concepts, in not using the states they mention, fail to refer to those states.
(David Papineau. Thinking about Consciousness, 2002)
Many readers are likely to be unconvinced by this line of argument. In most domains, the difference between using a representational device and mentioning it can be explicitly marked and easily reversed. We have conventions for indicating whether the word on the sign is green or “green”, and it is not difficult to change from one to the other. Delete the quotation marks, and we’re done. So Jackson’s challenge can be restated. Instead of asking why Mary cannot derive redness, we can ask what stops her from doing the neural equivalent of removing the quotation marks around her redness concept. Why can’t she perform the cognitive switch from mention to use?
And now we can finally return to the formatting issues that represent the true physical barrier to Mary’s attempt to drive redness. In the physical domain, the main thing that pre-release Mary lacks is an appropriately formatted primary colour cortex in the V4 region of her own occipital lobe. By formatting, I mean acquisition of the synaptic weights that allow her to represent colour as an attribute of surfaces and light sources, and to represent a range of colours with the appropriate similarity and difference relations that ordinarily locate each colour in the appropriate part of the human colour space.
The problem with Jackson’s argument is that we know why she lacks this formatting. The basic circuitry of vision processing in the brain was mapped decades ago. Black-and-white inputs fail to produce traffic in the relevant parts of the neural circuitry, so her primary colour cortex never gets its synaptic weights adjusted.
Viewing Mary’s brain as a purely physical organ, there is no reason to expect that inputting black-and-white text and diagrams to Mary’s eyes should enable her to format her own colour cortex to match the circuits described in her textbook. There are basic neuroanatomical factors that get in the way of this project. When processing vision, the brain segregates information streams based on movement, form and colour, sending the information to separate regions for processing. Because she never receives coloured inputs, Mary’s colour circuits would be expected to remain dormant, and the cognitive modularity of her brain means that she can’t activate the relevant circuits through a force of will. In the most literal, boring sense, there are synapses pointing the wrong way, and cognitive modules that are causally upstream instead of downstream. That is, there are entirely mundane physical factors that prevent black-and-white inputs from having any capacity to exert a formatting influence on specialized cortex that is embryologically and anatomically dedicated to colour representation.
The biggest irony of Jackson’s argument is that the best explanation for Mary’s frustration is a physical explanation.
On Mary’s release, what she acquires (in the accepted version of her story) is not awareness of a new fact, but an appropriately formatted colour cortex in the V4 region of her occipital lobe. In practice, after such severe sensory deprivation, a single exposure to colour would not provide the required formatting; nor would it rescue her colour cortex from its stunted state, so Mary would probably require intensive visual rehabilitation for months or years, and might never achieve normal colour vision.
Assuming, for the sake of the argument, that her V4 cortex is magically kept healthy and capable of single-exposure training, we can account for her new knowledge: she finally formats her colour cortex. On exposure to coloured light, the colour is detected as a difference in the activation of different retinal cone cell. Activation ratios that are consistent with coloured inputs can now stimulate pathways that modulate the primary colour cortex, inducing changes in synaptic weights. Associated with these formatting changes, Mary acquires several new abilities. For instance, when post-release Mary sees something red, she can compare the incoming information to the patterns in her newly formatted colour cortex, and so she knows that she is seeing something red. When she wants to imagine redness, she can now activate a neural ensemble in her newly formatted colour cortex. Unlike her previous abstract, generic V4 neurons that were themselves merely represented; she can now activate real, physical V4 neurons. These are appropriately connected to the rest of her cognition, such that other visual processing modules can draw on all the relevant associations and cognitive abilities associated with representing redness, including assigning the representation of redness to surfaces in her visual world.
Mary’s new cognitive relationship with redness is so different from her prior study of the same circuits that I believe it counts as a legitimate form of knowledge. We can concede to Jackson that she learns something. But we know why her cognition changes, and it is all consistent with physicalism. Her initial circumstances denied her ordinary knowledge of redness for reasons that conventional neuroscience already understands, and (if we accept the myth of an instantaneously rehabilitated V4 cortex), then Mary acquires this knowledge via physical processes that conventional neuroscience also understands. In terms of her cognition, the knowledge gap between her initial frustration and her later acquisition of knowing what red looks like is therefore an entirely explicable feature of the physical world.
Note that, whether or not we call Mary’s acquisition of a formatted colour cortex “learning” or acquiring “knowledge” makes no substantive difference to what we actually think happened, provided that we are consistent in our usage of the terms. These are all semantic quibbles. In the AI era, the concept of learning explicitly includes the adjustment of synaptic weights, so I think it would be silly to deny that Mary learns. Adjudicating on whether she gains knowledge is trickier, but if we decide that “knowledge” can apply to formatting changes, then all we imply by the word is that Mary undergoes formatting changes on her release. If we decide that the term “knowledge” must pick out major truths about the world – that it must involve the acquisition of facts – then we need to decide whether this includes indirect facts that are potentially derivable from other information. We also need to consider whether there are any facts that were already true before Mary’s release that she only learned after her release.
If an anti-physicalist could show that there was a true fact about the world not covered by Mary’s extensive knowledge prior to her release, then Jackson’s argument would have some merit. Instead, most anti-physicalists fall back on an unacknowledged use-mention conflation and simply assert that she learns a fact – the fact that redness looks like this.
Here, for instance, is Chalmers, commenting on the proposal that Mary merely acquires new abilities on her release, not new information (an argument put forward by Lewis and Nemirow, **[missing date]).
“…Lewis and Nemirow argue that, at most Mary gains a new ability. For example, she gains the ability to imagine the sight of red things, and to recognise them when she sees them, but this is only knowledge how, not knowledge that. When she first experiences red [according to Lewis and Nemirow], she learns no facts about the world. … This strategy does not suffer from internal problems. Its main problem is that it is deeply implausible. No doubt Mary does gain some abilities when she first experiences red, as she gains some abilities when she learns to ride a bicycle. But it certainly seems that she learns something else, some facts about the nature of experience. For all she knew before, the experience of red things might have been like this, or it might have been like that, or it might even have been like nothing at all. But now she knows that it is like this. (Chalmers, 1996)
Note the telltale use of pronouns: “this” and “that”. These pronouns are the parts of the supposed facts that hide the use-mention conflation; they correspond to the part of the thought in Chalmers’ head where he is unable to come up with an independent proposition for Mary to learn but instead must hold up an example of the relevant brain state. In place of any fact that Mary learns, Chalmers invites his readers to similarly hold up an example in their own heads, activating a brain state that either matches or fails to match the example that Mary acquires on her release. If we tried to replace the pronouns with anything more definite, the fact would evaporate:
For all she knew before, the experience of red things might have been like this example of me activating my representation of blue, or it might have been like that neural representation of yellow, or it might even have been like nothing at all. But now she knows that it is like this neural representation of red.
But if Mary knew everything physical, she already knew which representations in Chalmers head were the ones associated with redness. She already knew that she would acquire an similar neural representation on her release, and she could already refer to that representation – albeit without holding up an example.
I’ve been following the Mary debate for decades, now, waiting for any antiphysicalist to identify a genuine fact that Mary cannot derive that was already true before her release, and which she learns upon her release – where a fact is a proposition that does not rely on a pronoun to hide a use-mention conflation. No anti-physicalist has ever identified such a fact, but the issues get increasingly murky as arbitrary linguistic decrees come to dominate the debate. For what it is worth, I think Mary’s acquisition of the necessary occipital formatting constitutes genuine learning and genuine knowledge, but it does not constitute acquisition of any fact that was true before her release. Mary obviously acquires facts of the form, “The currently activated pattern in my newly formatted colour cortex is an example of a human brain imagining redness,” but facts of that nature were not true prior to her release. If we adjust the story so that she has perfect knowledge of the physical events that will occur upon her release, then she can refer to the example that she will gain on her release and, even before her release, say anything about that example that is true. What she gains on her release is the actual example, allowing her to use much more efficient ways to refer to those same facts. As in the passage from Chalmers, she can finally hold up an example of the relevant brain state and say, redness is like this.
The important question in all of this is whether Mary’s original deficit is best described as a deficit in her original information. Jackson’s original argument treated the idea that “redness is like this” as a fact, rather than as a case of holding up an example. Once we step away from this misconception, there is not much left of his argument. Mary already knows all the necessary formatting changes in her colour cortex that would be required to produce an example of redness in her head; she just can’t implement those changes merely by thinking about them. In principle, with nanoscopic surgical access to her synapses and enough computing power, she could prove that her information was adequate by using that information to reformat her occipital cortex to match the circuitry in her textbooks.
So, Jackson’s argument fails.
Does this exhaust criticism of Jackson’s argument and the usual notion of qualia? Well, no. In fact, we’re just getting started. Misunderstanding Mary’s situation can be achieved in a couple of paragraphs; understanding her situation will take most of this book.
This might seem like a lot of effort to waste on an argument that ultimately lacks merit, but I have always thought that, if antiphysicalists had not invented Mary, then physicalists would have done well to invent her themselves. We can use Mary’s story to gain greater insight into most of the faulty assumptions underlying the Hard Problem.
In particular, Mary shows how it is possible for important concepts to be left out of a complete theory of the mind-brain relation. And hence it prepares us for the other main rhetorical device that is often levelled against physicalism: the Zombie Argument, or what we might call Chalmers’ Razor.