You have the patience of a Greek god. All Hoel proved was third person observation is insufficient for claiming consciousness. That's the Hard Problem of Consciousness restated and says nothing about ontology. Thanks for addressing the minutia
Well, thanks! I think one of the valuable uses of blogging is surfacing assumptions/interesting details in more formal papers, where inexperienced folks might feel intimidated by the scope of the thing. So this is a service I'm happy to provide, although it did take a lot of work to get the details right :)
This objection isn't really an objection at all. It's basically admitting the argument and then arguing for two positions on consciousness that are seriously extreme, on either end of the spectrum (specifically, interactive dualism and illusionism).
If the proof is strong enough to constrain theories of consciousness to those two, then it's pretty darn powerful!
Making this seem like a general problem with the paper requires a shell game about definitions on your end. For instance, you write that:
>> " What K&H demonstrate is that if you deny consciousness as a causal role and deny mental causation and insist a materially falsifiable theory of consciousness should exist, you’re going to get stuck in a serious theoretical bind."
This is the heart of the shell game, because you're substituting in reasonable sounding terms like "deny consciousness a causal role" for "don't believe in interactive dualism." E.g., we don't deny consciousness the kind of causal role it takes in, e.g., IIT, or GNWT. So I'd like you to answer this: Would most people say that those theories deny consciousness a causal role? I think the obvious answer to that question is no. You are forced to answer "yes," and I'd like to hear your reasoning about it.
And that reasoning can't just be a repetition of "interactive dualism is true" (or otherwise, this is indeed a definitional shell game to make an extreme position of yours seem more general). I predict answering "yes" is hard. E.g., in IIT, consciousness is deeply associated with causation, to the point of basically being an identity. And yet, the reasoning of the disproof still applies to IIT.
The same shell game of presenting extreme positions with normal ones is occurring with regards to illusionism: yes, if consciousness is an illusion, then the proof doesn't apply... and then LLMs aren't conscious! That's not a very good objection.
I think a more interesting post might have simply said "Erik's proof doesn't apply to interactive dualism" and sketched your own version of what that might look like in detail as a candidate theory that avoids the dilemma, much as I did (but for a different theory class). But it's not like the paper is unaware of this option. I do mention quantum-based theories in the paper as a possible example of lenient dependency (which I think are the closest modern equivalent to interactive dualism). However, the reason I don't spend much time on it because I find it obvious that interactive dualism would still rule out LLM consciousness! It's basically impossible to have interactive dualism in a computer - it would have to happen in precisely a way that doesn't break, e.g., the machine code. You can read a really thorough case for why that is here:
I'll probably be more interested in your second objection. But I'll note that you just gave a good case here wherein there are indeed theories that pass through the dilemma in humans (here, interactive dualism) but not LLMs! Again I know you seem to believe interactive dualism makes sense in LLMs, but I'll stress that basically no one else I'm aware of does and it seems really difficult to make work. So for your second objection to be true, you'll have to prove (or at least give a very good argument) that interactive dualism of the same kind we'd expect to be associated with consciousness in humans can sensibly apply to LLMs. Again, I'd suggest checking out the linked paper, which basically shows that can't be the case due to how CPUs and GPUs work. So your further post on your next objection is going to be at odds with this one.
To sum up: you've said "this paper is wrong because it doesn't consider two extreme theories, which, if true, would each also mean that LLMs are not conscious." Like I said, not much of an objection!
Since you offer some mildly accusative language about the level of certainty here, I'll take the opportunity to clarify my position: The proof works as a proof. So if the proof holds, it's 100%. The point of that sentence you highlight as an overclaim (on my end) is to distinguish between other common methods, like weighting probabilities of theories of consciousness, which this approach isn't doing. Of course, the proof could be be wrong, because some assumption is untrue: just like any other proof. But talking about this is like cataloging "the unknown unknowns" and I don't think it's egregious to not list "unknown unknowns" in a blog post introducing something. And honestly, I think the claim underlying the shell game you're doing here - which is that computers might have some sort of viable interactive dualism just like humans do, and this is some sort of obvious theory we should take into consideration - is way more radical and wild than any sort of claim I'm making (even assuming you're saying "I think it's X probability") and could be criticized in exactly the same way.
Hey, thanks for the reply, Erik! I think these are the main points of disagreement:
1. I'm certainly not intending to pull some kind of "shell game!" I do apologize about my tone in the last paragraph, since it seems I misread your position. I think the forcefulness with which you hold your assumptions is unjustified given the high moral stakes which we both take very seriously, and I think my clumsy attempts to make this clear were more accusatory than they needed to be.
2. When I say "deny consciousness a causal role," what I am referring to is the assumption in your model that P → O → E and there is no arrow from E → O or E → P in the way that you contend there *is* an arrow from T → O in the temperature case. It seems to me that you treat experience alone as being unique in that it cannot be observed by a third party, in a way that every other property that belongs in the realm of scientific study can. That is what I am referring to: that these *experiential properties* have no detectable causal role; only the physical substrate causes anything, but you seem to believe that experience is something distinct from physical substrate.
If I am misunderstanding, I'd ask you to explain how IIT contends that both (a) consciousness (or experience, whichever word is appropriate) is private and unobservable, and (b) it plays a causal role distinct from its physical substrate. It's entirely possible I am confused, but the recent open letter against IIT suggests to me that this is not obvious to many scientists. (And also, sufficient inert logic gates having extremely high ϕ and being unboundedly more conscious than humans, according to Tononi does not suggest the relationship between consciousness and causation being "basically an identity", though perhaps this is just a demonstration of the Kliener-Hoel dilemma?)
3. On illusionism: illusionists don't deny consciousness. They deny that ineffable, intrinsic, private, and directly apprehensible qualia exist. Humans still have experiences of pleasure and pain, they are just not pleasure and pain qualia. If humans are worthy of moral consideration without qualia, then proving that LLMs don't have qualia proves nothing about their moral status, which you cite as one of the primary motivations for your paper.
4. On dualism: I don't think the idea of interaction in computers is significantly more extreme than the idea of interaction in brains, which both papers mention as a genuine avenue for a falsifiable theory of consciousness. However, I agree that it does seem quite technically challenging, and I assign a very low probability to it.
5. On the extremity of my position: like I say, perhaps your definitions of consciousness are scientifically mainstream, but in my experience type-A and type-D positions aren't so extreme in philosophy, and the PhilPapers survey shows similar results. Furthermore, if the Kleiner-Hoel dilemma truly holds for type-B materialism, that to me seems like a strong argument *against* believing in type-B materialism. There may be other routes out, as you indicate, but the existence of your argument should update us in favor of alternatives as well.
6. Nowhere do I "admit the argument". In this post, I am considering the background assumptions behind the argument. In next week's post, I am considering whether the argument holds for type-B materialism; I have mixed thoughts so far.
Appreciate the thought-out response Jack. I'm still convinced that this is basically a shell-game around interactive dualism. Point-by-point reply to your comment here:
> 1. "I think the forcefulness with which you hold your assumptions is unjustified given the high moral stakes which we both take very seriously, and I think my clumsy attempts to make this clear were more accusatory than they needed to be."
Fair enough, and appreciate it. But my broad point is that any criticism of overclaiming is a double-standard. You work with Graziano, yes? Here's a paper title of his from 2022: "Consciousness is already solved." In 2014, he wrote an op-ed in The New York Times with the title "Are We Really Conscious?" and arguing, basically: no. That seems like a higher moral stakes than any contemporary claim about LLMs, for a much bigger audience than my personal blog!
> 2. "When I say "deny consciousness a causal role," what I am referring to is the assumption in your model that P → O → E and there is no arrow from E → O or E → P in the way that you contend there *is* an arrow from T → O in the temperature case. It seems to me that you treat experience alone as being unique in that it cannot be observed by a third party, in a way that every other property that belongs in the realm of scientific study can."
The argument requires only a weaker claim, which is that consciousness is not a *physical* observable like temperature. But that doesn't, in turn, mean that consciousness is unique scientifically, and doesn't mean we are making some weird or unlikely assumption about it. See, for instance, the entire history of functionalism or computationalism. Is computation a physical observable? What about representation? What about Hume's entire famous line of reasoning about causation not being physically observable? Yes, their physical correlates might be observable, but the actual action of them is arguably not. The degree the K-H dilemma applies to other subjects is an open area of research for us right now, but the claim that the argument needs consciousness to be entirely unique in this regard isn't true. This has significant bearing on the extensiveness of the conclusions, since I think so many theories are classes of causal structure theories, or computationalist theories, or functionalist theories!
>> "3. On illusionism: illusionists don't deny consciousness. They deny that ineffable, intrinsic, private, and directly apprehensible qualia exist. Humans still have experiences of pleasure and pain, they are just not pleasure and pain qualia."
I think this is all just language games to soften the blow. Here's from Graziano, again in The New York Times:
"How does the brain go beyond processing information to become subjectively aware of information? The answer is: It doesn’t. The brain has arrived at a conclusion that is not correct. When we introspect and seem to find that ghostly thing — awareness, consciousness, the way green looks or pain feels — our cognitive machinery is accessing internal models and those models are providing information that is wrong. The machinery is computing an elaborate story about a magical-seeming property."
I think this is all just language games which are, at best, over-indexing on the zombie argument; at worst, illusionists are basically just purposefully soft-selling a radical conclusion to make it salable.
>> "4. On dualism: I don't think the idea of interaction in computers is significantly more extreme than the idea of interaction in brains, which both papers mention as a genuine avenue for a falsifiable theory of consciousness. However, I agree that it does seem quite technically challenging, and I assign a very low probability to it."
But dualism is far less technically challenging in humans, no? I can imagine all sorts of esoteric quantum wetware stuff. Not saying any of that is true, it's just such a theory seems quite imaginable to me. Whereas with LLMs... how do you do it without messing up the GPU? The machine code? The clocks? Even the slightest bit flip is a problem. So therefore, we have at least one (albeit radical, imo) theory that makes sense for humans, that arguably passes the dilemma, but not for LLMs! Orch-Or, believe it or don't (or quibble with its definition as interactive dualism or not - I think it qualifies) is at least well-developed and extensive. Nothing like that even makes sense for LLMs. Same even goes for, e.g., solving the combination problem: seems a lot easier to get extended temporal timescales and integration and so on in a brain than something running state-changes according to a fixed clock. Something at least to consider for the second piece, since again, it sounds like it will be in tension with this admission.
> "5. On the extremity of my position: like I say, perhaps your definitions of consciousness are scientifically mainstream, but in my experience type-A and type-D positions aren't so extreme in philosophy, and the PhilPapers survey shows similar results. Furthermore, if the Kleiner-Hoel dilemma truly holds for type-B materialism, that to me seems like a strong argument *against* believing in type-B materialism."
These categories were designed for stances on the zombie argument. I actually don't think they're super helpful for reasoning about classes of existing scientific theories. Can you name one current scientific theory of consciousness that's *not* interactive dualism (like some kind of Orch-Or) and that would obviously avoid both horns of the K-H dilemma? Which contemporary theory is immune to universal substitutions? Or doesn't, instead, base its predictions in the I/O of a system and become trivial? I'd certainly love to hear about those theories, legitimately, since those are great target theories for the next phase of this research. But if there aren't a bunch of theories on hand that the K-H dilemma fails to apply to, then indeed, this particular criticism is a shell-game for interactive dualism, since - many paragraphs in - it's still the only even-somewhat-defined-theory actually mentioned that specifically would pass through the dilemma. It seems that, if it's so obvious that such theories exist, and I ignored them due to limiting assumptions about consciousness, we should be able to name those theories easily!
I think most of the points raised here I will try to clarify in the follow-up post, so many thanks for the shaping of that! Here is the single thing I think we disagree on the most:
I do not know of a scientific theory which satisfies your formalism and accepts its assumptions. I think the assumptions behind your formalism, that:
1. experience is a private property which cannot be observed, only inferred
2. the hard problem of consciousness is real; i.e. solutions to the meta-hard problem are not solutions to the hard problem
3. there must be a scientifically falsifiable theory of consciousness which is non-trivial and not vulnerable to substitution; that is, if there is no such falsifiable theory for a system S then S cannot be conscious
ought to be rejected. If 1 & 2 hold, then why would consciousness be in the domain of scientific theories? If 3 holds, why would 1 hold? These assumptions seem to be in great tension, if not in contradiction.
If it's really the case that most neuroscientific theories hold 1-3 as true, then you have done something quite worthwhile in condemning them to the rubbish heap! But my conclusion from this, coming primarily from a philosophy background, is that most neuroscientific theories are philosophically confused, not that we can rule out systems from being conscious on these theoretical grounds.
> "I do not know of a scientific theory which satisfies your formalism and accepts its assumptions."
This sounds as if it's coherent to say, but let's look at the implication: Can you name a single theory - just one, any one - that would definitely not be testable in the way that's formalized in the paper? Because of some differing assumption it has? If so, which theory?
More broadly, I don't agree with the form these assumptions are written in. E.g., regarding (2), I don't think the Hard Problem is relevant to make the case, nor the meta-hard problem, etc.
Meanwhile, I think (1) and (3) are written in ways that make them seem more radical than they are, and each could equally be presented in ways far more convincing (or in a more conservative, acceptable form), all while the proof still works. In their more convincing/conservative presentations they are nigh universal, and you're pretending they're not (thus, the shell game).
Let's focus on what I'd change about (1): I already previously outlined why I think "inferred not observed" is not strange or unscientific or unique, and doesn't require your "private property" language (which is what makes it seem like a strong assumption), and indeed only renders consciousness similar to computation, representation, function, and causation (notably, many contemporary theories ground consciousness in such properties). Does anyone think we open up the skull and *physically observe* consciousness the way we do temperature? What theory holds that? Again, beyond interactive dualism?
And then as for (3), I'd replace it with something like: "yes, we assume for the proof that a scientifically falsifiable and non-trivial theory of consciousness exists," etc; which is explicitly discussed at length in the paper.
The proof is in the pudding. Even if, as you say, most neuroscientific theories are philosophically confused, for your original criticism to hold true it must be possible to name a single specific scientific theory of consciousness that the K-H dilemma doesn't apply to. But naming one appears quite difficult!
I like Tower of Babbel's response. On the illusionist side, attention schema theory would also work. You claim that attention schema theory is "trivial", but it provides falsifiable predictions of what produces reports of consciousness in humans, why we think in terms of vague, mysterious essences, etc. If you are an illusionist, there is nothing further which needs explaining, and the assumption that observations need to map onto predicted *qualitative experiences*—rather than self-reports, or dispositions, or other measurable things—is mistaken.
AST is a great example of a trivial and unfalsifiable theory that falls on the second horn of the K-H dilemma. That's why Graziano is able to title his papers "Attention Schema Theory: Why it Has to Be Right." Theories that have to be right are entirely trivial, and contain no scientific information. Here's another clear statement of it, in an abstract, this time from "We Are Machines that Claim to Be Conscious:"
"The attention schema theory explains how a biological,
information processing machine can claim to have consciousness..."
But yet, claims to consciousness are how we falsify theories! So this is just a gussied up form of behaviorism: you are conscious (whatever that means in the theory) because you can make claims to consciousness. It's mixing inferences and predictions, and that creates strict dependency, which creates unfalsifiability and triviality. Once you work it out, any prediction from AST about say, the state of experience based on the brain, has to be based on the accessibility of claims to consciousness within that brain, and claims to consciousness are also the experimental inferences, and the snake eats its own tail.
Sorry if you’ve answered this in the original paper, I’ve only read your Substack article:
Take a more moderate view than illusionism or dualism, like identity theory (I don’t think it will matter whether it is type, token, type-A, or type-B). If identity theory is true, then conscious mental states are identical to some neural state (the stock example is that pain is identical with C-fiber firings). If you accept an identity theory, the picture you have in mind where we infer the existence of some unobservable extra physical thing, over and above the underlying brain state, is just wrong headed. You do see the pain, when you look at the nervous system you see the firing of the C-fibers.
this sort of view isn’t as out there as illusionism or dualism, indeed I think something in the proximity is still relatively popular, and I don’t see how your proof would cut against these sorts of views.
Work through it. Why do you think C-fiber firings are identical to conscious experiences of pain? What scientific evidence can you marshal? How do you falsify the C-fibers = pain theory? I think you end back at a priori falsification via universal substitution very quickly, e.g., I can imagine a system without C-fibers (like a human born without them, etc), but that still reports pain. Or some other substrate that accomplishes the same basic signal-passing in humans. Regardless, those systems (humans without C-fibers, 10th generation Optimus bots, whatever) all report pain quite convincingly. You then... what? Disbelieve them? On what grounds? Because C-fibers are just the way its done for us? Much more sensibly, you'd realize that your C-fiber = pain identity is falsified by what are in the paper called "mismatches."
so ultimately, all you are saying is that you cant name an established theory that satisfies all these conditions? lol what a waste of time this was. on the other hand, i didnt read the paper and now i know not to bother. so i guess i got good value from from this discussion, and i should be more appreciative! to demonstrate this i wont criticize further.
" What K&H demonstrate is that if you deny consciousness as a causal role and deny mental causation and insist a materially falsifiable theory of consciousness should exist, you’re going to get stuck in a serious theoretical bind. ... But then the question is why you are justified in taking those three assumptions as your starting point."
If the paper takes these assumptions as a starting point, then it is largely meaningless.
It's not just that the assumptions need to be defended, it is that they are in mutual tension and cannot all be true.
If consciousness is epiphenomenal, then its presence can never be proven in LLMs or people. But it can never be disproven either. The epiphenomenal conception of consciousness totally and permanently rules out a materially falsifiable theory of consciousness (and also rules out the consciousness being posited as the cause of our original and on-going interest in any of these issues.)
I don't know if you have characterised the paper fairly, but it comes across as a incoherent mess. You have piqued my interest enough to read it, but you've also given me grounds for thinking that reading it would be a waste of time. But why would a neuroscience journal allow speculation about epiphenomenal entities? That strikes me as an unhealthy departure from science and good sense.
One point I would make is that the Type-A/Type-B distinction is made from a Chalmersian perspective, and it doesn't hold up well under other framings. When it comes to the A/B border, I swap sides based on which definition of "experience" is in play, and most discussions skip over the definitional step. From your post, it seems as though Erik has an epiphenomenal conception of consciousness, which would make me Type A. But I usually identify more strongly with Type B materialism because I think epiphenomenal conceptions of consciousness are total nonsense and I want to move on to discussing something more sensible.
If the paper did not make the definition explicit, then that alone is a flaw as big as the ones you've pointed out.
I'm offering Hoel the benefit of the doubt w.r.t. type B theories this week; you're right that he might even be closer to an epiphenomenalist, but I'm planning to analyze that further in the next post. To keep posts down to a manageable size, I'm just trying to make the more limited claim that Hoel's argument *definitely* doesn't work for type-A and type-D theories. And yes, K&H are not really explicit about what kind of consciousness they are postulating.
I really did try my best to review this paper honestly, and spent a while trying to get it right. I'm obviously critical of it, but I think that makes it even more important to stay accurate. That said, feeding the paper into your LLM of choice and asking some critical questions is probably worth it if you're interested, even if reading the whole paper isn't.
The irony here is that I didn’t read the paper or related posts because I take it as obvious that current LLMs are not conscious. I don’t think I could “prove” it because it really depends on one’s definition of consciousness. Personally, I would want to see some form attention schema and ongoing representation of interiority and self-hood.
I think that future AIs could easily be conscious, so there must be some notional line that would distinguish between current LLMs with their obvious non-consciousness and future AIs with their potential consciousness as good as ours. Perhaps that line is vague, but there would be cases on either side that would be convincing.
For me, any argument that current LLMs are on the non-conscious side of the line would have to appeal to specific features of the cognitive architecture of consciousness, not to broad philosophical brush-strokes or to tangential issues like epiphenomenal consciousness.
I think Michael and I are in general agreement on human consciousness.
I don’t think it would take much to give an LLM powered cognitive system an ongoing attention schema. What they lack currently is the continuity and ongoing narrative cohesion.
Hoel's paper seems to have a large hole. It depends on the argument that there exists a lookup table that would map the inputs of an LLM model to the output.
But for a typical LLM with a vocabulary of say 100k tokens, and a context length of just 8000 tokens, that table would then be 100,000 ^ 8000, or 10^40,000 entries. But there are only 10^80 particles in the observable universe.
And you could make a similar lookup table argument for an evolving neural block as he suggests is needed for consciousness, even a human brain: just imagine that all of the inputs are serialized into one file over one minute, and all the outputs in another. In principle, you could say there's a lookup table that maps any such series of inputs to outputs. Yet a human brain has an experience over that time.
You might argue that the fact the real universe makes these lookup tables trivially impossible doesn't matter for an argument in principle, but I think it does, because consciousness / experience exist in a real and finite reality, they are in some way a phenomena of real universe we live in.
Type-A Materialism would be better named Strawman Materialism, since so few of the alleged Type-A Materialists believe in the analytic truths distinctive of the so-called theory. Type-Q Materialism fares far better.
Sorry to kvetch about a small point but it seems to me like this argument relies on the falsity not only type-A materialism, but also type-B. A type-B materialist can still think that consciousness plays a causal role (they can even think this is necessary if they do some fancy footwork in denying modal rationalism). Regardless, type-b physicalists can definitely think that consciousness *is* whatever plays such-and-such a causal role without thinking that said identity between the thing specified by the causal role concept and the mental concept is something like an apriori identity. Anyway, I agree with everything else you say, this seems mostly like an argument that epiphenomenalism entails that llms cant be conscious (or maybe that we can never know that they are) which… seems… not that interesting? Maybe I’m missing something though!
Gotcha. Like I told Zinbiel I wanted to give Hoel the benefit of the doubt, but you might be right. Either way, this doesn't seem like a novel insight.
For the record, your Ghostbusters are the exact opposite of the characters from the movie, who are scientists that do believe in ghosts while the people around them don’t. Well, maybe that was intentional. I’ll just point it out for pedantry’s sake.
I did know that, but I couldn't think of a better pun name. So I asked Claude to generate pun names, and they were awful. So Ghostbusters it was. In my defense, the Ghostbusters logo, "I ain't afraid of no ghosts", and "Bustin' makes me feel good" could just as easily apply to a crew of eliminativists!
You have the patience of a Greek god. All Hoel proved was third person observation is insufficient for claiming consciousness. That's the Hard Problem of Consciousness restated and says nothing about ontology. Thanks for addressing the minutia
Well, thanks! I think one of the valuable uses of blogging is surfacing assumptions/interesting details in more formal papers, where inexperienced folks might feel intimidated by the scope of the thing. So this is a service I'm happy to provide, although it did take a lot of work to get the details right :)
Here's another take for you both: https://xxywise.substack.com/p/it-aint-just-code-bro-how-erik-hoel
This objection isn't really an objection at all. It's basically admitting the argument and then arguing for two positions on consciousness that are seriously extreme, on either end of the spectrum (specifically, interactive dualism and illusionism).
If the proof is strong enough to constrain theories of consciousness to those two, then it's pretty darn powerful!
Making this seem like a general problem with the paper requires a shell game about definitions on your end. For instance, you write that:
>> " What K&H demonstrate is that if you deny consciousness as a causal role and deny mental causation and insist a materially falsifiable theory of consciousness should exist, you’re going to get stuck in a serious theoretical bind."
This is the heart of the shell game, because you're substituting in reasonable sounding terms like "deny consciousness a causal role" for "don't believe in interactive dualism." E.g., we don't deny consciousness the kind of causal role it takes in, e.g., IIT, or GNWT. So I'd like you to answer this: Would most people say that those theories deny consciousness a causal role? I think the obvious answer to that question is no. You are forced to answer "yes," and I'd like to hear your reasoning about it.
And that reasoning can't just be a repetition of "interactive dualism is true" (or otherwise, this is indeed a definitional shell game to make an extreme position of yours seem more general). I predict answering "yes" is hard. E.g., in IIT, consciousness is deeply associated with causation, to the point of basically being an identity. And yet, the reasoning of the disproof still applies to IIT.
The same shell game of presenting extreme positions with normal ones is occurring with regards to illusionism: yes, if consciousness is an illusion, then the proof doesn't apply... and then LLMs aren't conscious! That's not a very good objection.
I think a more interesting post might have simply said "Erik's proof doesn't apply to interactive dualism" and sketched your own version of what that might look like in detail as a candidate theory that avoids the dilemma, much as I did (but for a different theory class). But it's not like the paper is unaware of this option. I do mention quantum-based theories in the paper as a possible example of lenient dependency (which I think are the closest modern equivalent to interactive dualism). However, the reason I don't spend much time on it because I find it obvious that interactive dualism would still rule out LLM consciousness! It's basically impossible to have interactive dualism in a computer - it would have to happen in precisely a way that doesn't break, e.g., the machine code. You can read a really thorough case for why that is here:
https://arxiv.org/abs/2304.05077
I'll probably be more interested in your second objection. But I'll note that you just gave a good case here wherein there are indeed theories that pass through the dilemma in humans (here, interactive dualism) but not LLMs! Again I know you seem to believe interactive dualism makes sense in LLMs, but I'll stress that basically no one else I'm aware of does and it seems really difficult to make work. So for your second objection to be true, you'll have to prove (or at least give a very good argument) that interactive dualism of the same kind we'd expect to be associated with consciousness in humans can sensibly apply to LLMs. Again, I'd suggest checking out the linked paper, which basically shows that can't be the case due to how CPUs and GPUs work. So your further post on your next objection is going to be at odds with this one.
To sum up: you've said "this paper is wrong because it doesn't consider two extreme theories, which, if true, would each also mean that LLMs are not conscious." Like I said, not much of an objection!
Since you offer some mildly accusative language about the level of certainty here, I'll take the opportunity to clarify my position: The proof works as a proof. So if the proof holds, it's 100%. The point of that sentence you highlight as an overclaim (on my end) is to distinguish between other common methods, like weighting probabilities of theories of consciousness, which this approach isn't doing. Of course, the proof could be be wrong, because some assumption is untrue: just like any other proof. But talking about this is like cataloging "the unknown unknowns" and I don't think it's egregious to not list "unknown unknowns" in a blog post introducing something. And honestly, I think the claim underlying the shell game you're doing here - which is that computers might have some sort of viable interactive dualism just like humans do, and this is some sort of obvious theory we should take into consideration - is way more radical and wild than any sort of claim I'm making (even assuming you're saying "I think it's X probability") and could be criticized in exactly the same way.
Hey, thanks for the reply, Erik! I think these are the main points of disagreement:
1. I'm certainly not intending to pull some kind of "shell game!" I do apologize about my tone in the last paragraph, since it seems I misread your position. I think the forcefulness with which you hold your assumptions is unjustified given the high moral stakes which we both take very seriously, and I think my clumsy attempts to make this clear were more accusatory than they needed to be.
2. When I say "deny consciousness a causal role," what I am referring to is the assumption in your model that P → O → E and there is no arrow from E → O or E → P in the way that you contend there *is* an arrow from T → O in the temperature case. It seems to me that you treat experience alone as being unique in that it cannot be observed by a third party, in a way that every other property that belongs in the realm of scientific study can. That is what I am referring to: that these *experiential properties* have no detectable causal role; only the physical substrate causes anything, but you seem to believe that experience is something distinct from physical substrate.
If I am misunderstanding, I'd ask you to explain how IIT contends that both (a) consciousness (or experience, whichever word is appropriate) is private and unobservable, and (b) it plays a causal role distinct from its physical substrate. It's entirely possible I am confused, but the recent open letter against IIT suggests to me that this is not obvious to many scientists. (And also, sufficient inert logic gates having extremely high ϕ and being unboundedly more conscious than humans, according to Tononi does not suggest the relationship between consciousness and causation being "basically an identity", though perhaps this is just a demonstration of the Kliener-Hoel dilemma?)
3. On illusionism: illusionists don't deny consciousness. They deny that ineffable, intrinsic, private, and directly apprehensible qualia exist. Humans still have experiences of pleasure and pain, they are just not pleasure and pain qualia. If humans are worthy of moral consideration without qualia, then proving that LLMs don't have qualia proves nothing about their moral status, which you cite as one of the primary motivations for your paper.
4. On dualism: I don't think the idea of interaction in computers is significantly more extreme than the idea of interaction in brains, which both papers mention as a genuine avenue for a falsifiable theory of consciousness. However, I agree that it does seem quite technically challenging, and I assign a very low probability to it.
5. On the extremity of my position: like I say, perhaps your definitions of consciousness are scientifically mainstream, but in my experience type-A and type-D positions aren't so extreme in philosophy, and the PhilPapers survey shows similar results. Furthermore, if the Kleiner-Hoel dilemma truly holds for type-B materialism, that to me seems like a strong argument *against* believing in type-B materialism. There may be other routes out, as you indicate, but the existence of your argument should update us in favor of alternatives as well.
6. Nowhere do I "admit the argument". In this post, I am considering the background assumptions behind the argument. In next week's post, I am considering whether the argument holds for type-B materialism; I have mixed thoughts so far.
Appreciate the thought-out response Jack. I'm still convinced that this is basically a shell-game around interactive dualism. Point-by-point reply to your comment here:
> 1. "I think the forcefulness with which you hold your assumptions is unjustified given the high moral stakes which we both take very seriously, and I think my clumsy attempts to make this clear were more accusatory than they needed to be."
Fair enough, and appreciate it. But my broad point is that any criticism of overclaiming is a double-standard. You work with Graziano, yes? Here's a paper title of his from 2022: "Consciousness is already solved." In 2014, he wrote an op-ed in The New York Times with the title "Are We Really Conscious?" and arguing, basically: no. That seems like a higher moral stakes than any contemporary claim about LLMs, for a much bigger audience than my personal blog!
> 2. "When I say "deny consciousness a causal role," what I am referring to is the assumption in your model that P → O → E and there is no arrow from E → O or E → P in the way that you contend there *is* an arrow from T → O in the temperature case. It seems to me that you treat experience alone as being unique in that it cannot be observed by a third party, in a way that every other property that belongs in the realm of scientific study can."
The argument requires only a weaker claim, which is that consciousness is not a *physical* observable like temperature. But that doesn't, in turn, mean that consciousness is unique scientifically, and doesn't mean we are making some weird or unlikely assumption about it. See, for instance, the entire history of functionalism or computationalism. Is computation a physical observable? What about representation? What about Hume's entire famous line of reasoning about causation not being physically observable? Yes, their physical correlates might be observable, but the actual action of them is arguably not. The degree the K-H dilemma applies to other subjects is an open area of research for us right now, but the claim that the argument needs consciousness to be entirely unique in this regard isn't true. This has significant bearing on the extensiveness of the conclusions, since I think so many theories are classes of causal structure theories, or computationalist theories, or functionalist theories!
>> "3. On illusionism: illusionists don't deny consciousness. They deny that ineffable, intrinsic, private, and directly apprehensible qualia exist. Humans still have experiences of pleasure and pain, they are just not pleasure and pain qualia."
I think this is all just language games to soften the blow. Here's from Graziano, again in The New York Times:
"How does the brain go beyond processing information to become subjectively aware of information? The answer is: It doesn’t. The brain has arrived at a conclusion that is not correct. When we introspect and seem to find that ghostly thing — awareness, consciousness, the way green looks or pain feels — our cognitive machinery is accessing internal models and those models are providing information that is wrong. The machinery is computing an elaborate story about a magical-seeming property."
I think this is all just language games which are, at best, over-indexing on the zombie argument; at worst, illusionists are basically just purposefully soft-selling a radical conclusion to make it salable.
>> "4. On dualism: I don't think the idea of interaction in computers is significantly more extreme than the idea of interaction in brains, which both papers mention as a genuine avenue for a falsifiable theory of consciousness. However, I agree that it does seem quite technically challenging, and I assign a very low probability to it."
But dualism is far less technically challenging in humans, no? I can imagine all sorts of esoteric quantum wetware stuff. Not saying any of that is true, it's just such a theory seems quite imaginable to me. Whereas with LLMs... how do you do it without messing up the GPU? The machine code? The clocks? Even the slightest bit flip is a problem. So therefore, we have at least one (albeit radical, imo) theory that makes sense for humans, that arguably passes the dilemma, but not for LLMs! Orch-Or, believe it or don't (or quibble with its definition as interactive dualism or not - I think it qualifies) is at least well-developed and extensive. Nothing like that even makes sense for LLMs. Same even goes for, e.g., solving the combination problem: seems a lot easier to get extended temporal timescales and integration and so on in a brain than something running state-changes according to a fixed clock. Something at least to consider for the second piece, since again, it sounds like it will be in tension with this admission.
> "5. On the extremity of my position: like I say, perhaps your definitions of consciousness are scientifically mainstream, but in my experience type-A and type-D positions aren't so extreme in philosophy, and the PhilPapers survey shows similar results. Furthermore, if the Kleiner-Hoel dilemma truly holds for type-B materialism, that to me seems like a strong argument *against* believing in type-B materialism."
These categories were designed for stances on the zombie argument. I actually don't think they're super helpful for reasoning about classes of existing scientific theories. Can you name one current scientific theory of consciousness that's *not* interactive dualism (like some kind of Orch-Or) and that would obviously avoid both horns of the K-H dilemma? Which contemporary theory is immune to universal substitutions? Or doesn't, instead, base its predictions in the I/O of a system and become trivial? I'd certainly love to hear about those theories, legitimately, since those are great target theories for the next phase of this research. But if there aren't a bunch of theories on hand that the K-H dilemma fails to apply to, then indeed, this particular criticism is a shell-game for interactive dualism, since - many paragraphs in - it's still the only even-somewhat-defined-theory actually mentioned that specifically would pass through the dilemma. It seems that, if it's so obvious that such theories exist, and I ignored them due to limiting assumptions about consciousness, we should be able to name those theories easily!
I think most of the points raised here I will try to clarify in the follow-up post, so many thanks for the shaping of that! Here is the single thing I think we disagree on the most:
I do not know of a scientific theory which satisfies your formalism and accepts its assumptions. I think the assumptions behind your formalism, that:
1. experience is a private property which cannot be observed, only inferred
2. the hard problem of consciousness is real; i.e. solutions to the meta-hard problem are not solutions to the hard problem
3. there must be a scientifically falsifiable theory of consciousness which is non-trivial and not vulnerable to substitution; that is, if there is no such falsifiable theory for a system S then S cannot be conscious
ought to be rejected. If 1 & 2 hold, then why would consciousness be in the domain of scientific theories? If 3 holds, why would 1 hold? These assumptions seem to be in great tension, if not in contradiction.
If it's really the case that most neuroscientific theories hold 1-3 as true, then you have done something quite worthwhile in condemning them to the rubbish heap! But my conclusion from this, coming primarily from a philosophy background, is that most neuroscientific theories are philosophically confused, not that we can rule out systems from being conscious on these theoretical grounds.
> "I do not know of a scientific theory which satisfies your formalism and accepts its assumptions."
This sounds as if it's coherent to say, but let's look at the implication: Can you name a single theory - just one, any one - that would definitely not be testable in the way that's formalized in the paper? Because of some differing assumption it has? If so, which theory?
More broadly, I don't agree with the form these assumptions are written in. E.g., regarding (2), I don't think the Hard Problem is relevant to make the case, nor the meta-hard problem, etc.
Meanwhile, I think (1) and (3) are written in ways that make them seem more radical than they are, and each could equally be presented in ways far more convincing (or in a more conservative, acceptable form), all while the proof still works. In their more convincing/conservative presentations they are nigh universal, and you're pretending they're not (thus, the shell game).
Let's focus on what I'd change about (1): I already previously outlined why I think "inferred not observed" is not strange or unscientific or unique, and doesn't require your "private property" language (which is what makes it seem like a strong assumption), and indeed only renders consciousness similar to computation, representation, function, and causation (notably, many contemporary theories ground consciousness in such properties). Does anyone think we open up the skull and *physically observe* consciousness the way we do temperature? What theory holds that? Again, beyond interactive dualism?
And then as for (3), I'd replace it with something like: "yes, we assume for the proof that a scientifically falsifiable and non-trivial theory of consciousness exists," etc; which is explicitly discussed at length in the paper.
The proof is in the pudding. Even if, as you say, most neuroscientific theories are philosophically confused, for your original criticism to hold true it must be possible to name a single specific scientific theory of consciousness that the K-H dilemma doesn't apply to. But naming one appears quite difficult!
I like Tower of Babbel's response. On the illusionist side, attention schema theory would also work. You claim that attention schema theory is "trivial", but it provides falsifiable predictions of what produces reports of consciousness in humans, why we think in terms of vague, mysterious essences, etc. If you are an illusionist, there is nothing further which needs explaining, and the assumption that observations need to map onto predicted *qualitative experiences*—rather than self-reports, or dispositions, or other measurable things—is mistaken.
AST is a great example of a trivial and unfalsifiable theory that falls on the second horn of the K-H dilemma. That's why Graziano is able to title his papers "Attention Schema Theory: Why it Has to Be Right." Theories that have to be right are entirely trivial, and contain no scientific information. Here's another clear statement of it, in an abstract, this time from "We Are Machines that Claim to Be Conscious:"
"The attention schema theory explains how a biological,
information processing machine can claim to have consciousness..."
But yet, claims to consciousness are how we falsify theories! So this is just a gussied up form of behaviorism: you are conscious (whatever that means in the theory) because you can make claims to consciousness. It's mixing inferences and predictions, and that creates strict dependency, which creates unfalsifiability and triviality. Once you work it out, any prediction from AST about say, the state of experience based on the brain, has to be based on the accessibility of claims to consciousness within that brain, and claims to consciousness are also the experimental inferences, and the snake eats its own tail.
Sorry if you’ve answered this in the original paper, I’ve only read your Substack article:
Take a more moderate view than illusionism or dualism, like identity theory (I don’t think it will matter whether it is type, token, type-A, or type-B). If identity theory is true, then conscious mental states are identical to some neural state (the stock example is that pain is identical with C-fiber firings). If you accept an identity theory, the picture you have in mind where we infer the existence of some unobservable extra physical thing, over and above the underlying brain state, is just wrong headed. You do see the pain, when you look at the nervous system you see the firing of the C-fibers.
this sort of view isn’t as out there as illusionism or dualism, indeed I think something in the proximity is still relatively popular, and I don’t see how your proof would cut against these sorts of views.
Work through it. Why do you think C-fiber firings are identical to conscious experiences of pain? What scientific evidence can you marshal? How do you falsify the C-fibers = pain theory? I think you end back at a priori falsification via universal substitution very quickly, e.g., I can imagine a system without C-fibers (like a human born without them, etc), but that still reports pain. Or some other substrate that accomplishes the same basic signal-passing in humans. Regardless, those systems (humans without C-fibers, 10th generation Optimus bots, whatever) all report pain quite convincingly. You then... what? Disbelieve them? On what grounds? Because C-fibers are just the way its done for us? Much more sensibly, you'd realize that your C-fiber = pain identity is falsified by what are in the paper called "mismatches."
so ultimately, all you are saying is that you cant name an established theory that satisfies all these conditions? lol what a waste of time this was. on the other hand, i didnt read the paper and now i know not to bother. so i guess i got good value from from this discussion, and i should be more appreciative! to demonstrate this i wont criticize further.
" What K&H demonstrate is that if you deny consciousness as a causal role and deny mental causation and insist a materially falsifiable theory of consciousness should exist, you’re going to get stuck in a serious theoretical bind. ... But then the question is why you are justified in taking those three assumptions as your starting point."
If the paper takes these assumptions as a starting point, then it is largely meaningless.
It's not just that the assumptions need to be defended, it is that they are in mutual tension and cannot all be true.
If consciousness is epiphenomenal, then its presence can never be proven in LLMs or people. But it can never be disproven either. The epiphenomenal conception of consciousness totally and permanently rules out a materially falsifiable theory of consciousness (and also rules out the consciousness being posited as the cause of our original and on-going interest in any of these issues.)
I don't know if you have characterised the paper fairly, but it comes across as a incoherent mess. You have piqued my interest enough to read it, but you've also given me grounds for thinking that reading it would be a waste of time. But why would a neuroscience journal allow speculation about epiphenomenal entities? That strikes me as an unhealthy departure from science and good sense.
One point I would make is that the Type-A/Type-B distinction is made from a Chalmersian perspective, and it doesn't hold up well under other framings. When it comes to the A/B border, I swap sides based on which definition of "experience" is in play, and most discussions skip over the definitional step. From your post, it seems as though Erik has an epiphenomenal conception of consciousness, which would make me Type A. But I usually identify more strongly with Type B materialism because I think epiphenomenal conceptions of consciousness are total nonsense and I want to move on to discussing something more sensible.
If the paper did not make the definition explicit, then that alone is a flaw as big as the ones you've pointed out.
I'm offering Hoel the benefit of the doubt w.r.t. type B theories this week; you're right that he might even be closer to an epiphenomenalist, but I'm planning to analyze that further in the next post. To keep posts down to a manageable size, I'm just trying to make the more limited claim that Hoel's argument *definitely* doesn't work for type-A and type-D theories. And yes, K&H are not really explicit about what kind of consciousness they are postulating.
I really did try my best to review this paper honestly, and spent a while trying to get it right. I'm obviously critical of it, but I think that makes it even more important to stay accurate. That said, feeding the paper into your LLM of choice and asking some critical questions is probably worth it if you're interested, even if reading the whole paper isn't.
The irony here is that I didn’t read the paper or related posts because I take it as obvious that current LLMs are not conscious. I don’t think I could “prove” it because it really depends on one’s definition of consciousness. Personally, I would want to see some form attention schema and ongoing representation of interiority and self-hood.
I think that future AIs could easily be conscious, so there must be some notional line that would distinguish between current LLMs with their obvious non-consciousness and future AIs with their potential consciousness as good as ours. Perhaps that line is vague, but there would be cases on either side that would be convincing.
For me, any argument that current LLMs are on the non-conscious side of the line would have to appeal to specific features of the cognitive architecture of consciousness, not to broad philosophical brush-strokes or to tangential issues like epiphenomenal consciousness.
Re: LLMs and attention schemas… Michael Graziano is my advisor for a paper I'm working on this semester, and it might change your mind. Stay tuned :)
thats awesome, AST is one of the best
Please let me know when you have the paper.
I’d also be happy to do a pre-submit review, and might have something useful to add; at least I’d be a fresh set of eyes.
If you’re working with Michael, that’s great. I think he has the best theory for what consciousness-itself actually is (as distinct from qualia).
It’s a shame his work is not more widely appreciated.
I think Michael and I are in general agreement on human consciousness.
I don’t think it would take much to give an LLM powered cognitive system an ongoing attention schema. What they lack currently is the continuity and ongoing narrative cohesion.
I think it’s a matter of degree.
continuity and narrative cohesion are precarious in humans. probably not news to you but worrh mentioning imo.
Hoel's paper seems to have a large hole. It depends on the argument that there exists a lookup table that would map the inputs of an LLM model to the output.
But for a typical LLM with a vocabulary of say 100k tokens, and a context length of just 8000 tokens, that table would then be 100,000 ^ 8000, or 10^40,000 entries. But there are only 10^80 particles in the observable universe.
And you could make a similar lookup table argument for an evolving neural block as he suggests is needed for consciousness, even a human brain: just imagine that all of the inputs are serialized into one file over one minute, and all the outputs in another. In principle, you could say there's a lookup table that maps any such series of inputs to outputs. Yet a human brain has an experience over that time.
You might argue that the fact the real universe makes these lookup tables trivially impossible doesn't matter for an argument in principle, but I think it does, because consciousness / experience exist in a real and finite reality, they are in some way a phenomena of real universe we live in.
fantastic write up and the comments section was absolute fire 🔥🔥🔥
thank you
Type-A Materialism would be better named Strawman Materialism, since so few of the alleged Type-A Materialists believe in the analytic truths distinctive of the so-called theory. Type-Q Materialism fares far better.
Sorry to kvetch about a small point but it seems to me like this argument relies on the falsity not only type-A materialism, but also type-B. A type-B materialist can still think that consciousness plays a causal role (they can even think this is necessary if they do some fancy footwork in denying modal rationalism). Regardless, type-b physicalists can definitely think that consciousness *is* whatever plays such-and-such a causal role without thinking that said identity between the thing specified by the causal role concept and the mental concept is something like an apriori identity. Anyway, I agree with everything else you say, this seems mostly like an argument that epiphenomenalism entails that llms cant be conscious (or maybe that we can never know that they are) which… seems… not that interesting? Maybe I’m missing something though!
Gotcha. Like I told Zinbiel I wanted to give Hoel the benefit of the doubt, but you might be right. Either way, this doesn't seem like a novel insight.
For the record, your Ghostbusters are the exact opposite of the characters from the movie, who are scientists that do believe in ghosts while the people around them don’t. Well, maybe that was intentional. I’ll just point it out for pedantry’s sake.
Nooo, my metaphor! :)
I did know that, but I couldn't think of a better pun name. So I asked Claude to generate pun names, and they were awful. So Ghostbusters it was. In my defense, the Ghostbusters logo, "I ain't afraid of no ghosts", and "Bustin' makes me feel good" could just as easily apply to a crew of eliminativists!
Here's a more interesting argument about LLM consciousness https://dhnecueno640274.substack.com/p/physicalism-and-llm-consciousness?utm_source=share&utm_medium=android&r=4t8vdh