I'd be careful of conflating "computers can't think", which is about substrate dependence, with "LLMs can't think", which might be more about architecture, etc.
For example, I'd expect that for something to be conscious it has to have some coherent notion of the passage of time for it--it has to entangle with entropic processes in some non trivial way. So a lookup table doesn't count, not because it's written on paper instead of brain tissue, but because the information contained in it is too static.
So a sim-brain could be made out of silicon, and still have this property; or you could have a lookup table written in brain tissue which wouldn't.
Whether your imagined sim brains would pass this test I'm not sure... But I don't think the fact that they're made out of sim-components is the source of my doubts.
I also wouldn't be too quick to write off the relevance of quantum stuff here; suppose you believe in something like Scott Aaronson's freebit idea for free will: that free will is possible because human beings can entangle with a source of previously-unentangled, low-entropy qubits left lying around from the big bang. If you think that, say, free will is a necessary component of consciousness, you might think anything incapable of using freebits as a resource can't be conscious, which might (though, also might not) rule out your sim-brains.
Re: LLMs; absolutely. The passage of time in particular does not seem essential to me, but there are just so many things we don't know about LLMs that we shouldn't be confident one way or the other.
Re: free will; if you are a qualia realist it is very difficult to see why free will is required for a red quale to exist, especially since we can have experiences without making any conscious choices. (If you are not a qualia realist then the question is more or less empty.)
Yeah, I don't know that there's any very obvious reason to think free will must bear on consciousness, beyond "they're both mysterious things we associate with being human"--but just to say there's a lot of stuff we don't know.
My probably more considered feelings about consciousness specifically is that, the big place where you'd expect to find substrate/microphysical stuff mattering is in the gap between speakable, classical information and other kinds of information: precisely because classical information is "fungible"--it can be transmitted independently of the specific physical system on which its represented--if consciousness depends only on it, it's hard to imagine why some physical systems shouldn't reproduce it.
So if you want a view like that, you basically need to assume that consciousness depends on more concretely physical information: either quantum information (not fungible because of no-cloning) as in the freebit picture, or indexical/reference frame information (not fungible because of the nature of being indexical), or something else I can't think of.
I personally suspect indexical stuff is going to have a strong connection to quantum info (you can share some reference frame info, without a shared reference frame, if you have an appropriate quantum channel), so ultimately, in my view, if you want consciousness to be substrate-dependent, you probably need to think about entangling with weird quantum resources; I think the freebit picture is the best example of a theory that has that kind of shape and has something to do with human cognition, so I think it's a connection worth keeping in mind, but I admit it's a pretty tenuous chain of links that gets you here.
Last thing I'll say is in defense of the view that you should believe you do need a theory that isn't just about fungible information for consciousness, and thus why you'd be motivated to start down this path of weird ideas in the first place:
Fungible information makes no sense without a protocol to interpret it; precisely because it's so independent of physical medium, the same pattern in a physical representation can stand for anything... This can't work for consciousness: this is basically the "is the random fluctuation of temperatures in a rock conscious?"--I think the presumptive answer has to be, "no"; it's not enough that one can identify some pattern as potentially representing a pattern associated with conscious thought.
But if you believe that, it's most natural to believe that there is some canonical, physical, unique "protocol" for associating conscious experience to underlying physical systems--unlike classical information where any association can work.
i recently got a lot out of sandberg’s ‘simulation’ vs. ‘emulation’ distinction:
“The term emulation originates in computer science, where it denotes mimicking the function of a program or computer hardware by having its low-level functions simulated by another program. While a simulation mimics the outward results, an emulation mimics the internal causal dynamics (at some suitable level of description). The emulation is regarded as successful if the emulated system produces the same outward behaviour and results as the original (possibly with a speed difference).”
Well, of course it isn’t an argument; for one, it’s an imperative! For another, it’s the premise for the rest of the article, which is an argument, even if you think it’s a bad one.
Well it is a hopelessly circular one since as far as we know physical reality contains thinking entities. And then there is the little detail that the assumption is incompatible with physics/known laws of nature e.g. see Feynman's 1981 "Simulating physics with computers" (https://s2.smu.edu/~mitch/class/5395/papers/feynman-quantum-1981.pdf)
1. In the initial form of the experiment, thinking things are outsourced as brains in vats. Then, I consider simulating the microphysics of brains, since as far as we know brains are made of ordinary atoms. Now, it's possible to think that the atoms won't move in the right ways without some kind of dualistic mental intervention, but I'd ask one to defend why. The point is only: the *fact that it is a simulation* alone is not enough to explain why you couldn't have minds work.
2. In that Feynman paper, he only demonstrates that simulating quantum physics is likely to be very *time-inefficient* using *classical* computers, on the order of 2^n, although this is only conjectured. It does not disprove the Church-Turing thesis, only the "strong" claim about computational efficiency. Furthermore, as I put in one of the footnotes, this computer could be a large quantum computer; it makes no difference to the hypothetical!
1. Well for all we know ordinary atoms and their behavior are Quantum mechanical. This has nothing to with dualistic mental intervention, the problem is that one can not make digital/simulated atoms move and act as physical atoms do. Here it suffices to look at how notoriously hard it is in practice to numerically *approximate* the shape or energy level of even simple molecules.
2. From Feynman's paper: "Can a quantum system be probabilistically simulated by a classical (probabilistic, I'd assume) universal computer? .... If you take the computer to be the classical kind I've described so far, (not the quantum kind described in the last section) and there're no changes
in any laws, and there's no hocus-pocus, the answer is certainly, No!" . And the exponential scaling hardly helps unless one is ready to accept that the tiniest spec of sim-fire would require a *classical computer comparable in size to the observable universe. And Quantum computers, contrary to common assumption, do not offer a way out either to quote Feynman again "could we imitate every quantum mechanical system which is discrete
and has a finite number of degrees of freedom?" i.e. Quantum computers by definition simulate *discrete quantum systems with a finite number of degrees of freedom*. However, most physical Quantum mechanical systems are simply not of this kind including atoms and molecules.
I'd be careful of conflating "computers can't think", which is about substrate dependence, with "LLMs can't think", which might be more about architecture, etc.
For example, I'd expect that for something to be conscious it has to have some coherent notion of the passage of time for it--it has to entangle with entropic processes in some non trivial way. So a lookup table doesn't count, not because it's written on paper instead of brain tissue, but because the information contained in it is too static.
So a sim-brain could be made out of silicon, and still have this property; or you could have a lookup table written in brain tissue which wouldn't.
Whether your imagined sim brains would pass this test I'm not sure... But I don't think the fact that they're made out of sim-components is the source of my doubts.
I also wouldn't be too quick to write off the relevance of quantum stuff here; suppose you believe in something like Scott Aaronson's freebit idea for free will: that free will is possible because human beings can entangle with a source of previously-unentangled, low-entropy qubits left lying around from the big bang. If you think that, say, free will is a necessary component of consciousness, you might think anything incapable of using freebits as a resource can't be conscious, which might (though, also might not) rule out your sim-brains.
Re: LLMs; absolutely. The passage of time in particular does not seem essential to me, but there are just so many things we don't know about LLMs that we shouldn't be confident one way or the other.
Re: free will; if you are a qualia realist it is very difficult to see why free will is required for a red quale to exist, especially since we can have experiences without making any conscious choices. (If you are not a qualia realist then the question is more or less empty.)
Yeah, I don't know that there's any very obvious reason to think free will must bear on consciousness, beyond "they're both mysterious things we associate with being human"--but just to say there's a lot of stuff we don't know.
My probably more considered feelings about consciousness specifically is that, the big place where you'd expect to find substrate/microphysical stuff mattering is in the gap between speakable, classical information and other kinds of information: precisely because classical information is "fungible"--it can be transmitted independently of the specific physical system on which its represented--if consciousness depends only on it, it's hard to imagine why some physical systems shouldn't reproduce it.
So if you want a view like that, you basically need to assume that consciousness depends on more concretely physical information: either quantum information (not fungible because of no-cloning) as in the freebit picture, or indexical/reference frame information (not fungible because of the nature of being indexical), or something else I can't think of.
I personally suspect indexical stuff is going to have a strong connection to quantum info (you can share some reference frame info, without a shared reference frame, if you have an appropriate quantum channel), so ultimately, in my view, if you want consciousness to be substrate-dependent, you probably need to think about entangling with weird quantum resources; I think the freebit picture is the best example of a theory that has that kind of shape and has something to do with human cognition, so I think it's a connection worth keeping in mind, but I admit it's a pretty tenuous chain of links that gets you here.
Last thing I'll say is in defense of the view that you should believe you do need a theory that isn't just about fungible information for consciousness, and thus why you'd be motivated to start down this path of weird ideas in the first place:
Fungible information makes no sense without a protocol to interpret it; precisely because it's so independent of physical medium, the same pattern in a physical representation can stand for anything... This can't work for consciousness: this is basically the "is the random fluctuation of temperatures in a rock conscious?"--I think the presumptive answer has to be, "no"; it's not enough that one can identify some pattern as potentially representing a pattern associated with conscious thought.
But if you believe that, it's most natural to believe that there is some canonical, physical, unique "protocol" for associating conscious experience to underlying physical systems--unlike classical information where any association can work.
i recently got a lot out of sandberg’s ‘simulation’ vs. ‘emulation’ distinction:
“The term emulation originates in computer science, where it denotes mimicking the function of a program or computer hardware by having its low-level functions simulated by another program. While a simulation mimics the outward results, an emulation mimics the internal causal dynamics (at some suitable level of description). The emulation is regarded as successful if the emulated system produces the same outward behaviour and results as the original (possibly with a speed difference).”
from https://www.openphilanthropy.org/wp-content/uploads/SandbergandBostrom2008.pdf
'Imagine a simulation that is indistinguishable from physical reality' isn't an argument either.
Well, of course it isn’t an argument; for one, it’s an imperative! For another, it’s the premise for the rest of the article, which is an argument, even if you think it’s a bad one.
Well it is a hopelessly circular one since as far as we know physical reality contains thinking entities. And then there is the little detail that the assumption is incompatible with physics/known laws of nature e.g. see Feynman's 1981 "Simulating physics with computers" (https://s2.smu.edu/~mitch/class/5395/papers/feynman-quantum-1981.pdf)
1. In the initial form of the experiment, thinking things are outsourced as brains in vats. Then, I consider simulating the microphysics of brains, since as far as we know brains are made of ordinary atoms. Now, it's possible to think that the atoms won't move in the right ways without some kind of dualistic mental intervention, but I'd ask one to defend why. The point is only: the *fact that it is a simulation* alone is not enough to explain why you couldn't have minds work.
2. In that Feynman paper, he only demonstrates that simulating quantum physics is likely to be very *time-inefficient* using *classical* computers, on the order of 2^n, although this is only conjectured. It does not disprove the Church-Turing thesis, only the "strong" claim about computational efficiency. Furthermore, as I put in one of the footnotes, this computer could be a large quantum computer; it makes no difference to the hypothetical!
1. Well for all we know ordinary atoms and their behavior are Quantum mechanical. This has nothing to with dualistic mental intervention, the problem is that one can not make digital/simulated atoms move and act as physical atoms do. Here it suffices to look at how notoriously hard it is in practice to numerically *approximate* the shape or energy level of even simple molecules.
2. From Feynman's paper: "Can a quantum system be probabilistically simulated by a classical (probabilistic, I'd assume) universal computer? .... If you take the computer to be the classical kind I've described so far, (not the quantum kind described in the last section) and there're no changes
in any laws, and there's no hocus-pocus, the answer is certainly, No!" . And the exponential scaling hardly helps unless one is ready to accept that the tiniest spec of sim-fire would require a *classical computer comparable in size to the observable universe. And Quantum computers, contrary to common assumption, do not offer a way out either to quote Feynman again "could we imitate every quantum mechanical system which is discrete
and has a finite number of degrees of freedom?" i.e. Quantum computers by definition simulate *discrete quantum systems with a finite number of degrees of freedom*. However, most physical Quantum mechanical systems are simply not of this kind including atoms and molecules.