Why couldn't LLMs have non-reductive reasoning powers?
Humans got them somehow!
Alex Strasser objects to my last article:
“It is insufficient to call LLMs “a system of probabilistic word prediction through word vectors,” when I could just as easily call a human “a system of probabilistic muscle movements through neural action potentials.””
I think a big difference between the first and second is that the first is true (or meaningfully representative) and the second is false (or not meaningfully representative). LLMs are explicitly built in a way that is fundamentally different than anything remotely resembling thinking: predicting which word/token is most likely to come next (and then select the most likely one with some randomness tossed in). Humans take in reasons and weigh them and decide based on that. Wording this in a reductionist way is only (meaningfully) accurate if reductionism is correct, which it isn’t, as I can know that via introspection and reason.
I am a reductionist, so I’ll grant that as a point of difference between me and Alex. However, I do not think this substantially undermines my argument.
I think non-reductionists should believe that whatever special non-reductive stuff humans have, it is plausible that LLMs have them too.
How the reductionist view works
Now, I also believe that humans take in reasons and weigh them! I just think reasons are instantiated physically. Just as The Brothers Karamazov can be described on a physical level as “nothing more” than a certain configuration of ink on a page, I think human reasons can be described as “nothing more” than a certain configuration of neural impulses.
I admit that “reasons” is a higher level of description than the neural impulses and thus there is some vagueness about the definition of “reasons,” just as The Brothers Karamazov is a higher level description than one set of cellulose and ink. For one thing, there are many copies of The Brothers Karamazov, in many languages, some with pages missing, some with coffee stains and typos, but they can all be meaningfully said to be The Brothers Karamazov. It could be said that The Brothers Karamazov isn’t a physical thing; it’s a story. And yet,
every instance of The Brothers Karamazov is a physical thing.
The Brothers Karamazov could exist in conceivable random universes without there being any sentient beings or any intent behind it.
There is a copy of The Brothers Karamazov in Borges’ library.
This is the sense in which I think reasons and The Brothers Karamazov are both reductively explained. On reductionism, it is perfectly possible for LLMs to act on reasons, just as it is perfectly possible for brains to act based on inspiration from the story. On reductionism, if I read The Brothers Karamazov and decide to bite your finger to mimic Ilyusha, it can be rightly said that The Brothers Karamazov was one of the causes of me acting that way, even though it can also be rightly said that The Brothers Karamazov isn’t a physical object.
What about non-reductionism?
I take non-reductionism to imply that the relation between The Brothers Karamazov and physical objects is not sufficient to explain the relation between reasons and events in physical brains. There is something else missing: a soul, an essence, some non-physical mental powers, something.
I think it’s possible we have something like this. Consciousness is strange!
The next question, though, is what reasons we have for believing we have this stuff and LLMs don’t.
Here is what seems true, even granting non-reductionism.
Humans have brains. These are fleshy things full of electrical impulses, chemical transmission, and cells that look pretty mechanistic. They also look necessary for reasoning: if a rock falls on your head and turns your brain into mush, you can no longer reason.
If a part of the brain that’s typically active in a particular task gets turned into mush, you typically can’t do that task as well, until your brain physically changes to adapt. Differences in brains correspond to differences in certain competencies, dispositions, and moods. So it looks like brains are, somehow, tightly coupled to reasoning.
How did brains happen? It looks like humans evolved. Evolution is a process which blindly selects organisms based on their reproductive fitness. Perhaps evolution was also guided by some other processes which were looking for intelligence or other features (Alex writes about faith, so he likely believes this).
Looking from the outside, it is not self-evident that other people operate on reasons. That is, on non-reductionism, it is conceivable to construct a hypothetical purely physical thing which mimics human reasoning super well and looks just like a person from the outside. I make no claims about whether it is introspectively knowable that you do in fact operate on reasons.
However, we are justified in believing that other people operate on reasons. Most of the time, they behave like they do. They seem very similar to us. Notice, however, that all of this evidence is physical: we are inferring some sort of non-reductive mental powers based on seeing, hearing, and feeling things which could be faked by physical puppets.
Given these observations, I think it plausible that whatever non-reductive elements humans somehow got, LLMs somehow got them too.
Yes, base LLMs are iteratively selected just to predict the next token. But we were iteratively selected just to make more copies of ourselves! That’s as dumb a process as next-token prediction. In fact, it’s theoretically dumber: to mimic humans effectively, you must have some internal mechanistic representations of sophisticated stuff that we talk about, in order to calculate what we likely would say to novel data. Bacteria are much too simple to be great next-token predictors, but they are excellent at making copies of themselves. So you’d expect next-token prediction to be more likely to be linked to non-reductive reasoning powers as compared to baby-making.
But perhaps this selection process was intelligently guided by some non-reductive power? Well, so was the creation of LLMs: humans made them, and humans make choices based on non-reductive reasons; therefore, non-reductive reasons are bound up in the cause of LLMs. In fact, humans were explicitly trying to search for something truly intelligent, and have added all kinds of reinforcement learning processes to shift away from pure next-token selection based on corpus probability to based on desirability for a number of tasks. Why couldn’t LLMs have acquired these powers through the guided evolution of human-directed gradient descent?
Similarly, you might argue that LLMs look like big piles of numbers from the outside. But humans look like big piles of flesh from the outside! Everything about your fellow man looks physical, except perhaps his outputs: the way he speaks, acts, laughs, communicates speaks to an intelligence like yours. Similarly, LLMs look like big piles of numbers. But their actions speak to an intelligence like yours, or at least like your little brother’s! They can do all of these remarkable expressive, creative, and mathematical things which no being without human non-reductive reasoning has been able to do at any point in history. That gives at least a prima facie case that they act on reasons too.
Good induction
The idea that LLMs have non-reductive ability to act on reasons seems strange, but it is a priori no stranger than the fact that all these flesh-things have non-reductive ability to act on reasons.
But I know I’ve got them, somehow. I’m not certain these other humans have them, but they’re like me in observable ways, so I’m pretty sure they do.
Are LLMs like me in observable ways? In many observable ways, no. But Shaquille O’Neal is, in many observable ways, is also not like me. The question is about the observable ways which seem to be bound up in thinking somehow, like having a brain and having certain behaviors and responding to stimuli in certain ways. LLMs are more different from me than I am from Shaq in those respects. But they are also similar in a lot of ways—more similar to me than I am to dogs, for instance. Therefore, whatever reasons I have for believing humans work like me should reasonably extend to LLMs.