I'm not confident a simplicity prior works for Chinese room cases; there are always multiple consistent interpretations of a symbolic language, and I expect at least some of those to be roughly equal in terms of Kolmogorov complexity (which varies by a constant according to your Turing machine implementation anyway). So I don't think it actually solves the philosophical problem with the Chinese room, and I have similar doubts for Doug's Law.
It will be quite practically useful for Gold cases though, as long as we are indeed talking about human languages which are favored by a simplicity prior.
Yeah I mean, I think there’s always going to be multiple consistent interpretations of most things. Even quantum mechanics has multiple consistent interpretations. It’s just about whether, even within those differing interpretations, there is enough you know to get the job done. What are the doubts about it fixing Doug’s law?
I mean multimodal LLMs get the same input as humans, basically. Images, language, code, error messages, etc. A simplicity prior seems to be ‘all you need’, as it were
Hmm. I have to think more, but soon I'll be writing a post on how I think LLMs get out of it, and I don't *think* it reduces to a simplicity prior. But we can discuss then about whether we actually are getting at the same thing :)
Sounds good. FWIW, I'm not claiming that a simplicity prior is all there is to the story, for sure. There's a lot more wacky stuff (even in humans, seems like we are in some ways evolutionarily trained for language, if you buy Chomsky's poverty of stimulus arguments, but then that's just a differently weighted prior).
But yeah, idk, if I'm totally honest I think that a lot of modern ML has shown something of the poverty of a lot of armchair reasoning that was going on in the 60s-present in linguistics/philosophy of linguistics (Chinese room, Gold's law, even the No free lunch theorem which was, I think, wildly misinterpreted and misapplied).
Very interesting post! These problems remind me a lot of the problem of induction/underdetermination of evidence. And as James suggests, it seems to me that some kind of rational restriction on priors is probably what should do the work (I don't know how that'd necessarily translate into LLMs).
Perhaps a deeper worry is related to the point you make about separating rules from meaning. I don't have very strong views on this, and haven't read too much about it, but it seems to me that meaning somehow consists in representing a way the world should be for a statement to be true. And I think that might be quite closely tied to sensory experience. (When saying "the cat is on the mat" I represent some sort of set of observations that would make me deem that true. That's probably not the whole story, but it seems like it's an aspect, and maybe a necessary condition for meaning.) So I wonder whether LLMs will ever get the ability to get "out there" and represent the world in any way, so long as they don't have any sensory input.
Just a few loose thoughts, don't know how plausible it is.
>>"but it seems to me that meaning somehow consists in representing a way the world should be for a statement to be true. And I think that might be quite closely tied to sensory experience.
[...]
So I wonder whether LLMs will ever get the ability to get "out there" and represent the world in any way, so long as they don't have any sensory input."
My somewhat half-baked thoughts on this (and I acknowledge in advance that there's some empirical evidence from LLMs that cuts against this):
If meaning is about the relationship between language and the world (which I think is correct) then to me that suggests at least a pretty high prior on needing sensory input. Maybe if we had the perfect scientific language dreamed of by the logical positivists you could train an LLM on a language whose internal structure was isomorphic enough to that of the world that learning the language would basically suffice; but one of the lessons of the early 20th C is that that dream doesn't work, at least not in any simple form. And certainly not with the actual language we have.
Another reason I think language alone might not be enough: there's a lot of information that can't be captured in language, even in principle: things like "knowing how" rather than "knowing that", or unspeakable physical information. The former also ties in to how we get feedback from the world: I imagine a lot of our training data is on things like "if I hear a sound, and then turn my head like so, the quality of the sound changes like so"--getting information from the feedback loop between our actions and our senses.
An LLM can't get this; it's feedback isn't directly from the world, but from linguistic representations of it, which per my first point is not nearly as good.
I'd also add you can (IMO) see the limits in learning just from linguistic structure in humans: even though science is highly mathematical we regularly find ourselves in need of experiment--we can't just use the structure of the formal language of the mathematics of our theory to advance our understanding of the world. I also think you can easily see people who get seduced by theories or representations of the world start to lose contact with it: I think even something like Goodheart's law often has this character, where we substitute some representation of what we care about in the world for the thing itself, and quickly discover that aiming at the representation comes apart from the thing.
None of this is to deny that representations are powerful, and reasoning using language and mathematics is an important component of reason--but I think we need to be regularly getting feedback from the world to actually keep our representations actually _representing_ the world.
> storing all their next-token predictions in a massive lookup table
LLMs don't make OR store next-token predictions, nor do they utilize any sort of lookup table. For every single token, in every single context, the model performs a fresh, forward pass through a massive network of matrices (weights) and nonlinear functions, resulting in a probability distribution over the vocabulary.
You'd think after 9 years, this would be common knowledge in the industry.
I never said that LLMs use a lookup table. I said IF they did so, I'd be sympathetic to those arguments, but since they don't, I'm not so sympathetic. The full quote is:
> "If LLMs were purely inductive models with no computational constraints, storing all their next-token predictions in a massive lookup table, and operating on an alien language with no semantic patterns in syntax, then yes, I would imagine they would fail pretty badly at understanding even if they become quite good mimics. But if LLMs receive any “help”, anything in their computational environment, training, or corpus that can link syntax to semantics, then these theoretical limits do not strictly apply."
Isn’t the real answer — for humans, for models, for any good simulators — just to have a simplicity prior?
I'm not confident a simplicity prior works for Chinese room cases; there are always multiple consistent interpretations of a symbolic language, and I expect at least some of those to be roughly equal in terms of Kolmogorov complexity (which varies by a constant according to your Turing machine implementation anyway). So I don't think it actually solves the philosophical problem with the Chinese room, and I have similar doubts for Doug's Law.
It will be quite practically useful for Gold cases though, as long as we are indeed talking about human languages which are favored by a simplicity prior.
Yeah I mean, I think there’s always going to be multiple consistent interpretations of most things. Even quantum mechanics has multiple consistent interpretations. It’s just about whether, even within those differing interpretations, there is enough you know to get the job done. What are the doubts about it fixing Doug’s law?
I mean multimodal LLMs get the same input as humans, basically. Images, language, code, error messages, etc. A simplicity prior seems to be ‘all you need’, as it were
Hmm. I have to think more, but soon I'll be writing a post on how I think LLMs get out of it, and I don't *think* it reduces to a simplicity prior. But we can discuss then about whether we actually are getting at the same thing :)
Sounds good. FWIW, I'm not claiming that a simplicity prior is all there is to the story, for sure. There's a lot more wacky stuff (even in humans, seems like we are in some ways evolutionarily trained for language, if you buy Chomsky's poverty of stimulus arguments, but then that's just a differently weighted prior).
But yeah, idk, if I'm totally honest I think that a lot of modern ML has shown something of the poverty of a lot of armchair reasoning that was going on in the 60s-present in linguistics/philosophy of linguistics (Chinese room, Gold's law, even the No free lunch theorem which was, I think, wildly misinterpreted and misapplied).
Very interesting post! These problems remind me a lot of the problem of induction/underdetermination of evidence. And as James suggests, it seems to me that some kind of rational restriction on priors is probably what should do the work (I don't know how that'd necessarily translate into LLMs).
Perhaps a deeper worry is related to the point you make about separating rules from meaning. I don't have very strong views on this, and haven't read too much about it, but it seems to me that meaning somehow consists in representing a way the world should be for a statement to be true. And I think that might be quite closely tied to sensory experience. (When saying "the cat is on the mat" I represent some sort of set of observations that would make me deem that true. That's probably not the whole story, but it seems like it's an aspect, and maybe a necessary condition for meaning.) So I wonder whether LLMs will ever get the ability to get "out there" and represent the world in any way, so long as they don't have any sensory input.
Just a few loose thoughts, don't know how plausible it is.
>>"but it seems to me that meaning somehow consists in representing a way the world should be for a statement to be true. And I think that might be quite closely tied to sensory experience.
[...]
So I wonder whether LLMs will ever get the ability to get "out there" and represent the world in any way, so long as they don't have any sensory input."
My somewhat half-baked thoughts on this (and I acknowledge in advance that there's some empirical evidence from LLMs that cuts against this):
If meaning is about the relationship between language and the world (which I think is correct) then to me that suggests at least a pretty high prior on needing sensory input. Maybe if we had the perfect scientific language dreamed of by the logical positivists you could train an LLM on a language whose internal structure was isomorphic enough to that of the world that learning the language would basically suffice; but one of the lessons of the early 20th C is that that dream doesn't work, at least not in any simple form. And certainly not with the actual language we have.
Another reason I think language alone might not be enough: there's a lot of information that can't be captured in language, even in principle: things like "knowing how" rather than "knowing that", or unspeakable physical information. The former also ties in to how we get feedback from the world: I imagine a lot of our training data is on things like "if I hear a sound, and then turn my head like so, the quality of the sound changes like so"--getting information from the feedback loop between our actions and our senses.
An LLM can't get this; it's feedback isn't directly from the world, but from linguistic representations of it, which per my first point is not nearly as good.
I'd also add you can (IMO) see the limits in learning just from linguistic structure in humans: even though science is highly mathematical we regularly find ourselves in need of experiment--we can't just use the structure of the formal language of the mathematics of our theory to advance our understanding of the world. I also think you can easily see people who get seduced by theories or representations of the world start to lose contact with it: I think even something like Goodheart's law often has this character, where we substitute some representation of what we care about in the world for the thing itself, and quickly discover that aiming at the representation comes apart from the thing.
None of this is to deny that representations are powerful, and reasoning using language and mathematics is an important component of reason--but I think we need to be regularly getting feedback from the world to actually keep our representations actually _representing_ the world.
> storing all their next-token predictions in a massive lookup table
LLMs don't make OR store next-token predictions, nor do they utilize any sort of lookup table. For every single token, in every single context, the model performs a fresh, forward pass through a massive network of matrices (weights) and nonlinear functions, resulting in a probability distribution over the vocabulary.
You'd think after 9 years, this would be common knowledge in the industry.
I never said that LLMs use a lookup table. I said IF they did so, I'd be sympathetic to those arguments, but since they don't, I'm not so sympathetic. The full quote is:
> "If LLMs were purely inductive models with no computational constraints, storing all their next-token predictions in a massive lookup table, and operating on an alien language with no semantic patterns in syntax, then yes, I would imagine they would fail pretty badly at understanding even if they become quite good mimics. But if LLMs receive any “help”, anything in their computational environment, training, or corpus that can link syntax to semantics, then these theoretical limits do not strictly apply."
It's still completely irrelevant because you're assuming that they train on syntax, which takes maybe 1,000 pages to learn, not trillions of tokens.
I'm not sure what you mean by "train on syntax." Elaborate?
Under some version of ultrafinitism, would we be safe from Gold’s theorem? It would seem so