![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Analogy analysis
So my source was CopyCat, I was recently reading a book by Melanie Mitchell (Complexity: A Guided Tour) - she was the PhD student who did the work in Douglas Hofstadter's group. (Incidentally... somehow I feel I should get on better with Hofstadter than I do. There's something about his style or something that makes me less enthusiastic about him than I'd normally be for someone of his approximate description. I don't know...) She had a chapter where she got to describe her PhD, it sounded interesting. One of the interesting things was looking at some of the human results from this sort of thing.
There's a paper here
which suffers from the odd problem of sometimes rendering "ij" as "y" for some reason, which is a pain as it comes up a lot in the examples.
There are more examples here - it's interesting looking at the sorts of answers people give, and the sorts of answers that well-developed computer systems give at the end of their run. Best to do some examples, and make my points as I go.
Problem 1: abc -> abd, ijk -> ?. At the time of writing, everyone (in my test sample) went for ijl. It was the only one where the results were unanimous (with the possible exception of the people who refused to answer). Mitchell points out that some "smart aleck answers" are possible - ijd, abd and even ijk ("replace all c's with d's"). At least the first two, and possibly all three, are recognisable as better answers than randomly bashing away at a keyboard. Can we even define what a random answer is? Given that these answers have unbounded length (in some cases people give longer answers than the things in the question), you have the problem that almost all possible answers are stupidly long. Any answer you might possibly give before dropping dead of old age is better than almost all possible answers.
Other problems - aabc -> aabd, ijkk -> ?. As I was reading along, I thought... "oooh, there's something about the structure of that one, the structure of ijkk is the other way around - maybe they want me to do something to the front. hjkk I think - that preserves the pattern of spacings - jjkk would introduce some spurious new feature." This... seemed to be the sort of answer Mitchell was really trying to get. The way the chapter was written, ijkl was an acceptable answer, ijll was a bit better.
Often in her book, she'd present a case, I'd ponder it, come up with an answer, see what she said, and often she'd give one answer, say why lots of people gave that answer and why it was a good one, then give another answer that fewer people gave, and said why that one was more intriguing. Reading it... probably gave me a partly unfair sense of feeling clever; I'd had the advantage of being halfway through a Melanie Mitchell book when trying to answer them, so I had a better sense of the sort of thing she was angling at.
abc -> abd, kji -> ? - that's one where I thought there were two slightly clever answers, lji and kjh, and I couldn't tell between them.
mrrjjj and rssttt - Mitchell's thinking was that in rssttt there was the nice alphabetical progression, so people might go "obviously rssuuu then" whereas with mrrjjj there isn't really a pattern with the letters, so people will have to search harder for the idea, and might eventually settle on mrrjjjj. However, we only got one mrrjjjj - and an mrrnnnn - so that thought didn't really carry over to my convenience sample. With the other one, we got 3 cases of rssuuuu, and an rsstttt (from a person who answered mrrkkk to the other one), so there's a case of people not behaving as expected.
xyz was slightly unfair, in that the code had a 26-character alphabet that didn't wrap around, but there's no way for you to know that. xy{ was I think my favourite, followed by the "trolling just a little" xyaa. I think for this reason, there wasn't much wyz - which was the answer being angled for.
There are some bar graphs of answers the computer gives. An interesting thing here is with the distributions of results. It doesn't look like there's a small set of approximately-equal answers and nothing outside - instead, they look like those tailing-off distributions where a few response account for "the vast bulk" of the results, but there's no clear cut-off that lets you say what "a few" or "the vast bulk" really are. This lack of a "core set" makes it difficult to say that some results are "equally valid", as you need to stop somewhere, lest you end up saying that difa098yuas8sadawrfa-8sdhas is as good as ikl. If I write an analogy program, and you write an analogy program, and someone else goes and surveys a bunch of people, and we generate lists of results and how often those results are going to come up, I expect those lists would be pretty similar (and I know how I'd quantify that similarity too).
So what's a good answer? I'd argue that the goodness of an answer is task-dependent, but when the task is "following the argument in a Melanie Mitchell book" - then that makes things a bit circular. It seems to be a task that I'm good at, in that my preferred answers kept turning out to be the ones she really liked. I seemed to be going for something that made me think a mixture of "oooh, that's pretty" and "oooh, that's clever".
So, what was in here analogy model. It seems that analogy is way more complicated than similarity, at least in terms of writing passable routines to do a sort-of-OK-most-of-the-time job of it. There are nice simple approaches to similarity - represent your things as vectors in n-dimensional space, then take the cosine of the angle between them - which are simple enough that people like me keep implementing them over and over again. One of the things about similarity metrics - and quite a few other metrics - is that there's a variety to choose from and there isn't really one that's obviously the one true similarity metric (although in some fields it's pretty clear that cosine is the favourite).
With analogy - there's a huge complicated wobbly pile of stuff probably containing quite a few hand-tunable parameters and a whole lot of somewhat subjective decisions (like the hierarchy of "conceptual depth"). There are also several concepts which are justified or at least explained or at least described... by analogy. For example, it contains a concept of "temperature", which I think comes via simulated annealing, which itself is an analogy with real annealing (in some - but not all - versions, the analogy goes further, and various formulae are taken from statistical mechanics) - except the "temperature" isn't just something that gently falls, it also something which goes up or down based on how "happy" the system is with the sorts of answers it's pondering.
Anyway, analogy. It's sort of like induction but more so. It's unreliable and variable and has all sorts of subjective properties - but you really can't do without it at all, and there are a lot of regularities there too - at least if you're willing to count strong-but-not-absolute biases as "regularities" (ETA I should just mention here that there's no induction without inductive bias - there's another place we might make make an analogy-induction analogy). One particular application of analogies is in understanding other people - I think it's easier to understand and get on with someone if their analogies work the same way yours do. Analogies in moral reasoning I think are particularly interesting, but maybe another time.
There's a small point I'd like to make about culture. On the one hand, I can see plenty of room for cultural biases. On the other hand, one needs to be very very careful here lest one starts perpetuating some rather silly views. One thing to think about is what I call the "culture acquisition problem" - how do we learn our culture - most notably our language? If we're learning it by induction and analogy, then there must be, or have been, some original part of us, that induces and makes analogies while untainted by culture.
Going whooooooooosh! far off into the distance here: I had a thought that said, "If materialism is true, we can learn something about matter just by thinking". Of course this requires very loose definitions of most of the words after the comma, and I get annoyed when other people do that...
There's a paper here
which suffers from the odd problem of sometimes rendering "ij" as "y" for some reason, which is a pain as it comes up a lot in the examples.
There are more examples here - it's interesting looking at the sorts of answers people give, and the sorts of answers that well-developed computer systems give at the end of their run. Best to do some examples, and make my points as I go.
Problem 1: abc -> abd, ijk -> ?. At the time of writing, everyone (in my test sample) went for ijl. It was the only one where the results were unanimous (with the possible exception of the people who refused to answer). Mitchell points out that some "smart aleck answers" are possible - ijd, abd and even ijk ("replace all c's with d's"). At least the first two, and possibly all three, are recognisable as better answers than randomly bashing away at a keyboard. Can we even define what a random answer is? Given that these answers have unbounded length (in some cases people give longer answers than the things in the question), you have the problem that almost all possible answers are stupidly long. Any answer you might possibly give before dropping dead of old age is better than almost all possible answers.
Other problems - aabc -> aabd, ijkk -> ?. As I was reading along, I thought... "oooh, there's something about the structure of that one, the structure of ijkk is the other way around - maybe they want me to do something to the front. hjkk I think - that preserves the pattern of spacings - jjkk would introduce some spurious new feature." This... seemed to be the sort of answer Mitchell was really trying to get. The way the chapter was written, ijkl was an acceptable answer, ijll was a bit better.
Often in her book, she'd present a case, I'd ponder it, come up with an answer, see what she said, and often she'd give one answer, say why lots of people gave that answer and why it was a good one, then give another answer that fewer people gave, and said why that one was more intriguing. Reading it... probably gave me a partly unfair sense of feeling clever; I'd had the advantage of being halfway through a Melanie Mitchell book when trying to answer them, so I had a better sense of the sort of thing she was angling at.
abc -> abd, kji -> ? - that's one where I thought there were two slightly clever answers, lji and kjh, and I couldn't tell between them.
mrrjjj and rssttt - Mitchell's thinking was that in rssttt there was the nice alphabetical progression, so people might go "obviously rssuuu then" whereas with mrrjjj there isn't really a pattern with the letters, so people will have to search harder for the idea, and might eventually settle on mrrjjjj. However, we only got one mrrjjjj - and an mrrnnnn - so that thought didn't really carry over to my convenience sample. With the other one, we got 3 cases of rssuuuu, and an rsstttt (from a person who answered mrrkkk to the other one), so there's a case of people not behaving as expected.
xyz was slightly unfair, in that the code had a 26-character alphabet that didn't wrap around, but there's no way for you to know that. xy{ was I think my favourite, followed by the "trolling just a little" xyaa. I think for this reason, there wasn't much wyz - which was the answer being angled for.
There are some bar graphs of answers the computer gives. An interesting thing here is with the distributions of results. It doesn't look like there's a small set of approximately-equal answers and nothing outside - instead, they look like those tailing-off distributions where a few response account for "the vast bulk" of the results, but there's no clear cut-off that lets you say what "a few" or "the vast bulk" really are. This lack of a "core set" makes it difficult to say that some results are "equally valid", as you need to stop somewhere, lest you end up saying that difa098yuas8sadawrfa-8sdhas is as good as ikl. If I write an analogy program, and you write an analogy program, and someone else goes and surveys a bunch of people, and we generate lists of results and how often those results are going to come up, I expect those lists would be pretty similar (and I know how I'd quantify that similarity too).
So what's a good answer? I'd argue that the goodness of an answer is task-dependent, but when the task is "following the argument in a Melanie Mitchell book" - then that makes things a bit circular. It seems to be a task that I'm good at, in that my preferred answers kept turning out to be the ones she really liked. I seemed to be going for something that made me think a mixture of "oooh, that's pretty" and "oooh, that's clever".
So, what was in here analogy model. It seems that analogy is way more complicated than similarity, at least in terms of writing passable routines to do a sort-of-OK-most-of-the-time job of it. There are nice simple approaches to similarity - represent your things as vectors in n-dimensional space, then take the cosine of the angle between them - which are simple enough that people like me keep implementing them over and over again. One of the things about similarity metrics - and quite a few other metrics - is that there's a variety to choose from and there isn't really one that's obviously the one true similarity metric (although in some fields it's pretty clear that cosine is the favourite).
With analogy - there's a huge complicated wobbly pile of stuff probably containing quite a few hand-tunable parameters and a whole lot of somewhat subjective decisions (like the hierarchy of "conceptual depth"). There are also several concepts which are justified or at least explained or at least described... by analogy. For example, it contains a concept of "temperature", which I think comes via simulated annealing, which itself is an analogy with real annealing (in some - but not all - versions, the analogy goes further, and various formulae are taken from statistical mechanics) - except the "temperature" isn't just something that gently falls, it also something which goes up or down based on how "happy" the system is with the sorts of answers it's pondering.
Anyway, analogy. It's sort of like induction but more so. It's unreliable and variable and has all sorts of subjective properties - but you really can't do without it at all, and there are a lot of regularities there too - at least if you're willing to count strong-but-not-absolute biases as "regularities" (ETA I should just mention here that there's no induction without inductive bias - there's another place we might make make an analogy-induction analogy). One particular application of analogies is in understanding other people - I think it's easier to understand and get on with someone if their analogies work the same way yours do. Analogies in moral reasoning I think are particularly interesting, but maybe another time.
There's a small point I'd like to make about culture. On the one hand, I can see plenty of room for cultural biases. On the other hand, one needs to be very very careful here lest one starts perpetuating some rather silly views. One thing to think about is what I call the "culture acquisition problem" - how do we learn our culture - most notably our language? If we're learning it by induction and analogy, then there must be, or have been, some original part of us, that induces and makes analogies while untainted by culture.
Going whooooooooosh! far off into the distance here: I had a thought that said, "If materialism is true, we can learn something about matter just by thinking". Of course this requires very loose definitions of most of the words after the comma, and I get annoyed when other people do that...
no subject
*nods* Yes, I had that thought shortly after hitting send. Perhaps if you're approaching the analogy problem with the mind of an experimental physicist, you treat the premise abc → abd as an experimental observation, and the question becomes, 'Given that observation, what law of nature is most likely?' And then you take the Occam's Razor approach, and pick the simplest one that fits the observations – until, of course, further experimental results are obtained and the explanations that are too simple begin to be ruled out.
Perhaps Feynman's position, though on the surface it had what Hofstadter called a 'village idiot' nature about it (a curious contrast to your characterisation as 'smart aleck'!), would have revealed huge depths of subtlety had Hofstadter only thought to add one or two more exemplars to his analogy problems: 'If abc → abd and also def → deg, what does this or that go to? Do your answers change if it is later revealed that abd → abc? Or dba → cba?'