posted by ![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
ptc24 at 06:35pm on 19/03/2014
ptc24 at 06:35pm on 19/03/2014So my source was CopyCat, I was recently reading a book by Melanie Mitchell (Complexity: A Guided Tour) - she was the PhD student who did the work in Douglas Hofstadter's group. (Incidentally... somehow I feel I should get on better with Hofstadter than I do. There's something about his style or something that makes me less enthusiastic about him than I'd normally be for someone of his approximate description. I don't know...) She had a chapter where she got to describe her PhD, it sounded interesting. One of the interesting things was looking at some of the human results from this sort of thing.
There's a paper here
which suffers from the odd problem of sometimes rendering "ij" as "y" for some reason, which is a pain as it comes up a lot in the examples.
There are more examples here - it's interesting looking at the sorts of answers people give, and the sorts of answers that well-developed computer systems give at the end of their run. Best to do some examples, and make my points as I go.
Problem 1: abc -> abd, ijk -> ?. At the time of writing, everyone (in my test sample) went for ijl. It was the only one where the results were unanimous (with the possible exception of the people who refused to answer). Mitchell points out that some "smart aleck answers" are possible - ijd, abd and even ijk ("replace all c's with d's"). At least the first two, and possibly all three, are recognisable as better answers than randomly bashing away at a keyboard. Can we even define what a random answer is? Given that these answers have unbounded length (in some cases people give longer answers than the things in the question), you have the problem that almost all possible answers are stupidly long. Any answer you might possibly give before dropping dead of old age is better than almost all possible answers.
Other problems - aabc -> aabd, ijkk -> ?. As I was reading along, I thought... "oooh, there's something about the structure of that one, the structure of ijkk is the other way around - maybe they want me to do something to the front. hjkk I think - that preserves the pattern of spacings - jjkk would introduce some spurious new feature." This... seemed to be the sort of answer Mitchell was really trying to get. The way the chapter was written, ijkl was an acceptable answer, ijll was a bit better.
Often in her book, she'd present a case, I'd ponder it, come up with an answer, see what she said, and often she'd give one answer, say why lots of people gave that answer and why it was a good one, then give another answer that fewer people gave, and said why that one was more intriguing. Reading it... probably gave me a partly unfair sense of feeling clever; I'd had the advantage of being halfway through a Melanie Mitchell book when trying to answer them, so I had a better sense of the sort of thing she was angling at.
abc -> abd, kji -> ? - that's one where I thought there were two slightly clever answers, lji and kjh, and I couldn't tell between them.
mrrjjj and rssttt - Mitchell's thinking was that in rssttt there was the nice alphabetical progression, so people might go "obviously rssuuu then" whereas with mrrjjj there isn't really a pattern with the letters, so people will have to search harder for the idea, and might eventually settle on mrrjjjj. However, we only got one mrrjjjj - and an mrrnnnn - so that thought didn't really carry over to my convenience sample. With the other one, we got 3 cases of rssuuuu, and an rsstttt (from a person who answered mrrkkk to the other one), so there's a case of people not behaving as expected.
xyz was slightly unfair, in that the code had a 26-character alphabet that didn't wrap around, but there's no way for you to know that. xy{ was I think my favourite, followed by the "trolling just a little" xyaa. I think for this reason, there wasn't much wyz - which was the answer being angled for.
There are some bar graphs of answers the computer gives. An interesting thing here is with the distributions of results. It doesn't look like there's a small set of approximately-equal answers and nothing outside - instead, they look like those tailing-off distributions where a few response account for "the vast bulk" of the results, but there's no clear cut-off that lets you say what "a few" or "the vast bulk" really are. This lack of a "core set" makes it difficult to say that some results are "equally valid", as you need to stop somewhere, lest you end up saying that difa098yuas8sadawrfa-8sdhas is as good as ikl. If I write an analogy program, and you write an analogy program, and someone else goes and surveys a bunch of people, and we generate lists of results and how often those results are going to come up, I expect those lists would be pretty similar (and I know how I'd quantify that similarity too).
So what's a good answer? I'd argue that the goodness of an answer is task-dependent, but when the task is "following the argument in a Melanie Mitchell book" - then that makes things a bit circular. It seems to be a task that I'm good at, in that my preferred answers kept turning out to be the ones she really liked. I seemed to be going for something that made me think a mixture of "oooh, that's pretty" and "oooh, that's clever".
So, what was in here analogy model. It seems that analogy is way more complicated than similarity, at least in terms of writing passable routines to do a sort-of-OK-most-of-the-time job of it. There are nice simple approaches to similarity - represent your things as vectors in n-dimensional space, then take the cosine of the angle between them - which are simple enough that people like me keep implementing them over and over again. One of the things about similarity metrics - and quite a few other metrics - is that there's a variety to choose from and there isn't really one that's obviously the one true similarity metric (although in some fields it's pretty clear that cosine is the favourite).
With analogy - there's a huge complicated wobbly pile of stuff probably containing quite a few hand-tunable parameters and a whole lot of somewhat subjective decisions (like the hierarchy of "conceptual depth"). There are also several concepts which are justified or at least explained or at least described... by analogy. For example, it contains a concept of "temperature", which I think comes via simulated annealing, which itself is an analogy with real annealing (in some - but not all - versions, the analogy goes further, and various formulae are taken from statistical mechanics) - except the "temperature" isn't just something that gently falls, it also something which goes up or down based on how "happy" the system is with the sorts of answers it's pondering.
Anyway, analogy. It's sort of like induction but more so. It's unreliable and variable and has all sorts of subjective properties - but you really can't do without it at all, and there are a lot of regularities there too - at least if you're willing to count strong-but-not-absolute biases as "regularities" (ETA I should just mention here that there's no induction without inductive bias - there's another place we might make make an analogy-induction analogy). One particular application of analogies is in understanding other people - I think it's easier to understand and get on with someone if their analogies work the same way yours do. Analogies in moral reasoning I think are particularly interesting, but maybe another time.
There's a small point I'd like to make about culture. On the one hand, I can see plenty of room for cultural biases. On the other hand, one needs to be very very careful here lest one starts perpetuating some rather silly views. One thing to think about is what I call the "culture acquisition problem" - how do we learn our culture - most notably our language? If we're learning it by induction and analogy, then there must be, or have been, some original part of us, that induces and makes analogies while untainted by culture.
Going whooooooooosh! far off into the distance here: I had a thought that said, "If materialism is true, we can learn something about matter just by thinking". Of course this requires very loose definitions of most of the words after the comma, and I get annoyed when other people do that...
There's a paper here
which suffers from the odd problem of sometimes rendering "ij" as "y" for some reason, which is a pain as it comes up a lot in the examples.
There are more examples here - it's interesting looking at the sorts of answers people give, and the sorts of answers that well-developed computer systems give at the end of their run. Best to do some examples, and make my points as I go.
Problem 1: abc -> abd, ijk -> ?. At the time of writing, everyone (in my test sample) went for ijl. It was the only one where the results were unanimous (with the possible exception of the people who refused to answer). Mitchell points out that some "smart aleck answers" are possible - ijd, abd and even ijk ("replace all c's with d's"). At least the first two, and possibly all three, are recognisable as better answers than randomly bashing away at a keyboard. Can we even define what a random answer is? Given that these answers have unbounded length (in some cases people give longer answers than the things in the question), you have the problem that almost all possible answers are stupidly long. Any answer you might possibly give before dropping dead of old age is better than almost all possible answers.
Other problems - aabc -> aabd, ijkk -> ?. As I was reading along, I thought... "oooh, there's something about the structure of that one, the structure of ijkk is the other way around - maybe they want me to do something to the front. hjkk I think - that preserves the pattern of spacings - jjkk would introduce some spurious new feature." This... seemed to be the sort of answer Mitchell was really trying to get. The way the chapter was written, ijkl was an acceptable answer, ijll was a bit better.
Often in her book, she'd present a case, I'd ponder it, come up with an answer, see what she said, and often she'd give one answer, say why lots of people gave that answer and why it was a good one, then give another answer that fewer people gave, and said why that one was more intriguing. Reading it... probably gave me a partly unfair sense of feeling clever; I'd had the advantage of being halfway through a Melanie Mitchell book when trying to answer them, so I had a better sense of the sort of thing she was angling at.
abc -> abd, kji -> ? - that's one where I thought there were two slightly clever answers, lji and kjh, and I couldn't tell between them.
mrrjjj and rssttt - Mitchell's thinking was that in rssttt there was the nice alphabetical progression, so people might go "obviously rssuuu then" whereas with mrrjjj there isn't really a pattern with the letters, so people will have to search harder for the idea, and might eventually settle on mrrjjjj. However, we only got one mrrjjjj - and an mrrnnnn - so that thought didn't really carry over to my convenience sample. With the other one, we got 3 cases of rssuuuu, and an rsstttt (from a person who answered mrrkkk to the other one), so there's a case of people not behaving as expected.
xyz was slightly unfair, in that the code had a 26-character alphabet that didn't wrap around, but there's no way for you to know that. xy{ was I think my favourite, followed by the "trolling just a little" xyaa. I think for this reason, there wasn't much wyz - which was the answer being angled for.
There are some bar graphs of answers the computer gives. An interesting thing here is with the distributions of results. It doesn't look like there's a small set of approximately-equal answers and nothing outside - instead, they look like those tailing-off distributions where a few response account for "the vast bulk" of the results, but there's no clear cut-off that lets you say what "a few" or "the vast bulk" really are. This lack of a "core set" makes it difficult to say that some results are "equally valid", as you need to stop somewhere, lest you end up saying that difa098yuas8sadawrfa-8sdhas is as good as ikl. If I write an analogy program, and you write an analogy program, and someone else goes and surveys a bunch of people, and we generate lists of results and how often those results are going to come up, I expect those lists would be pretty similar (and I know how I'd quantify that similarity too).
So what's a good answer? I'd argue that the goodness of an answer is task-dependent, but when the task is "following the argument in a Melanie Mitchell book" - then that makes things a bit circular. It seems to be a task that I'm good at, in that my preferred answers kept turning out to be the ones she really liked. I seemed to be going for something that made me think a mixture of "oooh, that's pretty" and "oooh, that's clever".
So, what was in here analogy model. It seems that analogy is way more complicated than similarity, at least in terms of writing passable routines to do a sort-of-OK-most-of-the-time job of it. There are nice simple approaches to similarity - represent your things as vectors in n-dimensional space, then take the cosine of the angle between them - which are simple enough that people like me keep implementing them over and over again. One of the things about similarity metrics - and quite a few other metrics - is that there's a variety to choose from and there isn't really one that's obviously the one true similarity metric (although in some fields it's pretty clear that cosine is the favourite).
With analogy - there's a huge complicated wobbly pile of stuff probably containing quite a few hand-tunable parameters and a whole lot of somewhat subjective decisions (like the hierarchy of "conceptual depth"). There are also several concepts which are justified or at least explained or at least described... by analogy. For example, it contains a concept of "temperature", which I think comes via simulated annealing, which itself is an analogy with real annealing (in some - but not all - versions, the analogy goes further, and various formulae are taken from statistical mechanics) - except the "temperature" isn't just something that gently falls, it also something which goes up or down based on how "happy" the system is with the sorts of answers it's pondering.
Anyway, analogy. It's sort of like induction but more so. It's unreliable and variable and has all sorts of subjective properties - but you really can't do without it at all, and there are a lot of regularities there too - at least if you're willing to count strong-but-not-absolute biases as "regularities" (ETA I should just mention here that there's no induction without inductive bias - there's another place we might make make an analogy-induction analogy). One particular application of analogies is in understanding other people - I think it's easier to understand and get on with someone if their analogies work the same way yours do. Analogies in moral reasoning I think are particularly interesting, but maybe another time.
There's a small point I'd like to make about culture. On the one hand, I can see plenty of room for cultural biases. On the other hand, one needs to be very very careful here lest one starts perpetuating some rather silly views. One thing to think about is what I call the "culture acquisition problem" - how do we learn our culture - most notably our language? If we're learning it by induction and analogy, then there must be, or have been, some original part of us, that induces and makes analogies while untainted by culture.
Going whooooooooosh! far off into the distance here: I had a thought that said, "If materialism is true, we can learn something about matter just by thinking". Of course this requires very loose definitions of most of the words after the comma, and I get annoyed when other people do that...
(no subject)
I'm quite inclined to think of the problem in a more holistic style, thinking of the whole space of transformations instead of one specific element of it: "if abc goes to abd, what (plausible-to-humans) mapping function might that be an individual element of?" And once you ask the question that way, you can imagine assorted generalisations of "abc → abd" that vary in their domain of applicability, and (at least along the Pareto frontier) vary in a precisely opposed fashion in their recognisable similarity to the original archetype element. In that spirit I answered 'EDOM' to one of your questions, because the idea of 'generalise abc → abd' I had in my mind at the time only went so far, which was the price it paid for feeling reasonably sensible in the areas it did reach. The Feynman answer of generalising 'abc → abd' to the function 'X → abd, for all X' is listed in your post as 'smart-aleck', but of course one advantage it does have is full generality! There's no input for which you can scratch your head and wonder how to best apply the underlying rule, and no input for which you give up and say 'no answer exists, try something actually in the domain of my function'.
So asking for several individual values in sequence caused me to repeatedly refine my idea of the mapping function I had in mind, and I probably didn't end up with there actually being a single sensible function that all my answers were consistent with. But that's an effect of the sequential presentation, not (or at least not obviously) of any genuine inconsistency in my thinking.
(no subject)
Feynman - if you're doing science, then arguably by definition you get to test your hypotheses, and the smart-aleck hypotheses have got to be worth testing.
(no subject)
*nods* Yes, I had that thought shortly after hitting send. Perhaps if you're approaching the analogy problem with the mind of an experimental physicist, you treat the premise abc → abd as an experimental observation, and the question becomes, 'Given that observation, what law of nature is most likely?' And then you take the Occam's Razor approach, and pick the simplest one that fits the observations – until, of course, further experimental results are obtained and the explanations that are too simple begin to be ruled out.
Perhaps Feynman's position, though on the surface it had what Hofstadter called a 'village idiot' nature about it (a curious contrast to your characterisation as 'smart aleck'!), would have revealed huge depths of subtlety had Hofstadter only thought to add one or two more exemplars to his analogy problems: 'If abc → abd and also def → deg, what does this or that go to? Do your answers change if it is later revealed that abd → abc? Or dba → cba?'
(no subject)
* That's because it was in a non-useful context. If you're learning a language, then you do these contextual deductions, but there are a) plenty of examples and b) the opportunity to check very quickly whether your deduction was correct, or whether you just said there was a frog in your bidet.
(no subject)
I found abc -> abd, mrrjjj -> ? the most interesting case, and was determined to find a sensible pattern in it. My answer of mrrnnnn was motivated by the pattern I decided upon: here, abc is paralleled by mrrjjj in the sense that each is composed of three members of a series: (a)(b)(c) and (m)(rr)(jjj). In abc one adjusts the initial member a to form each later member by [1] incrementing the alphabetic position by +1, +2, +3 etc. and [2] incrementing the number of letters composing the member by 0. In mrrjjj one adjusts the initial member m to form each later member by [1] incrementing the alphabetic position by +5, -3, +1 etc. and [2] incrementing the number of letters composing the member by +1, +2, +3 etc. The transformation abc -> abd means advancing the third member by one place in the series, and so I applied the same transformation to mrrjjj: the fourth member would increment m by +1 alphabetically and increment the number of identical letters composing it by +3, therefore mrrnnnn. (I followed a similar rationale for the simpler abc -> abd, rssttt -> ?, and in that case my answer was shared by two others.)
(no subject)