I think you may be misunderstanding the nature of the paper here. The point isn't to point and laugh at the Go AI for being such a failure of an AI, ha ha ha! The point is that the resulting Go AI was still very good, even under the conditions it was limited to. I'm sure it could still beat a fair number of human players. So, we humans tend to assume that the Go AI has the same "shape" as a human player, just maybe not as good as the best ones. This is a demonstration that the resulting AI has a very different "shape" than a human player by demonstrating what in humans would be a gaping weakness even at very low levels of play.
And that's really all. It's not about goodness or badness, it's about "shape".
I scare quote shape because I don't have a good word for it. But you can get the same sense if you just sit down for a bit and start reading a lot of GPT-3 output, or interacting with it. On the one hand, GPT-3 is spectacular at writing sentences. It is better than quite a few humans! But on the other hand, if you sit down with it for a while, you'll start to notice there's something just a bit off about its output. It is impossible to put into words what that is, but you'll pick up on it, if you haven't already.
One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view. I personally think this may also be the source of why GPT-3 thinks it can just randomly introduce characters, locations, etc. whenever it feels like it, which is one of the "off" things it does. In real writing, such things are generally "established", but from GPT-3's point of view they are more often just introduced out of the blue.
A less concrete way in which it may be "off" is that while a given piece of text may have, let's say, 5 ways it may go, and then after a bit of continuation, there may be another 5 ways it can go, and so on and so forth, that doesn't mean those ways are uncorrelated. If I'm going for a humorous tone, I may have preferences, if serious, I may have other preferences, and so on. GPT-3 randomly picks these paths and does so in a way that no intentional human ever would at the paragraph scale. I wouldn't say it veers drunkenly around this sort of style matter, it's not that bad, but it's still just... off. This one is more subtle and harder to wrap words around.
I use GPT-3 as my example here because it's something you can interact with. The general point remains: Even as AIs are certainly improving (no denying that!), they continue to be very.... weird. There is something about all of them that is definitely not human. Clock some time with DALL-E and you'll see the same effect. And in this case, I'm not even talking about mere quality issues that may fixed over time... spot DALL-E the imperfections in the image and look just at the higher level abstractions of what it puts out. It's both very, very good, far better than I could dream of becoming any time soon myself... and yet, there's also something just off about it. (People mostly use DALL-E by generating lots of images then discarding most of them and picking the best. In this case, I want you to look at all the output.) This is that "offness" being expressed about a Go AI.
This is not even to say that that "offness" is objectively bad. I am not personally using the standard of "it must be exactly human to be AI". It is entirely plausible that these AIs will in fact be better by some standard than even an augmented human-like AI, or, to put it another way, it may well be that humans are the ones that are "off" in some way relative to some objective standard of performance in the end. (Evidence: A simple adversarial AI took apart the good AI. It is reasonable to think that a human might never have come up with the strategy that did so. I'm not counting on this, it could go either way, but it's reasonable. If true, a human would not be the benchmark of performance here!) If one imagines any of the three AIs simply being improved in whatever direction they are currently improving, they will certainly be yet more useful than they are today, even if they retain their "offness" or even see it expand.
Nevertheless, if one seeks accurate understanding of the AIs, understanding these issues is important, to use them better as engineering, to improve them in the future, and if pushed hard enough, to improve our own understanding of the human condition.
> There is something about all of them that is definitely not human
My "gut" says that current AI is very similar to some part of the human (or other species) brain, but that the (organic) mind substrate is not just more of the same, there are other modules that perform fundamentally different functions in a complementary way.
For an analogy, people tried for centuries to make a flying machine, but didn't have the complementary power source or perhaps the theory of governing it in flight. Better wings weren't the whole story.
I think that, in general, and particularly among futurists and AI enthusiasts, mental illness is considered uninteresting, but I believe studying abnormal brain functioning can potentially allow teasing out the separate parts of a mind that are difficult to distinguish when operating in unison.
Some of what I read about existing AI makes me think of "loose associations" and hallucinations - that maybe human minds have something similar in them which is only apparent when it's a bit out of sync with the rest of the mechanism.
Human minds also always occupy a social context, and discussion of AI that I read tends not to acknowledge this. It raises thorny questions - never mind whether a computer can or can't interact socially, why would we ever want it to? If it's not a joke, like Microsoft Bob, isn't it terrifying, a la the Terminator? But if it can't, then substituting for humans should be off the table.
Your point about the "shape" is interesting, and I think critical, to the future of AI (not to get hyperbolic or anything...).
For example, suppose we have a cancer-diagnosing/treatment planning algorithm. It's possible that it's much better than human doctors: out of a thousand patients, human doctors will save 300 and the algorithm 500; but also that the 500 is not a strict superset of the 300.
And to your point, it's possible that for some of the 300 that are not part of the group of 500, that the diagnosis/treatment recommended by the algorithm is obviously/hilariously wrong to a human.
If so, will we insert a human into the mix? How will we decide when it's correct for the human to override the algorithm? Because if they do all the time, we're back to the 300. And maybe the times when it's correct to override are not all obvious.
Or are we willing to simply accept the algorithm's judgment, knowing that an additional 200 will be saved? We know this is an unlikely outcome because a substantial portion of the population is unwilling to accept the idea that vaccines save more lives than they cost, simply because the lives they cost are different than the ones they save.
> One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view.
This description reminds me of simple Markov chains. You just ingest a bunch of text taking a window of, say, 10 characters and recording all the possible continuations thereof. So you might get [This remind] => "s" or such. Then you reverse that by picking a starting node and spinning text by picking a random continuation as you slide the window 'forward' to output.
I think there's something interesting in your post. However:
> The point is that the resulting Go AI was still very good, even under the conditions it was limited to. I'm sure it could still beat a fair number of human players.
If you mean the AI that they trained (the one that defeats KataGo) this is wrong. Look at the games: they're terrible: https://goattack.alignmentfund.org/.
No, I meant that KataGo is still very good. My apologies for the lack of clarity, I see how you could have read it that way. I do understand the adversarial AI is not good; that is in fact part of the "offness" I mean. Any AI that defeats something "truly" good should itself have to be "good", and yes, I know that's got enough mathematical fuzziness to drive a truck through, but I know we don't have the English vocabulary to make that statement rigorous and I am reasonably confident we don't even have the mathematical vocabulary to do it.
Thanks! In that case, the thing you say about KataGo can be strengthened:
> I'm sure it could still beat a fair number of human players.
KataGo can reliably beat any human player while giving them a handicap. The best pros lose a majority of games to a handful of top AI while receiving a 2 stone handicap, and are not locks to win with 3 stones.
Note: they did test two variants of KataGo, with and without search (search is very beneficial). Both versions are quite strong, and they had good results against both but they had their best results against the non-search version.
I understand the top comment as follows: The AIs were trained under one set of rules (remove obvious dead stones from your territory before counting) but are judged (in the paper) by another set of rules (if you have one opposing stone in your territory, that territory does not count).
Thus its no surprise that the AI can be attacked in this way: if you would apply the set of rules that it was trained with, all games from the paper would result in a (huge!) win for the AI.
And that's really all. It's not about goodness or badness, it's about "shape".
I scare quote shape because I don't have a good word for it. But you can get the same sense if you just sit down for a bit and start reading a lot of GPT-3 output, or interacting with it. On the one hand, GPT-3 is spectacular at writing sentences. It is better than quite a few humans! But on the other hand, if you sit down with it for a while, you'll start to notice there's something just a bit off about its output. It is impossible to put into words what that is, but you'll pick up on it, if you haven't already.
One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view. I personally think this may also be the source of why GPT-3 thinks it can just randomly introduce characters, locations, etc. whenever it feels like it, which is one of the "off" things it does. In real writing, such things are generally "established", but from GPT-3's point of view they are more often just introduced out of the blue.
A less concrete way in which it may be "off" is that while a given piece of text may have, let's say, 5 ways it may go, and then after a bit of continuation, there may be another 5 ways it can go, and so on and so forth, that doesn't mean those ways are uncorrelated. If I'm going for a humorous tone, I may have preferences, if serious, I may have other preferences, and so on. GPT-3 randomly picks these paths and does so in a way that no intentional human ever would at the paragraph scale. I wouldn't say it veers drunkenly around this sort of style matter, it's not that bad, but it's still just... off. This one is more subtle and harder to wrap words around.
I use GPT-3 as my example here because it's something you can interact with. The general point remains: Even as AIs are certainly improving (no denying that!), they continue to be very.... weird. There is something about all of them that is definitely not human. Clock some time with DALL-E and you'll see the same effect. And in this case, I'm not even talking about mere quality issues that may fixed over time... spot DALL-E the imperfections in the image and look just at the higher level abstractions of what it puts out. It's both very, very good, far better than I could dream of becoming any time soon myself... and yet, there's also something just off about it. (People mostly use DALL-E by generating lots of images then discarding most of them and picking the best. In this case, I want you to look at all the output.) This is that "offness" being expressed about a Go AI.
This is not even to say that that "offness" is objectively bad. I am not personally using the standard of "it must be exactly human to be AI". It is entirely plausible that these AIs will in fact be better by some standard than even an augmented human-like AI, or, to put it another way, it may well be that humans are the ones that are "off" in some way relative to some objective standard of performance in the end. (Evidence: A simple adversarial AI took apart the good AI. It is reasonable to think that a human might never have come up with the strategy that did so. I'm not counting on this, it could go either way, but it's reasonable. If true, a human would not be the benchmark of performance here!) If one imagines any of the three AIs simply being improved in whatever direction they are currently improving, they will certainly be yet more useful than they are today, even if they retain their "offness" or even see it expand.
Nevertheless, if one seeks accurate understanding of the AIs, understanding these issues is important, to use them better as engineering, to improve them in the future, and if pushed hard enough, to improve our own understanding of the human condition.