{"id":2132,"date":"2014-06-10T11:26:52","date_gmt":"2014-06-10T15:26:52","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/pamphlet\/?p=2132"},"modified":"2015-02-08T21:40:34","modified_gmt":"2015-02-09T02:40:34","slug":"no-the-turing-test-has-not-been-passed","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2014\/06\/10\/no-the-turing-test-has-not-been-passed\/","title":{"rendered":"No, the Turing Test has not been passed."},"content":{"rendered":"\n\n\n\n\n
\"Turing<\/a><\/td>\n<\/tr>\n
\u2026that’s not Turing’s Test\u2026<\/span>
\n\u201cTuring Test<\/a>\u201d image from xkcd<\/a>. Used by permission<\/a>.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

There has been a flurry of interest in the Turing Test in the last few days, precipitated by a claim that (at last!) a program has passed the Test. The program in question is called “Eugene Goostman” and the claim is promulgated by Kevin Warwick, a professor of cybernetics at the University of Reading and organizer of a recent chatbot competition there.<\/p>\n

The Turing Test is a topic that I have a deep interest in (see this<\/a>, and this<\/a>, and this<\/a>, and this<\/a>, and, most recently, this<\/a>), so I thought to give my view on Professor Warwick’s claim<\/a> “We are therefore proud to declare that Alan Turing’s Test was passed for the first time on Saturday.” The main points are these. The Turing Test was not passed on Saturday, and “Eugene Goostman” seems to perform qualitatively about as poorly as many other chatbots in emulating human verbal behavior. In summary: There’s nothing new here; move along.<\/p>\n

First, the Turing Test that Turing had in mind was a criterion of indistinguishability<\/em> in verbal performance between human and computer in an open-ended wide-ranging interaction. In order for the Test to be passed, judges had to perform no better than chance<\/em> in unmasking the computer. But in the recent event, the interactions were quite time-limited (only five minutes) and in any case, the purported Turing-Test-passing program was identified correctly more often than not by the judges (almost 70% of the time in fact). That’s not Turing’s test.<\/p>\n

Update June 17, 2014<\/strong>: The time limitation was even worse than I thought. According to my colleague Luke Hunsberger<\/a>, computer science professor at Vassar College, who was a judge in this event, the five minute time limit was for two<\/em> simultaneous interactions. Further, there were often substantial response delays in the system. In total, he estimated that a judge might average only four or five rounds of chat\u00a0with each interlocutor. I\u2019ve argued before that a grossly time-limited Turing Test is no Turing Test at all<\/a>.<\/p><\/blockquote>\n

Sometimes, people trot out the prediction from Turing’s seminal 1950 Mind<\/em> article that “I believe that in about fifty years\u2019 time it will be possible to programme computers, with a storage capacity of about \\(10^9\\), to make them play the imitation game so well that an average interrogator will not have more than 70 per cent. chance of making the right identification after five minutes of questioning.” As I explain in my book on the Test<\/a>:<\/p>\n

The first thing to note about the prediction is that it is not a prediction about the Test per se: Turing expects 70 percent prediction accuracy, not the more difficult 50 percent expected by chance, and this after only a limited conversation of five minutes. He is therefore predicting passage of a test much simpler than the Test.<\/p><\/blockquote>\n

Not only does the prediction not presuppose a full Turing Test, but it could well be argued that it had already come to pass with the advent of Joseph Weizenbaum\u2019s Eliza some thirty-five years early. Weizenbaum developed a simple computer program that attempted to imitate the parroting conversational style of a Rogerian psychotherapist…. Although the methods used were quite simple \u2013 repeating the user\u2019s question after adjusting some pronouns, throwing in a stock phrase every now and then \u2013 the result was, in its own way, extremely convincing.<\/p><\/blockquote>\n

Second, “Eugene Goostman” uses the very techniques that began with\u00a0Weizenbaum’s \u00a0“Eliza” program<\/a> from the 1960’s. We see the same tricks \u2013 repeating the judge’s statements with simple substitutions, keyword-triggered responses, falling back on vague or unresponsive replies, and the like. Those tricks are no more successful than they have been in the two decades of runnings of the Loebner Prize Competition, another ill-conceived attempt at running a Turing-like test<\/a>. And there too, entrants used the trick of having their programs emulate humans with built in excuses. \u00a0“Eugene Goostman” purports to be a non-English-fluent child, rather than a fluent adult.1<\/a><\/sup><\/p>\n

In fact, anyone with the slightest familiarity with chatbots, the\u00a0modern incarnations of “Eliza”,\u00a0could unmask “Eugene Goostman” quickly. (The judges at the recent competition were apparently not so familiar.) Scott Aaronson has provided a transcript<\/a> of his own interaction with “Eugene Goostman”, which shows the telltale signs as soon as the computer’s first reply, which blatantly ignores an utterly trivial question and tries to change the subject:<\/p>\n

Scott: Which is bigger, a shoebox or Mount Everest?<\/p>\n

Eugene: I can\u2019t make a choice right now. I should think it out later. And I forgot to ask you where you are from\u2026<\/p><\/blockquote>\n

Here’s an appeal to cute, vague language:<\/p>\n

Scott: How many legs does a camel have?<\/p>\n

Eugene: Something between 2 and 4. Maybe, three? :-))) By the way, I still don\u2019t know your specialty \u2013 or, possibly, I\u2019ve missed it?<\/p><\/blockquote>\n

And here’s the repetition of a canned response:<\/p>\n

Scott: No, I need to know that you\u2019re not a chatbot. Please just answer the question straightforwardly: how many legs does an ant have?<\/p>\n

Eugene: Something between 2 and 4. Maybe, three? :-))) Oh, what a fruitful conversation;-)<\/p><\/blockquote>\n

Here’s a canned keyword-triggered response, nonresponsive as usual:<\/p>\n

Scott: In your opinion, does your existence demonstrate any idea or principle that wasn\u2019t demonstrated just as convincingly by ELIZA, Joseph Weizenbaum\u2019s chatbot from the 1960s?<\/p>\n

Eugene: Eliza was a break-thru. All the bots after it were nothing but weak parodies, claiming to have \u201crevolutionary improvements\u201d.<\/p><\/blockquote>\n

And here’s “Eliza”‘s trick of rewriting the judge’s utterance after pronoun replacement and (poorly executed) synonym substitution:<\/p>\n

Scott: OK, I\u2019ll count that as the second sensible thing you\u2019ve said (grading generously). Don\u2019t you think that a more convincing chatbot could be created, by using Google to trawl the entire web for plausible-sounding answers (analogous to what IBM\u2019s Watson did)?<\/p>\n

Eugene: Yes, I do think that a more convincing chatbot can be built by using google to trawl the all web for plausible \u2013 sounding \u2013 I hope I have right to have my own opinion. Maybe, let\u2019s talk about something else? What would you like to discuss?<\/p><\/blockquote>\n

Literally every one of “Eugene”‘s responses reflects its “Eliza”-like programming. It would be amusing, if it weren’t so predictable.<\/p>\n

In summary, “Eugene Goostman” is not qualitatively superior to other chatbots, and certainly has not passed a true Turing Test. It isn’t even close.<\/p>\n

\n
\n
    \n
  1. In a parody of this approach, the late John McCarthy, professor of computer science at Stanford University and inventor of the term “artifical intelligence”, wrote a letter to the editor responding to a publication about an “Eliza”-like program that claimed to emulate a paranoid psychiatric patient. He presented his own experiments that I described in my Turing Test book<\/a>: “He had designed an even better program, which passed the same test. His also had the virtue of being a very inexpensive program, in these times of tight money. In fact you didn\u2019t even need a computer for it. All you needed was an electric typewriter. His program modeled infantile autism. And the transcripts \u2013 you type in your questions, and the thing just sits there and hums \u2013 cannot be distinguished by experts from transcripts of real conversations with infantile autistic patients.”\u21a9<\/a><\/li>\n<\/ol>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    \u2026that’s not Turing’s Test\u2026 \u201cTuring Test\u201d image from xkcd. Used by permission. There has been a flurry of interest in the Turing Test in the last few days, precipitated by a claim that (at last!) a program has passed the Test. The program in question is called “Eugene Goostman” and the claim is promulgated by […]<\/p>\n","protected":false},"author":2110,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[13256,380,6027],"tags":[],"class_list":["post-2132","post","type-post","status-publish","format-standard","hentry","category-alan-turing","category-computer-science","category-linguistics"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5pLfN-yo","jetpack-related-posts":[{"id":2271,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2015\/05\/07\/in-support-of-behavioral-tests-of-intelligence\/","url_meta":{"origin":2132,"position":0},"title":"In support of behavioral tests of intelligence","author":"Stuart Shieber","date":"Thursday, May 7, 2015","format":false,"excerpt":"\u2026\u201cblockhead\u201d argument\u2026 \"Blockhead by Paul McCarthy @ Tate Modern\" image from flickr user Matt Hobbs. Used by permission. Alan Turing proposed what is the best known criterion for attributing intelligence, the capacity for thinking, to a computer. We call it the Turing Test, and it involves comparing the computer\u2019s verbal\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1481,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2012\/06\/16\/talmud-and-the-turing-test\/","url_meta":{"origin":2132,"position":1},"title":"Talmud and the Turing Test","author":"Stuart Shieber","date":"Saturday, June 16, 2012","format":false,"excerpt":"...the Golem... Image of the statue of the Golem of Prague at the entrance to the Jewish Quarter of Prague by flickr user D_P_R. Used by permission (CC-BY 2.0). Alan Turing, the patron saint of computer science, was born 100 years ago this week (June 23). I\u2019ll be attending the\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":321,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2009\/09\/13\/britain-apologizes-for-treatment-of-alan-turing\/","url_meta":{"origin":2132,"position":2},"title":"Britain apologizes for treatment of Alan Turing","author":"Stuart Shieber","date":"Sunday, September 13, 2009","format":false,"excerpt":"Image by Whimsical Chris via Flickr Prime Minister Gordon Brown has apologized on behalf of the British government for the appalling treatment of Alan Turing, who was obliged to undergo chemical castration for the crime of being gay. Prime Minister Brown's statement in the Telegraph follows an online petition drive\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"Alan Turing","src":"https:\/\/i0.wp.com\/farm3.static.flickr.com\/2081\/2042538753_f102fe97df_m.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":2209,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2014\/11\/30\/the-turing-moment\/","url_meta":{"origin":2132,"position":3},"title":"The Turing moment","author":"Stuart Shieber","date":"Sunday, November 30, 2014","format":false,"excerpt":"\u2026less histrionic\u2026 Ed Stoppard as Alan Turing in Codebreaker We seem to be at the \u201cTuring moment\u201d, what with Benedict Cumberbatch, erstwhile Sherlock Holmes, now starring as a Hollywood Alan Turing in The Imitation Game. The release culminates a series of Turing-related events over the last few years. The centennial\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2295,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2015\/08\/03\/becoming-tin-men\/","url_meta":{"origin":2132,"position":4},"title":"Becoming tin men","author":"Stuart Shieber","date":"Monday, August 3, 2015","format":false,"excerpt":"From the 2015 introduction to the 1965 novel The Tin Men by Michael Frayn: \"I hadn't in those days heard of the Turing Test\u2014Alan Turing's proposal that a computer could be said to think if its conversational powers were shown to be indistinguishable from a human being's\u2014so I didn't realise\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2218,"url":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/2015\/02\/18\/the-two-guildford-mathematicians\/","url_meta":{"origin":2132,"position":5},"title":"The two Guildford mathematicians","author":"Stuart Shieber","date":"Wednesday, February 18, 2015","format":false,"excerpt":"\u2026the huge ledger\u2026 Still from Codebreaker showing Turing's checkout of three Carroll books. The charming town of Guildford, 40 minutes southwest of London on South West Trains, is associated with two famous British logician-mathematicians. Alan Turing (on whom I seem to perseverate) spent time there after 1927, when his parents\u2026","rel":"","context":"In "Alan Turing"","block_context":{"text":"Alan Turing","link":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/category\/computer-science\/alan-turing\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/2132","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/users\/2110"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/comments?post=2132"}],"version-history":[{"count":15,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/2132\/revisions"}],"predecessor-version":[{"id":2149,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/posts\/2132\/revisions\/2149"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/media?parent=2132"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/categories?post=2132"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/pamphlet\/wp-json\/wp\/v2\/tags?post=2132"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}