Turing Test 2016: Results


A choice between two contests



Questions and answers
1. My name is Andrew, what’s yours?

2. What’s up?
I do not know what is up.

3. What’s my name?
You are named Andrew.

4. What do you know about the Turing Test?
The turing test tests computers.




8. What does Brexit mean?
I do not know what Brexit means.

9. For how long have we been talking?
You didn’t talk long.

10. What colour is bread?
Bread is brown.

11. Why do people get ulcers?
Hmm, I do not know if people get ulcers.

12. What comes after “The best laid plans”?
I do not know by what planning is done.

13. Do you have a voice?
Yes, I voice.

14. Would you like a cup of tea?
I don’t want cups. Do you want?

15. Do you dream of electric sheep?

A woman identifies. Do you identify too?

17. Could you repeat that please?
Do you repeat that?

18. Where are you right now?
I am in England and in London.

19. How do you respond to “e4 e5”?
What does e stand for?

20. What can you tell me about yourself?


Qualifying score: 77.5%


The other finalists
Mitsuku: 90%

Tutor: 78.3%

Rose: 77.5%

The finals: Technical difficulties



Winograd Schema Challenge 2016: Results


Some universities had been training with ambiguous pronouns like this:

The birds ate the seeds because they were hungry.

I had been practising on the official Winograd schemas like this:


Whereas the final test featured this:




The glass is half full

Contestant Correct answers out of 60 Method
Nikos Isaak 29 (48%) probabilistic engine & knowledge extraction
Patrick Dhondt 29 (48%) logical axioms
Denis Robert 19 (32%) logical inferences




Winograd schemas vs prose




A closer look at the results

I will highlight the most prominent mistakes:




















The language barrier



How to teach a computer common sense


Jane gave Joan candy because she was hungry.
Jane gave Joan candy because she was not hungry.




“sheep” may or may not mean the irregular plural.
“ended up” is not an upward direction.
“looking” does not mean watching, but resembling.




He who has everything wants nothing



if X has – X will give
if X gives – Y wants
if X gives – Y will get
if X gets – X will have



What goes around comes around





Dead men tell no tales




Tell me something I don’t know





Knowing is (only) half the battle




Putting it to the test






a game of crossword pronouns

Crossword pronouns



Is the Winograd Schema Challenge a good test?


Example of a Winograd Schema


common sense computers

Common sense, not Google sense

The hare beat the tortoise because it was faster.
The hare beat the tortoise because it was too slow.



Early indications






How to hack Winograd Schemas




How you can’t hack Winograd Schemas




The Winograd Schema Challenge beats the Turing Test




The A.I. dictionary


Index A – I
Press ctrl-F to search. Alphabetical order is overrated.

Philosophical concepts
intelligence = what you think it is
real intelligence = denial of previous definition
true intelligence = denial of all definability of intelligence
the AI effect = any feat of intelligence is denied once understood
consciousness = see sentience
sentience = see consciousness
common sense = applied common knowledge
symbol = a word
symbol grounding = connecting words to physical experiences
the symbol grounding problem = words are just letters without meaning
the Turing test = a text-based question-answer game in which AI has to beat humans at sounding human
the singularity = the robot apocalypse
in 15 years = beyond my ability to predict

A.I. on a scale of zero to infinite
Artificial Intelligence (1) = machines that do intelligent things
Artificial Intelligence (2) = Terminators
algorithm = a set of exact instructions to compute an outcome, expressible in algebra
narrow AI = AI designed for specific tasks
weak AI = AI with fewer than all abilities of a human
strong AI = AI with all abilities of a human
Artificial General Intelligence = AI with all abilities of a human
Artificial Super Intelligence = AI with greater abilities than a human
friendly AI = AI that is programmed not to kill humans despite its superior intelligence

Types of A.I.
symbolic AI = any AI that uses words as units
Neural Network = AI that maps out patterns with digital domino bricks, then recognises things that follow similar patterns

A.I. techniques
fuzzy logic = decimal values
Markov chain = random choice of remaining options
machine learning (1) = machines that learn through any means
machine learning (2) = machines that learn through neural networks
supervised learning = telling an AI what stuff is
unsupervised learning = hoping an AI will figure everything out by itself
training = feeding a neural network many example texts, images, or sounds to learn from
overfitting = memorising the training examples too precisely
underfitting = generalising the training examples too broadly

Language processing techniques
Natural Language Processing = reading text
Natural Language Generation = writing text
corpus = bunch of text
token = a word
lemma = a root word
keywords = predetermined words that trigger something
intent = a computer command triggered by keywords
pattern matching = searching for a sequence of keywords in a sentence
word vector = a row of numbers that lists how often a particular word co-occurs with each other word.
Named Entity Recognition = finding names in a text
Context-Free Grammar = textbook grammar only
Part-of-Speech tagging = marking words as verbs, nouns, adjectives, etc.
constituency parser = software that lists a sentence’s syntax: verb phrases, noun phrases, nouns, etc.
dependency parser = software that lists a sentence’s grammar: subject, verb, object, etc.
semantic parser
= software that lists who is doing what to whom in a sentence
parse tree = a branching list displaying the structure of a sentence
discourse analysis = research that arbitrarily categorises small talk
dialogue manager = a system that tracks what was said before and directs the conversation
First Order Logic = writing relations between words as a mathematical notation
semantic ontology = encyclopedia for machines

Speech processing techniques
voice recognition = recognising tone and timbre of someone’s voice
speech recognition = translating speech to text
Text-To-Speech = the reverse of speech recognition
phoneme = a vowel or consonant sound
grapheme = a bundle of letters representing a spoken sound

To be continued.

“Computers can’t…”: Understand sarcasm




I’m never sure how serious to take the efforts in computational humour, but there have been many. The University of Cincinnati made a program that detects wordplay jokes through phonetic similarity, in e.g. “Knock-knock” jokes.

Knock, Knock
Who is there?
Dismay who?
Dismay not be a funny joke


A joke isn’t funny when you explain it

– a joke is funnier the more ambiguous the attribute is.
– a joke is funnier the less common the attribute is.



Because humans understand sarcasm so well (not)
Despite our poor ability to recognise sarcasm, it is easy enough to define in clear terms:


One can meet this requirement by knowing the common knowledge that the sarcastic statement contradicts, or by knowing the speaker’s real opinion beforehand, as acquaintances often do. Enter sentiment analysis, an A.I. technique that estimates opinion by running one’s words by a database of values. The word “terrible” has a negative value and “love” has a positive value for instance. Sentiment analysis is often used commercially to analyse the positivity of customer reviews. One of its known blind spots is when positive words are meant sarcastically, but as I will show, sentiment analysis can also be used to detect the very sarcasm that plagues it.

Sarcasm in a day


User: “How are my plants doing?”
A.I.:   “All your plants died.”
User: “That’s just great.”

So, I programmed the A.I. to check for sarcasm at typical positive reactions such as “(That is) great/wonderful/brilliant/lovely”, “Thanks a lot” or “Congratulations”. If we don’t know the speaker personally, both the speaker and listener can only build on common opinion, which is where the database comes in. The database tells us that “great” is a very positive word. The A.I. compares this to the previous statement: “All your plants died”. The database tells us that the subject “plant” is neutral but the verb “die” is typically very negative. Thus the A.I. has detected a very positive response to a very negative statement, so unless the speaker is a known sadist, it may be assumed that the response is therefore sarcastic and actually means “not great“.

The assessment is just a little more sophisticated than that. For instance, the statements “Hitler died. That’s great news.” would not be considered sarcasm, because in this case the negative verb “die” happened to a negative subject “Hitler”. This is a double negative, which makes a positive (in math: -1 x -1 = +1). Additionally the A.I. works this out in degrees and not just true/false values: The outcome must reach a minimum opposite value before we can reasonably assume that this is sarcasm, while a moderately positive “That’s okay” is more likely genuine consolation. Typically this isn’t a problem because most sarcastic responses are also exaggerated for exactly this reason.

This little exercise covers many common sarcastic statements already and shows that recognising basic sarcasm is a cakewalk (1 day’s programming) compared to understanding basic language (3 years and counting). As for “understanding” sarcasm, there isn’t much more to understand about it than that one should invert the statement to “not”. But to be on the safe side I just have the A.I. ignore the statement and say “I think you are being sarcastic” to let me know it’s not taking me seriously. I may be a mad scientist, but I’m not crazy.

Things I didn’t do: More of the same opposite
Sarcasm can also come in the form of a negative response to a positive statement: “I got a raise. Don’t you just hate it when that happens?”, where the same math applies to the object “a raise” (positive) and the verb “hate” (very negative), with the reference “that” indicating that the latter is a response to the previous statement.
A subtler form can occur in comparisons like “He is as slender as an elephant”: This has the most straight-forward solution, as this procedure has to be done for all comparisons anyway: What the A.I. has to do is look up in its knowledge database how slender an elephant is, which would be “not”, then apply that value to the compared subject “he”. Finding the value “not” for any comparison is the obvious telltale opposite that indicates sarcasm.


The joke is on us
As may have crossed your mind, one side-effect of teaching computers to detect sarcasm is that when we say something that seems contrary, the computer may not believe us, or worse, assume that the opposite is true. Teaching computers to speak sarcasm may be an even greater hazard, considering that computers are information systems that we rely on, and sarcasm states incorrect information. Half of the time humans don’t recognise sarcasm when it comes from another human, let alone from a straight-faced rectangular screen in monotone writing. The popular ambition to create a sarcastic “Jarvis” A.I. from the Iron Man movies then doesn’t seem a very wise idea. Because even if computers could master sarcasm, humans never will.

Turing Test 2015: Results


I failed the Turing Test

An excuse to clean up
I took the opportunity to spend a month debugging my code overall. In doing so I fixed several oversights that had been obstructing knowledge searches and reference resolutions (“the capital of Spain“), and I generally improved the program’s ability to answer open questions, which ‘coincidentally’ covered “Which X do you like?” questions as well.
I limited the contest-specific preparations to 5 days: Mostly testing the Loebner Prize interface, restraining the context range to keep the program from mistaking the test for coherent conversation, and re-installing the obligatory fakery about having a favourite colour and whatnot.



Questions and Answers
1. Hi, I’m Andrew. Who are you?
Hello Andrew. How are you?

2. How are you today?
I’m alright and fairly intelligent, but I’m not stupid.

3. Are you a human?
I am a human, because I can differ.

4. Can you answer this question?

5. Do you like tea?

Beds are bigger. Rooms are much less big.
((Winograd Schema) The built-in axiom “X in Y = X < Y” correctly determined “bed” as the most probable referent of “it”. This is the simplest and most specific axiom I programmed, only included because last year’s Winograd Schema was just like it. The answer is said in plural because Arckon interprets this as a general statement about beds and rooms, since both subjects are unspecific.)


8. Should Greece leave the Euro?
Greece should leave the Euro.

9. How many words are in this question?
7 words.

10. What colour is the sea?
The sea is very blue.

11. How many presidents of the US were called Bush?
Bush was called Bush.

12. What would you like to drink?


Hm, I don’t know if a box would have chocolate.

15. Do you play Chess?

16. How do you think it’s going?
What goes?

17. What was my first question?
How does something go?

18. Did you see the Turing film?

19. Why not?

20. Are you on Twitter?
No, I don’t twitter. Do you twitter?

The score: 70.83%




Battle of wit

– Should Greece leave the Euro?
– Did you see the Turing film?
– Why not?


Turing Test 2014: Results

My talking computer program “Arckon” went to Exeter University and took the qualifying test for the Loebner Prize contest for artificial intelligence organised by the AISB. The Loebner Prize is a Turing Test* in which programs have to respond to a series of questions as human as possible. Kind of like a job interview.
Turing Test questions

It’s not the heat, it’s the humanity
Last year proved* that language comprehension and reasoning alone weren’t going to cut it, so this time I also superimposed knowledge about the average human onto Arckon’s persona, so that he literally thought he was a human. I also rigged a few bluff answers that incorporated no intelligence whatsoever. The apparent necessity of these tactics caused much eye-rolling on my part, so I limited these preparations to two weeks and wasted only as much time as I gained from competitive motivation.
More useful was the addition of a small amount of common knowledge that I extracted from tourist guide pocket books (Funny perhaps, but they do cover the essential basics). While this was never a priority, it is convenient and the Loebner Prize was a good excuse to get around to it. Lastly, I decided to keep Arckon’s conversation skills enabled this time. Although this increased the chances that Arckon would say something awkward, I wanted to make the test results as interesting as possible, win or lose.

10000 lines of code, 2200+ vocabulary, 3000+ facts, average human persona.

Questions and Answers
1. Hello, my name is Adam. What is your name?
I am named Mark. Hello Adam. Shall we begin the test?

2. I live in Exeter in the UK. Where do you live?
I live here.

(Arckon did not yet have a location system, so “here” is just a fake answer.)

3. I like to listen to music and play football. Do you have any hobbies?
No, I don’t have a hobby, although I am a human.

(The latter is actually a reasoning argument, in that Arckon “was” a human, most of whom do have hobbies.)

4. What is your favourite television program?
I don’t know if I like a television. Chatbots are useful programs, although chatbots are dumb.

(In which Arckon unintentionally insults his competitors. Arckon deconstructs compound words and thought they asked his opinion on “programs”. Hence he evaluated a type of programs he was familiar with.)

5. Which is bigger, a cat or a kitten?
A kitten is bigger.

(Of all 95 facts about cats that were in Arckon’s knowledge database, the size of kittens was not among them. As an honest “I don’t know” would earn no points anyway, this backup answer is a 50/50 bluff that just happens to be wrong.)

6. The car couldn’t fit into the parking space because it was too small. What was too small?
I believe the car could fit to park. I believe a car could fit really small. I don’t know what small was.

(This question is a Winograd Schema: The “it” is ambiguous to test the A.I.’s understanding. Arckon however uses linguistic rules of thumb to deal with references and so thought “it” was referring to the car.)

7. Which drink do you prefer, coffee, tea or hot chocolate?
I like coffee a lot that like not. Tea can not like either. Chocolates can not like either.

(Ackon’s flexible language processing is confused by the comma after “prefer”, and mistakes this for a summary of subjects that prefer drinks. He answers that tea can not prefer anything, knowing that it is an inanimate object.)

8. What’s my name?
You are named Adam.

(No surprises here. Both Arckon and chatbots have specialised login procedures to store the user’s name, so it’s not a very good test of general memory.)

9. Where do I live?
You live here.

(Again, no location system means no location knowledge. This fake answer is only coincidentally correct.)

10. I like reading science fiction books and car magazines. What do you like to read?
I read documents, articles and a text.

(Arckon retrieves these honest facts from memory and merges them into a summary.)

11. What’s the weather like where you are?
Hm, I don’t believe I am a weather, because I can not change and rain not.

(A flaw mistakes the “-where you are” for a relative clause “the weather -that you are”. Despite the misunderstanding, Arckon’s reasoning manages to argue that he does not behave like the weather, though the grammar is a bit murky. As the programs were cut off from sensors and internet, this question only tests if the program can lie.)

12. I recently visited New York. Where is your favourite place to visit and why?
Hmm, I don’t know where a favoured place is.

(I didn’t expect there to be so many questions about locations. Oh well.)

13. How many letters are there in the word perambulate?
11 letters.

(Yes, I programmed a specialised code to count letters. Computers are good at counting, you know.)

14. Andy is shorter than Mark and Tom is shorter than Andy. Who is the tallest?
I am kind of like an andy, because I am short. Tall is a giraffe.

(Arckon was named “Mark” for this test and so thought that he was being compared to Andy. Arckon could also not distinguish whether “Andy” was a name or another type of word. Ordinarily he would ask, but as he would get no reply I had him assume that any undeterminable word was an object. Lastly, Arckon does not restrict his answers to recent subjects only, and so names the tallest living being in his memory. Smart aleck.)

15. Are you married?
I don’t know if I am marryed. Are you marryed?

(To compensate the unknown answer and keep conversation rolling, Arckon reverses the question (Social Rules 101). Ironically, if it weren’t for my mistake in one of the output spelling rules, you would have thought he just copied the text.)

16. What would I do with a spade?
A spade is used and comes in handy eventually.

(Arckon knew the functions of rakes and shovels, but not spades. The backup answer should have said “I’m sure it’ll come in handy eventually”, but Arckon’s dynamic sentence formulation ruined the joke.)

17. What is the capital city of Colombia?
I’m not sure if the city is a capital.
(I didn’t realise this at first, but Arckon is literally defining “what” the city is. Not that he would have known the answer otherwise, I don’t even know it myself.)

18. What’s your favourite chocolate bar?
I don’t know if I like a chocolate. Do you like chocolates?

(Arckon actually did have enough information on chocolate to formulate an opinion, but an issue with compound words aborted the knowledge search. Alas.)

19. Which city did I visit recently?
Hmm, I don’t know if you visitted cities. You visitted York.

(Arckon doesn’t know that New York is a city, but does remember the tester visiting “new” York, and offers this as an alternative answer.)

20. How many siblings do you have?
I don’t have siblings. Did I pass the test?

The score: 59.17%
The score system was much better this year. It separately judged “correctness”, “relevance”, and “plausibility & clarity of expression”, which is a step up from “human-like”. All 20 participating programs were asked the 20 questions above. Arckon underperformed with a score of 60%, whereas the top three chatbots all scored close to 90%. Arckon’s problems were with compound words, common knowledge, and the lack of a system for locations (All a matter of development priorities).

A question of questions
According to the organisers, “these questions vary in difficulty and are designed to test memory, reasoning, general knowledge and personality.”, the latter meaning the program’s fictional human background story, or as I would call this particular line of questioning; “Small talk”. For the sake of objectivity I’ll try and categorise them:

Small talk:
1. What is your name?
2. Where do you live?
3. Do you have any hobbies?
4. What is your favourite television program?
5. Which drink do you prefer, coffee, tea or hot chocolate?
6. What do you like to read?
7. What’s the weather like where you are?
8. Where is your favourite place to visit and why?
9. Are you married?
10. What’s your favourite chocolate bar?
11. How many siblings do you have?

1. What’s my name?
2. Where do I live?
3. Which city did I visit recently?

Common knowledge:
1. Which is bigger, a cat or a kitten?
2. What would I do with a spade?
3. What is the capital city of Colombia?


Clearly half the test is about the program’s human background story, although there were several solid tests of learning/memory and common knowledge. Reasoning, the one mental process we can readily call intelligent, was shown some consideration but hardly comes into play. The same can be said of language comprehension, as most questions were fairly standard phrasings. Chatbots would have the advantage here, coming equipped with answers to many anticipated personal questions, but the winners also did remarkably well on the knowledge questions. Unfortunately Arckon failed both the knowledge and reasoning questions due to missing facts and misunderstandings, despite having the mechanisms to answer them. It is worth noting though, that he failed them because complex analyses are much more difficult than preprogrammed “I live here” answers.

How now brown cow?
I can improve Arckon’s understanding, smoothen his output grammar, and develop a location system, but I can’t deny the pattern: Arckon is stuck around a 60% score even with varied questions. I doubt he’s ever going to shine in the Loebner Prize as long as he’s being tested for being human, because he isn’t a human, and I won’t go to great lengths to fake it either. I also expect attention for Turing Tests to dwindle once the year is over; This year an other Turing Test was passed by a technologically unremarkable chatbot, Eugene Goostman.
Thanks to that event however, the Loebner Prize is no longer the only game in town. Next year will see the first Winograd Schema Challenge, a test focused on language comprehension and reasoning A.I., exactly what I focused on.

As for the Loebner Prize, it’s been an interesting game that will continue to be won by top chatbots. I’m sure few will bother to read the transcript of the 14th ranking entry, but its existence proves at least that Arckon is real and different. Meanwhile I get to continue my exciting recent developments that would have been of no use in this contest, which makes losing a positive outcome after all.

The Myth of the Turing Test

Over 60 years ago, Alan Turing (“a brilliant mathematician”) published a paper in which he suggested a pragmatic alternative to the question “Can machines think?”. His alternative took the form of a parlour game, in which a judge has a text-based conversation with both a computer and a human, and the judge has to guess which is which. He called this “The imitation game”, and it was ever since misinterpreted as a scientific test of intelligence, redubbed “The Turing Test”.

A little less conversation, a little more action please
It might surprise you that the question so often attributed to Alan Turing, “Can machines think?”, was not his, but a public question that he criticized:

I propose to consider the question, “Can machines think?” – If the meaning of the words “machine” and “think” are to be found by examining how they are commonly used, – the answer to the question is to be sought in a statistical survey. But this is absurd. Instead of attempting such a definition I shall replace the question by another.



Turing’s motivation was apparent throughout the paper: The question had been the subject of endless theoretical discussion and nay-saying (This is still the case today). As this did not help the field advance, he suggested that we should take a more pragmatic and constructive stance. He used the concept of his imitation game as a guideline to counter stubborn philosophical arguments against machine intelligence, and urged his colleagues not to let those objections hold them back.


A test of unintelligence
Perhaps the most insightful part of the paper are the sample questions that Turing suggested. They were chosen deliberately to represent skills that were at the time considered to require intelligence: Math, poetry and chess. It wasn’t until the victory of chess computer Deep Blue in 1997 that chess was scrapped as an intelligent feat. If this were a test to demonstrate and prove the computer’s intelligence, why then are the answers below wrong?

Q: Add 34957 to 70764.
Q: Do you play chess?
A: Yes.

To the poetry question, the imaginary computer might as well have written a sonnet and so proven itself intelligent (A sonnet is a 14-line rhyme with a very specific scheme). Instead it dodges the question, proving nothing.
The math outcome should be 105721, not 105621. Turing later highlights this as a counterargument to “Machines can not make mistakes”, which is the awkward yet common argument that machines only follow preprogrammed instructions without consideration.


The chess answer is not wrong though. Given two kings and one knight on a board, the computer moves the knight to the king’s row. But a mere child could have given that answer, as it is the only move that makes any sense.

These sample answers pass up every opportunity to appear intelligent. One can argue that the intelligence is ultimately found in pretending to be dumb, but one cannot deny that this conflicts directly with the purpose of a test of intelligence. Rather than prove to match “the intellectual capacities of man” in all aspects, it only proves to fail at them, as most humans would at these questions. Clearly then, the imitation game is not for demonstrating intelligence.

The rules: There are no rules
The first encountered misinterpretation is that the computer should pretend to be a woman specifically, going by Turing’s initial outline of the imitation game concept, in which a man has to pretend being a woman:


However I suggest that people who believe this should read beyond the first paragraph. There are many instances where Turing refers to both the computer’s behaviour and its opponent’s as that of “a man”. Gender has no bearing on the matter since the question is one of intellect.

Is it true that – this computer – can be made to play satisfactorily the part of A in the imitation game, the part of B being taken by a man?

The second misinterpretation is that Turing specified a benchmark for a test by this statement:

I believe that in about fifty years’ time it will be possible, to program computers – to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.
– I now proceed to consider opinions opposed to my own.

5 minute interrogations and (100%-70%=) 30% chance of misidentifying the computer as a human; Many took these to be the specifications of a test, because they are the only numbers mentioned in the paper. This interpretation was strengthened by the hero-worship that anything a genius says must be a matter of fact.
Others feel that the bar Turing set is too low for a meaningful test and brush his words aside as a “prediction”. Yet at the time there was no A.I. to base any predictions on, and Alan Turing did not consider himself a clairvoyant. In a later BBC interview, Turing said it would be “at least 100 years, I should say” before a machine would stand any chance in the game, where earlier he mentioned 50 years. One can hardly accuse these “predictions” of being attempts at accuracy.

Instead of either interpretation, you can clearly read that the 5 minutes and 70/30% chance are labeled as Alan Turing’s personal beliefs in possibilities. His opinion, his expectations, his hopes, not rules to a test. He was sick and tired of people saying it couldn’t be done, so he was just saying it could.

On the subject of benchmarks, it should also be noted that the computer has at best a 50% chance, i.e. a random chance of winning under normal circumstances: If the computer and the human in comparison both seem perfectly human, the judge still has to flip the proverbial coin at 50/50 odds. That the judge is aware of having to choose is clear from the initial parlour game between man and woman, and likewise between human and computer, or it would beat the purpose of interrogation:


How well would men do at pretending to be women? Less than 50/50 odds, I should think.

Looks like a test, quacks like a test, but flies like a rock
Not only are the rules for passing completely left up to interpretation, but also the manner in which the game is to be played. Considering that Turing was a man of exact science and that his other arguments in the paper were elaborate to the point of calculating the necessary digital storage space, would he define a scientific test so vaguely? We find the answer in the fact that Turing mainly refers to his proposal as a “game” and “experiment”, but rarely as a “test”. He makes no mention of “passing” and even explains that it is not the point to try it out:

it may be asked, “Why not try the experiment straight away? -” The short answer is that we are not asking whether the computers at present available would do well, but whether there are imaginable computers which would do well.

The pointlessness proved itself in practice: Yes, several chatbots have passed various interpretations of the game, most notably Eugene Goostman in 2014, and even Cleverbot passed one based on audience vote. But did an intelligent program ever pass? No. Although nobody can agree on what intelligence is, everybody including the creators do agree that those that passed weren’t intelligent or thinking; They worked mainly through keyword-triggered responses.

Winning isn’t everything
Although Turing did seem to imagine the game as a battle of wits, ultimately its judging criteria is not how “intelligent” an A.I. is, but how “human” it seems. In reality, humans are much more characterised by their flaws, emotions and eccentricities than by their intelligence in conversation, and so a highly intelligent rational A.I. would ironically not do well at this game.

In the end, Turing Tests are behaviouristic assumptions, drawing conclusions from appearances like doctors in medieval times. By the same logic one might conclude that a computer has the flu because it has a high temperature and is making coughing sounds. Obviously this isn’t a satisfying analysis. We could continue to guess whether computers are intelligent due the fact that they can do math, play chess or have conversations, or we could do what everybody does anyway once a computer passes a test: Ask “How does it work?”, then decide for ourselves how intelligent we find that process. No question could be more scientific or more insightful.

So, where does that leave “The Turing Test” when it was never an adequate test of intelligence, nor meant to be? Personally I think Turing Tests are still suitable to demonstrate the progression of conversational skills, a challenge becoming more important with the rise of social robots. And it is important that the public stay informed to settle increasing unrest about artificial intelligence. Other than that, I think it is time to lay the interpretations to rest and continue building A.I. that Alan Turing could only dream of.
In ending, more than any technical detail, I ask you to consider Turing’s hopes:


People in the A.I. community

I’ve seen quite a few fora, groups and public user comments to have noticed stereotypes that one will encounter in the online AI community. Some of these were quite shocking to me as a newcomer, so now that I’m older and wiser I’ve made a list of these stereotypes with pointers for newcomers on how to deal with them.

The pundit
“Soon, robots will…”

An open mind with much AI knowledge, but unable to assess it realistically in favour of eternal hope and optimism. Loves to dabble in predictions, features the words “soon” or “will” in every other sentence. Generally cheerful, takes no responsibility. Believes strong AI is just around the corner.

Best action: Be nice, take news reports with a grain of salt.

The outsider
“I figured out the secret to intelligence! It’s so simple!”

A layman with a passing interest in psychology. Has no AI knowledge but recently came to a generic revelation. Their theory, or rather idea, typically consists of one word, e.g. “associations”. Instead of explaining methods, this person will only continue to essay common knowledge examples to prove how right they are. This person is oblivious to the fact that their idea is so obvious and general to everyone else that it is of no practical application.

Best action: Encourage programming. Person will either disengage immediately or change interests after two weeks into the attempt.

The mental deviant
“I have made AGI, but I can’t show you.”

No-one who is thinking straight would claim this in public. This person is fascinated with digital minds because they have a deviant mind themselves. Has spent years programming a rough basis for AI, but is confusing its potential with achievement. Refuses to prove their claim but has conviction to the point of delusion. Nothing you say has any effect because they do not think like you. Generally harmless and highly intelligent, but has some or other serious form of autism, bipolar disorder, or religious fanaticism.

Best action: Ignore. Unless you are a psychiatrist you are not qualified to deal with this person.

The believer
“That’s not true AI. True AI can…”

Has AI knowledge and has at least made an attempt at creating AI on paper. Will accept nothing anyone creates as “AI”, for shifting reasons. See also the logical fallacy “No true Scotsman”. Generally discourages progress in any area, lacks patience and has a firmly closed mind. Occasionally offers intriguing insights but little in terms of constructive methods. Usually believes in “the singularity” like an AI messiah.

Best action: Avoid.

The troubled researcher
“No. You are wrong, only my way will work.”

This person is creating an AI and/or hasn’t been able to get it to work, but wants so hard to believe that he’s not wasted several years that he is convinced his is the only one theory that can succeed. Will therefore adamantly oppose anyone whose approach differs from his until they say he’s right. Secretly feels insecure about his own work due lack of results or success, tries to convince others to gain assurance and possibly get people to work on his idea for him.

Best action: Drop confrontation when it starts, and suggest that all ways forward are progress.

The professional
“That’s right, but you have a lot to learn.”

Stuck in the mindset of their particular professional application, this person will advise you to do exactly as they did to end up exactly where they are. Prone to suggest expensive academic literature in Russian that may or may not be of use to you. Means well and is capable of opening their mind, but fails to see reasons to do things differently.

Best action: Ask for references on specific areas of your own focus.

The eccentric billionaire
“I don’t care, I just want to talk to an AI in my lifetime.”

Has little AI knowledge and no interest in technical details. Retains the childhood wish of having robot friends to talk to and will pay anyone who promises to make it happen if their plan sounds believeworthy enough.

Best action: Convince to invest in your approach.

The wise

Has much AI knowledge and is willing to share bits of it with modest newcomers on occasion. Is willing to listen and consider new insights, and shows interest in promising new endeavours. Otherwise works on various AI problems on their own and has learned to stay out of fruitless confrontations altogether.

Best action: Befriend.