Discussion of the 1950 paper by Alan Turing that proposed what is now called the Turing Test. This is one of the most impactful papers in the history of AI and the first paper in the AI paper club on our Discord. Join here: https://discord.gg/8RwBPRs

Slides for this video: https://bit.ly/2VIAp2R
References sheet: https://bit.ly/turing-test-paper
Lex + AI Podcast Discord: https://discord.gg/8RwBPRs

OUTLINE:
0:00 - Introduction
1:02 - Paper opening lines
3:11 - Paper overview
7:39 - Loebner Prize
11:36 - Eugene Goostman
13:43 - Google's Meena
17:17 - Objections to the Turing Test
17:29 - Objection 1: Religious
18:07 - Objection 2: "Heads in the Sand"
19:18 - Objection 3: Godel Incompleteness Theorem
19:51 - Objection 4: Consciousness
20:54 - Objection 5: Machines will never do X
21:47 - Objection 6: Ada Lovelace
23:22 - Objection 7: Brain in analog
23:49 - Objection 8: Determinism
24:55 - Objection 9: Mind-reading
26:34 - Chinese Room thought experiment
27:21 - Coffee break
31:42 - Turing Test extensions and alternatives 
36:54 - Winograd Schema Challenge
38:55 - Alexa Prize
41:17 - Hutter Prize
43:18 - Francois Chollet's Abstraction and Reasoning Challenge (ARC)
49:32 - Takeaways
56:51 - Discord community
57:56 - AI Paper Reading Club

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman

In this video, I propose to ask the question that was asked by Alan Turing almost 70 years ago in his paper, Computing Machinery and Intelligence.

This is the first paper in a paper reading club that we started focused on artificial intelligence, but also including mathematics, physics, computer science, neuroscience, all of the scientific and engineering disciplines.

On the surface, this is a philosophical paper, but really, it's 1 of the most impactful, important first steps towards actually engineering intelligent systems by providing a test, a benchmark that we call today the Turing test of how we can actually know quantifiably that a system has become intelligent.

So I'd like to talk about an overview of ideas in the paper, provide some of the objections inside the paper and external to the paper, consider some alternatives to the test proposed within the paper, and then finished with some takeaways.

Like I said, the title of the paper was Computing Machinery and Intelligence, published almost 70 years ago in 1950, author Alan Turing.

And to me, now we can argue about this, On the slide I say it's 1 of the most impactful papers.

To me, it probably is the most impactful paper in the history of artificial intelligence, while only being a philosophy paper.

I think the number of researchers from inside computer science and from outside that has inspired, it has made dream at a collective intelligence level of our species, inspired that this is possible, I think is immeasurable.

For all the major engineering breakthroughs and computer science breakthroughs and papers, stretching all the way back to the 30s and 40s with even the work by Alan Turing with the Turing machine, some of the mathematical foundations of computer science to today with deep learning, a sequence of papers from the very practical Alex Ned paper to the back propagation paper.

So all of these papers that underlie the actual successes of the field, I think the seed was planted.

And it happens to have some of my favorite opening lines of any paper I've ever read.

It goes, I propose to consider the question, can machines think?

This should begin with the definitions of the meaning of the terms machine and think.

The definition might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous.

If the meaning of the words machine and think are to be found in examining how they're commonly used, it is difficult to escape the conclusion that the meaning and the answer to the question, can machines think, is to be sought in a statistical survey such as a Gallup poll.

Instead of attempting such a definition, I shall replace the question by another, which is closely related to it, and is expressed in relatively unambiguous terms." And he goes on to define the imitation game, the construction that we today call the Turing test, which goes like this.

There's a human interrogator on 1 side of the wall, and there's 2 entities, 1 a machine, 1 a human on the other side.

And The human interrogator communicates with the 2 entities on the other side of the wall by written word, by passing notes back and forth.

And after some time of this conversation, the human interrogator is tasked with making a decision, which of the other 2 entities is a human and which is a machine?

I think this is a powerful leap of engineering, which is take an ambiguous but a profound question like can machines think and convert it into a concrete test that can serve as a benchmark of intelligence.

But there's echoes in this question to some of the other profound questions that we often ask.

So not only can machines think, but can machines be conscious?

Can machines enjoy a delicious meal, a piece of chocolate cake?

I think these are really, really important questions, but very difficult to ask when we're trying to create a non-human system that tries to achieve human level capabilities.

So that's where Turing formulates this imitation game.

And his prediction was that by the year 2000, or in 50 years since the paper, that a machine with 100 megabytes of storage will fool 30% of humans in a 5 minute test of conversation.

Another broader societal prediction he made, which I think is also interesting, is that people will no longer consider a phrase like thinking machine contradictory.

So basically artificial intelligence at a human level becomes so commonplace that we would just take it for granted.

And the other part that he goes at length towards the end of the paper to describe, which he believes that learning machines or machine learning will be a critical component of this success.

I think it's also useful to break apart 2 implied claims within the paper, open claims, open questions.

1 is that the imitation game as Turing proposes is a good test of intelligence.

And the second is that machines can actually pass this test.

So when you say can machines think, you're both proposing an engineering benchmark for the word think and raising the questions, can machines pass this benchmark?

1 of the perhaps tragic but also exciting aspects of this whole area of work is that we still have a lot of work to do.

So throughout this presentation, I will not only describe some of the ideas in the paper and outside of it in the years since, but also some of the open questions that remain, both at the philosophical, the psychological, and the technical levels.

So here the open question stands, is it even possible to create a test of intelligence for artificial systems that will be convincing to us?

A corollary of that question is, looking at the prediction that Turing made, that people will no longer find the phrase thinking machines contradictory, why do we still find that phrase contradictory?

Why do we still think that computers are not at all intelligent?

For many people, the game of chess was seen as the highest level of intelligence in these early days.

In fact, we assign a lot of intelligence to Garry Kasparov for being 1 of the greatest, if not the greatest chess players of all time as a human.

Why do we not assign at least an inkling of that to IBM Dblue when it beat Garry Kasparov?

Now, of course, you might start saying that it's a brute force algorithm, or in the case of AlphaGo and AlphaZero, you know how the learning mechanisms behind those algorithms work when they mastered the game of Go and the game of chess.

And we'll get to some of those objections, but there's something deeply psychological within those objections that almost fear an artificial intelligence that passes the test.

So the Turing test is very interesting as a thought experiment, as a philosophical construct, but it's also interesting as a real engineering test.

And 1 of the implementations of it has been called the Lobner Prize, which has been running since 1991 to today.

And the awards behind it, the award structure, is $25,000 for a system that using text alone passes the test and $100,000 that uses other modalities like visual and auditory input.

The rules of the competition have changed through the years but they're currently are as follows.

It's a 25 minute conversation, and in order to win, to pass the test, you have to fool 50% of the judges with which the system communicates.

Mitsuko and Rose from Steve Warswick and Bruce Wilcox have been dominating the past 10 years, winning all but 1 of the years.

Some details, Mitsuko and Rose are both mostly scripted rule-based chatbots, so they're not end-to-end learning systems.

I believe there is a little bit of machine learning, but as I understand, at the core, they're mostly scripted.

And on the next slide, we'll go through a transcript of a conversation with Matsuko to give you a sense of what those conversations are like.

But I'd like to mention sort of a tragic note really that as far as I understand, the prize is no longer funded, but money aside, it shows that there's not a broad interest in this construction, both inside and outside the research community.

It's really surprising to me that Google DeepMind, Facebook AI, major groups, major individual researchers in the field of AI have not taken on this challenge.

I hope to actually be a small part of changing that a little bit, because I think there is a lot of value in considering the Turing test.

And I'll return to that point later on in this presentation.

So you can pause and maybe read parts of this transcript.

I'll highlight on the right, at the top of the right column.

The human says you're a robot Mitsuko answers.

Human many people live lives akin to robots Mitsuko how many more than a million?

Human yes Mitsuko that seems like rather a lot Human.

Do you think Southampton will beat Swansea?

Yes, it is a city in the north of England.

So you see in this transcript, it's just some little gap of inhumanity that becomes apparent.

There's a lingering and repeatability of points.

There's certain gaps in understanding, inability to follow tangents, all those kinds of things.

But it's still not clear to me as an open question how to make explicit where exactly the point of the failure of the test is.

I believe that hasn't actually been really researched that well in these constructions.

As opposed to decision making at the very end of a conversation, is this human or not, rather marking parts of a conversation as more or less human, like suspicious parts that make you wonder, this is not human.

I think that'll be really interesting to see if it's possible to make explicit what aspects of the conversation are the failure points.

1 of the times that the claim that the Turing test was passed, I think most famously was in 2014 at an exhibition event that marked the 60th anniversary of Turing's death, Eugene Guzman fooled 33% of the event judges.

And the method he used was to portray a 13-year-old Ukrainian boy that had a bunch of different personality quirks and obviously the language barrier, and had some humor and a constant sort of drive towards misdirecting the conversation back to the places where it was comfortable doing.

So there's some criticism that you can make of this event due to some sort of smoke and mirrors, kind of the PR and marketing side of things that I think is always there with these kind of exhibition events.

But setting that aside, I think the interesting lessons here is that the parameters, the rules of the actual engineering of the Turing test can determine whether it contains sort of the spirit of the Turing test, which is the test that captures the ability of an agent to have a deep, meaningful conversation.

So in this case, you can argue that a few tricks were used to circumvent the need to have a deep, meaningful conversation.

And 30% of judges were fooled without rigorous, thorough, transparent, open domain testing.

On the left is a transcript with Scott Aronson, the famed computer scientist, the quantum computing researcher.

Talked to him on the podcast, brilliant guy.

He posted some of the conversation that he had with Eugene, he was 1 of the judges, on his blog that I think is really interesting.

So it shows that the judge, the interrogator, when they're an expert, they can drive, they can truly put the bot to the test.

As Scott did, he really didn't allow the kind of misdirection that Eugene nonstop tried to do.

So As I mentioned, despite the waning, I guess, popularity of the Lobner Prize and the Turing Test idea in general, Google has published a paper and proposed a system called MINA.

That's a chatbot, that's an end-to-end deep learning system.

The representational goal in the 2.6 billion parameters is to capture the conversational context well, to be able to generate the text that fits the conversational context well.

Now, 1 interesting aspect of this, besides being a serious attempt at creating a learning-based system for open domain conversational agents is that a new metric is proposed and it's a two-part metric of sensibleness and specificity.

Now, sensibleness is that a bot's responses have to make sense in context.

Just give you a sense for humans who have 97% sensibleness.

So ability to match what we're saying to the context.

Now, the reason you need another side of that metric is because you can be sensible, you can fit the context by being boring, by being generic, by making statements like, I don't know, or that's a good point.

So these generic statements that fit a lot of different kinds of context.

So the other side of the metric is specificity.

Basically, the goal being there is don't be boring.

It's to say something very specific to this context.

So not only does it match the context, but it captures something very unique to this particular set of lines of conversation that form the context.

I think it's fair to say that the beauty of the music, the humor, the wit of conversation comes from that ability to play with the specifics, the specificity metric.

Humans achieve 86% sensibleness and specificity.

Mina achieves 79% compared to Mitsuko, who achieves 56%.

I wanna be very careful here because there is also, not to throw shade, but it's closed source currently, And there's a little bit of a feeling of a PR marketing situation here.

Naturally, perhaps, the paper is made in such a way, the methodology and the results are made in such a way that benefit the way the learning framework was constructed.

Now that's, I don't want to over-criticize that because I think there's still a lot of interesting ideas in this paper, but in terms of looking at the actual percentages of 86% human performance and 79% MENA performance, I think we're quite away from being able to make conclusive statements about a system achieving human level conversational capabilities.

So those plots should be taken with a grain of salt, but the actual content of the ideas, I think is really interesting.

I think quite obviously the future, long-term, but hopefully short-term, is in learning end-to-end learning-based approaches to open domain conversation.

So just like Turing described, funny enough, 70 years ago in his paper, that machine learning will be essential to success.

It's a lot less interesting and revolutionary to think so today, but I believe that machine learning will also need to be a very central part of achieving human-level conversational capabilities.

9 of them are highlighted by Turing himself in his paper.

Here I provide some informal, highly informal summaries.

The first objection is religious, which connects thinking to quote unquote the soul.

And God presumably is the giver of the soul to humans.

Now Turing's response to that is God is all powerful.

There is no reason why he can't assign souls to anything biological or artificial.

So it doesn't seem that Whatever mechanism by which the soul arrives in the human cannot also be repeated for artificial creatures.

The second objection is the quote-unquote head in the sand.

It's a bit of a ridiculous 1, but I think it's an important 1 because it keeps coming up often, even in today's context, highlighted by folks like Elon Musk, Stuart Russell, and so on.

The head in the sand objection is that AGI is scary, so human level and superhuman level intelligence is kind of scary.

It seems like the world would be totally transformed if we have something like that.

Then it could be transformed in a highly negative way.

Because it's kind of seems far away, So it probably won't happen, so let's just not think about it.

That's kind of the objection of the Turing test.

It's so far away, it's not worthwhile to even think about a test for this intelligence or what human level intelligence means or what superhuman level intelligence means.

The response quite naturally is that it doesn't matter how you feel about something and whether it's going to happen or not.

So we kind of have to set our feelings aside and not allow fear or emotion to model our thinking or detract us from thinking about it at all.

The third objection is from Gato's incompleteness theorem, saying there's limits to computation.

This is the Roger Penrose line of thinking that basically if a machine is a computation system, there is limits to its capabilities in that it can never be a perfectly rational system.

Turing's response to this is that humans are not rational either, they're flawed.

Nowhere does it say that intelligence equals infallibility.

In fact, it could probably be argued that fallibility is at the core of intelligence.

The fourth objection is that consciousness may be required for intelligence.

Turing's response to this is to separate whether something is conscious and whether something appears to be conscious.

So the focus of the Turing test is how something appears.

And so in some sense, humans, to us, as far as we know, only appear to be conscious.

We can't prove that they're actually conscious, humans outside of ourselves.

And so, since humans only appear to be conscious, there's no reason to think that machines can't also appear to be conscious.

And that's at the core of the Turing test.

So, the Turing test kind of skirts around the question whether something is or isn't intelligence, whether is or isn't conscious, the fundamental question is, does it appear to be intelligent, does it appear to be conscious?

So he actually doesn't respond to the idea that consciousness is or isn't required for intelligence.

He just says that if it is, there's no reason why you can't fake it.

And that will be sufficient to achieve the display of intelligence.

The fifth objection is the negative Nancy objection of machines will never be able to do X, whatever X is.

You can make it love, joke, humor, understand or generate humor, eat, enjoy food, create art, music, poetry, and so on.

So there's a lot of things we can put in that X that machines can never do.

And basically highlighting our human intuition about the limitations of machines.

Just like with the second objection, naturally the response here is that the objection that machines will never do X doesn't have any actual reasoning behind it.

It is just a vapid opinion based on the world today, refusing to believe that the world of tomorrow will be different.

The sixth objection, probably the most important, 1 of the most interesting, comes by way of Ada Lovelace, Lady Lovelace, the mother of computer science, with the basic idea that machines can only do what we program them to do.

Now this is an objection that appears in many forms throughout before Turing and after Turing.

And I think it's a really important objection to think about.

So in this particular case, I think Turing's response is quite shallow, but it is nevertheless pretty interesting, and we'll talk about it again later on.

His response is, well, if machines can only do what we program them to do, we can rephrase that statement as saying, machines can't surprise us.

And when you rephrase it that way, it becomes clearer that machines actually surprise us all the time.

A system that is sufficiently complex will no longer be 1 of which we have a solid intuition of how it behaves, even if we built all the individual pieces of code.

For those of you who have programmed things, so I've written a lot of programs, in the initial design stage you have an intuition about how it should behave, there's a design, there's a plan, you know what the individual functions do, but as the piece of code grows, your ability to intuit exactly the mapping from input to output fades with the size of the code base, even if you understand everything about the code and even if you set logical and syntactic bugs aside.

The seventh objection looks to the brain and looks to the continuous analog nature of that particular neural network system.

So Turing's response to that is, sure, the brain might be analog, and then digital computers are discrete, but if you have a big enough digital computer, it can sufficiently approximate the analog system, meaning to a sufficient degree that it would appear intelligent.

The eighth objection is the free will objection, right?

Is that when you have deterministic rules, laws, algorithms, they're going to result in predictable behavior.

And this kind of exactly deterministic, predictable behavior doesn't quite feel like the mind that we know as humans is possessing.

This kind of feeling that underlies what's required for intelligence for a mind, I think is behind the Chinese room thought experiment that we'll talk about next.

So Turing's response here is that humans very well could be a complex collection of rules.

Just because we don't understand or don't even have the tools to explore the kind of rules that underlie our brain doesn't mean it's not just a collection of deterministic, perfectly predictable sets of rules.

Objection number 9 is kind of fun, quite possibly Turing is trolling us, but more likely the ideas of mind reading, extra sensory perception, telepathy, were a little bit more popular in his time.

So the objection here is what if mind reading was used to cheat the test?

So basically, if human to human communication through telepathy could be used, then a machine can't achieve that same kind of telepathic communication.

And so that could be used to circumvent the effectiveness of the test.

Now, Turing's response to this is, well, you just have to design a room that not only protects you from being able to see, whether it's a robot or a human, but also design a telepathy-proof room that prevents telepathic communication.

Again, could be Turing trolling us, but I think more importantly, I think it's a nice illustration at the time, and even still today, that there's a lot of mystery about how our mind works.

If you chuckle and completely laugh off the possibility of telepathic communication, I think you're assuming too much about your own knowledge about how our mind works.

I think we know very little about how our mind works.

It is true, we have very little scientific evidence of telepathic communication, but you shouldn't take the next leap and have a feeling like you understand that telepathic communication is impossible.

You should nevertheless maintain an open mind.

But as an objection, it doesn't seem to be a very effective 1.

I wanted to dedicate just 1 slide and probably the most famous objection to the Turing Test proposed by John Searle in 1980 in his paper, Minds, Brains, and programs, commonly known as the Chinese Room Thought Experiment.

And it's kind of a combination of number 4, number 6 and number 8 objections on the previous slide, which is the consciousness is required for intelligence, the Ada Lovelace objection that programs can only do what we program them to do, and the deterministic free will objection that deterministic rules lead to predictable behavior, and that doesn't seem to be like what the mind does.

So there's echoes of all those objections that Turing anticipated all put together into the Chinese room.

I did not sleep last night, so This video is brought to you by this magic potion called Nitro Cold Brew, an excessively expensive canned beverage from Starbucks that fuels me this wonderful Saturday morning.

Okay, the Chinese room involves following instructions of an algorithm.

So there's a human sitting inside a room that doesn't know how to speak Chinese, but there's notes being passed to them inside the room from outside in Chinese, and all they do is follow a set of rules in order to respond to that language.

So the idea is if the brain inside the system that passes the Turing test is simply following a set of rules that it's not truly understanding.

So there's not, for my computer science engineering self, there's not enough meat in it to even make it that interesting.

It's very human-centric, but allow us to explore it further.

So the key argument is that programs, computational systems, are formal, and so they can capture syntactic structure.

Minds, our brains, have mental content, so they can capture semantics.

And so the claim that I think is the most important and clearest in the paper is that syntax by itself is neither constitutive of nor sufficient for semantics.

So just because you can replicate the syntax of the language doesn't mean you can truly understand it.

Now this is the same kind of criticism we hear of language models of today with transformers, that OpenAI's GP2 really doesn't understand the language, it's just mimicking the statistics of it so well that it can generate syntactically correct and even like have echoes of semantic structure that indicates some kind of understanding, but it doesn't.

To me, that argument is not very interesting from an engineering perspective because it just sounds like saying humans can understand things, humans are special, therefore machines cannot understand things.

It's a very human-centric argument that's not allowing us to rigorously explore what exactly does understanding mean from a computational perspective.

Or put in other words, if understanding, intelligence, consciousness, either 1 of those is not achievable through computation, then where is the point that computation hits the wall?

The most interesting open questions to me here are on the point of faking things, or mimicking, or the appearance of things.

Does the mimicking of thinking equal thinking?

Does the mimicking of consciousness equal consciousness?

This is something that I think a lot about and depending on the day go back and forth.

But I tend to believe from an engineering perspective, I tend to agree with the spirit and the work of Alan Turing in that at this time as engineers, we can only focus on building the appearance of thinking, the appearance of consciousness, the appearance of love.

I think as we work towards creating that appearance, we'll actually begin to understand the fundamentals of what it means to be conscious, what it means to love, what it means to think.

You may have even heard me say sometimes that the appearance of consciousness is consciousness.

I think that's me being a little bit poetic, but I think from our perspective, from our exceptionally limited understanding, both problems are in the same direction.

So it's not like if we focus on creating the appearance of consciousness, that's gonna lead us astray.

In my personal view, it's going to lead us very far down the road of actually understanding and maybe 1 day engineering consciousness.

And now I'd like to talk about some alternatives and variations of the Turing test that I find quite interesting.

So there's a lot of kind of natural variations and extensions to the Turing test.

First, the total Turing test proposed in 1989.

See all Lex Fridman transcripts on Youtube

Turing Test: Can Machines Think?