Full episode with Rohit Prasad (Dec 2019): https://www.youtube.com/watch?v=Ad89JYS-uZM
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Rohit Prasad is the vice president and head scientist of Amazon Alexa and one of its original creators.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

And what have you learned and what's surprising?

What have you seen that's surprising from this incredible competition?

Absolutely, it's a very exciting competition.

Alexa Prize is essentially a grand challenge in conversational artificial intelligence, where we threw the gauntlet to the universities who do active research in the field to say, can you build what we call a social bot that can converse with you coherently and engagingly for 20 minutes?

That is an extremely hard challenge, talking to someone who you're meeting for the first time, or even if you've met them quite often, To speak at 20 minutes on any topic, an evolving nature of topics, is super hard.

We have completed 2 successful years of the competition.

The first was 1 with the University of Washington, second, the University of California.

We have an extremely strong team of 10 cohorts and the third instance of the Alexa Prize is underway now.

It was a lot of things to be put together.

We had to build a lot of infrastructure to enable these universities to be able to build magical experiences and do high quality research.

Just a few quick questions, Sorry for the interruption.

What does failure look like in the 20 minute session?

So what does it mean to fail not to reach the 20 minute mark?

So there are 1, first of all, I forgot to mention 1 more detail.

It's not just 20 minutes, but the quality of the conversation too that matters.

And the beauty of this competition, before I answer that question on what failure means, is first that you actually converse with millions and millions of customers as the social bots.

So during the judging phases, there are multiple phases, Before we get to the finals, which is a very controlled judging in a situation where we bring in judges and we have interactors who interact with these social bots, that is a much more controlled setting.

But till the point we get to the finals, all the judging is essentially by the customers of Alexa.

And there you basically rate on a simple question how good your experience was.

So that's where we are not testing for a 20 minute boundary being crossed because you do want it to be very much like a clear cut winner be chosen and it's an absolute bar.

So did you really break that 20 minute barrier is why we have to test it in a more controlled setting with actors, essentially interactors, and see how the conversation goes.

So this is why it's a subtle difference between how it's being tested in the field with real customers versus in the lab to award the prize.

So on the latter 1, what it means is that essentially there are 3 judges and 2 of them have to say this conversation has stalled, essentially.

Got it, and the judges are human experts.

So the DARPA challenge in the first year, the autonomous vehicles, nobody finished.

In the second year, a few more finished in the desert.

So how far along in this, I would say, much harder challenge, are we?

This challenge has come a long way to the extent that we're definitely not close to the 20 minute barrier being with coherence and engaging conversation.

I think we are still 5 to 10 years away in that horizon to complete that.

Like, what you're finding is the accuracy in what kind of responses these social bots generate is getting better and better.

What's even amazing to see that now there's humor coming in.

You know, you're talking about ultimate science of intelligence.

I think humor is a very high bar in terms of what it takes to create humor.

I really mean good sense of humor is also a sign of intelligence in my mind and something very hard to do.

So these social bots are now exploring not only what we think of natural language abilities, but also personality attributes and aspects of when to inject an appropriate joke, when you don't know the domain, how you come back with something more intelligible so that you can continue the conversation.

If you and I are talking about AI and we are domain experts, we can speak to it, but if you suddenly switch a topic to that I don't know of, how do I change the conversation?

So you're starting to notice these elements as well.

And that's coming from partly by the nature of the 20 minute challenge that people are getting quite clever on how to really converse and essentially mask some of the understanding defects if they exist.

So some of this, this is not Alexa the product, this is somewhat for fun, for research, for innovation and so on.

I have a question, sort of in this modern era, there's a lot of, if you look at Twitter and Facebook and so on, there's discourse, public discourse going on, and some things that are a little bit too edgy, people get blocked and so on.

I'm just out of curiosity, are people in this context pushing the limits?

Is anyone sort of pushing back, sort of, you know, arguing, I guess I should say, as part of the dialogue to really draw people in?

First of all, let me just back up a bit in terms of why we are doing this, right?

I think fun is more part of the engaging part for customers.

It is 1 of the most used skills as well in our skill store.

But that apart, the real goal was essentially, what was happening is, with a lot of AI research moving to industry, we felt that academia has the risk of not being able to have the same resources at disposal that we have, which is lots of data, massive computing power and a clear ways to test these AI advances with real customer benefits.

So we brought all these 3 together in the Alexa price.

That's why it's 1 of my favorite projects in Amazon.

And with that, the secondary effect is, yes, it has become engaging for our customers as well.

We're not there in terms of where we want it to be, right?

But coming back to your question on how do the conversations evolve?

Yes, there is some natural attributes of what you said in terms of argument and some amount of swearing.

The way we take care of that is that there is a sensitive filter we have built.

It's more than keywords, a little more in terms of, of course, there's keyword based too, but there's more in terms of, these words can be very contextual, as you can see, and also the topic can be something that you don't want a conversation to happen, because this is a communal device as well.

So we have put a lot of guardrails for the conversation to be more useful for advancing AI and not so much of these other issues you attributed, what's happening in the AI field as well.

Right, so this is actually a serious opportunity.

I think it's an open opportunity to do some of the best innovation and conversational agents in the world.

It's also to, if you think about the other aspect of where the whole industry is moving with AI, there's a dearth of talent given the demands.

So you do want universities to have a clear place where they can invent and research and not fall behind with that they can't motivate students.

Imagine all grad students left to industry like us or faculty members, which has happened too.

So this is a way that if you're so passionate about the field where you feel industry and academia need to work well, this is a great example and a great way for universities to participate.

you think it takes to build a system that wins the Alexa Prize?

I think you have to start focusing on aspects of reasoning that it is, there are still more lookups of what intents the customer is asking for and responding to those rather than really reasoning about the elements of the conversation.

For instance, if you're playing, if the conversation is about games and it's about a recent sports event, there's so much context involved and you have to understand the entities that are being mentioned so that the conversation is coherent rather than you suddenly just switch to knowing some fact about a sports entity and you're just relaying that rather than understanding the true context of the game.

If you just said, I learned this fun fact about Tom Brady rather than really say how he played the game the previous night, then the conversation is not really that intelligent.

So you have to go to more reasoning elements of understanding the context of the dialogue and giving more appropriate responses, which tells you that we are still quite far because a lot of times it's more facts being looked up and something that's close enough as an answer, but not really the answer.

So That is where the research needs to go more, an actual true understanding and reasoning.

And that's why I feel it's a great way to do it because you have an engaged set of users working to make, help these AI advances happen in this case.

You mentioned customers there quite a bit, and there's a skill.

What is the experience for the user that's helping?

So just to clarify, this isn't, as far as I understand, the Alexa, so this skill is a standalone for the Alexa prize.

It's not you ordering certain things and Amazon.com or checking the weather or playing Spotify, right?

I don't know, how do people, how do customers think of it?

And let me tell you how you invoke the skill.

So all you have to say, Alexa, let's chat.

And then the first time you say, Alexa, let's chat, it comes back with a clear message that you're interacting with 1 of those university social bots.

And there's a clear, so you know exactly how you interact, right?

And we have a lot of mechanisms where as we are in the first phase of feedback phase, then you send a lot of emails to our customers and then they know that the team needs a lot of interactions to improve the accuracy of the system.

So we know we have a lot of customers who really want to help these university bots and they're conversing with that.

And some are just having fun with just saying Alexa, let's chat and also some adversarial behavior to see whether, How much do you understand as a social bot?

So I think we have a good healthy mix of all 3 situations.

So what is the, if we talk about solving the Alexa challenge, the Alexa prize, what's the data set of really engaging pleasant conversations look like?

Because if we think of this as a supervised learning problem, I don't know if it has to be, but if it does, maybe you can comment on that.

Do you think there needs to be a data set of what it means to be an engaging, successful, fulfilling conversation.

I think that's part of the research question here.

This was, I think, we at least got the first part right, which is have a way for universities to build and test in a real world setting.

Now you're asking in terms of the next phase of questions, which we are also asking, by the way, what does success look like from a optimization function?

That's what you're asking in terms of, we as researchers are used to having a great corpus of annotated data and then making, then sort of tune our algorithms on those, right?

And fortunately and unfortunately, in this world of Alexa Prize, that is not the way we are going after it.

So you have to focus more on learning based on live feedback.

That is another element that's unique where just now I started with giving you how you ingress and experience this capability as a customer.

So they ask you a simple question on a scale of 1 to 5, how likely are you to interact with this social bot again?

That is a good feedback and customers can also leave more open-ended feedback.

And I think partly that to me is 1 part of the question you're asking, which I'm saying is a mental model shift that as researchers also, you have to change your mindset that this is not a DARPA evaluation or a NSF funded study and you have a nice corpus, this is where it's real world, you have real data.

And then the customer, the user can quit the conversation at any time.

The user can, that is also a signal for how good you were at that point.

So, and then on a scale of 1 to 5, 1 to 3, do they say how likely are you, or is it just a binary?

That's such a beautifully constructed challenge, okay.


Speaker 0: And what have you learned and what's surprising? What have you seen that's surprising from this incredible competition?

Speaker 1: Absolutely, it's a very exciting competition. Alexa Prize is essentially a grand challenge in conversational artificial intelligence, where we threw the gauntlet to the universities who do active research in the field to say, can you build what we call a social bot that can converse with you coherently and engagingly for 20 minutes? That is an extremely hard challenge, talking to someone who you're meeting for the first time, or even if you've met them quite often, To speak at 20 minutes on any topic, an evolving nature of topics, is super hard. We have completed 2 successful years of the competition. The first was 1 with the University of Washington, second, the University of California.

We are in our third instance. We have an extremely strong team of 10 cohorts and the third instance of the Alexa Prize is underway now. And we are seeing a constant evolution. First year was definitely a learning. It was a lot of things to be put together.

We had to build a lot of infrastructure to enable these universities to be able to build magical experiences and do high quality research.

Speaker 0: Just a few quick questions, Sorry for the interruption. What does failure look like in the 20 minute session? So what does it mean to fail not to reach the 20 minute mark?

Speaker 1: Oh, awesome question. So there are 1, first of all, I forgot to mention 1 more detail. It's not just 20 minutes, but the quality of the conversation too that matters. And the beauty of this competition, before I answer that question on what failure means, is first that you actually converse with millions and millions of customers as the social bots. So during the judging phases, there are multiple phases, Before we get to the finals, which is a very controlled judging in a situation where we bring in judges and we have interactors who interact with these social bots, that is a much more controlled setting.

But till the point we get to the finals, all the judging is essentially by the customers of Alexa. And there you basically rate on a simple question how good your experience was. So that's where we are not testing for a 20 minute boundary being crossed because you do want it to be very much like a clear cut winner be chosen and it's an absolute bar. So did you really break that 20 minute barrier is why we have to test it in a more controlled setting with actors, essentially interactors, and see how the conversation goes. So this is why it's a subtle difference between how it's being tested in the field with real customers versus in the lab to award the prize.

So on the latter 1, what it means is that essentially there are 3 judges and 2 of them have to say this conversation has stalled, essentially.

Speaker 0: Got it, and the judges are human experts. Judges

Speaker 1: are human experts.

Speaker 0: Okay, great. So this is in the third year. So what's been the evolution? How far? So the DARPA challenge in the first year, the autonomous vehicles, nobody finished.

In the second year, a few more finished in the desert. So how far along in this, I would say, much harder challenge, are we?

Speaker 1: This challenge has come a long way to the extent that we're definitely not close to the 20 minute barrier being with coherence and engaging conversation. I think we are still 5 to 10 years away in that horizon to complete that. But the progress is immense. Like, what you're finding is the accuracy in what kind of responses these social bots generate is getting better and better. What's even amazing to see that now there's humor coming in.

The bots are quite... Awesome. You know, you're talking about ultimate science of intelligence. I think humor is a very high bar in terms of what it takes to create humor. And I don't mean just being goofy.

I really mean good sense of humor is also a sign of intelligence in my mind and something very hard to do. So these social bots are now exploring not only what we think of natural language abilities, but also personality attributes and aspects of when to inject an appropriate joke, when you don't know the domain, how you come back with something more intelligible so that you can continue the conversation. If you and I are talking about AI and we are domain experts, we can speak to it, but if you suddenly switch a topic to that I don't know of, how do I change the conversation? So you're starting to notice these elements as well. And that's coming from partly by the nature of the 20 minute challenge that people are getting quite clever on how to really converse and essentially mask some of the understanding defects if they exist.

Speaker 0: So some of this, this is not Alexa the product, this is somewhat for fun, for research, for innovation and so on. I have a question, sort of in this modern era, there's a lot of, if you look at Twitter and Facebook and so on, there's discourse, public discourse going on, and some things that are a little bit too edgy, people get blocked and so on. I'm just out of curiosity, are people in this context pushing the limits? Is anyone using the F word? Is anyone sort of pushing back, sort of, you know, arguing, I guess I should say, as part of the dialogue to really draw people in?

Speaker 1: First of all, let me just back up a bit in terms of why we are doing this, right? So you said it's fun. I think fun is more part of the engaging part for customers. It is 1 of the most used skills as well in our skill store. But that apart, the real goal was essentially, what was happening is, with a lot of AI research moving to industry, we felt that academia has the risk of not being able to have the same resources at disposal that we have, which is lots of data, massive computing power and a clear ways to test these AI advances with real customer benefits.

So we brought all these 3 together in the Alexa price. That's why it's 1 of my favorite projects in Amazon. And with that, the secondary effect is, yes, it has become engaging for our customers as well. We're not there in terms of where we want it to be, right? But it's a huge progress.

But coming back to your question on how do the conversations evolve? Yes, there is some natural attributes of what you said in terms of argument and some amount of swearing. The way we take care of that is that there is a sensitive filter we have built. That's these... Keywords and so on.

It's more than keywords, a little more in terms of, of course, there's keyword based too, but there's more in terms of, these words can be very contextual, as you can see, and also the topic can be something that you don't want a conversation to happen, because this is a communal device as well. A lot of people use these devices. So we have put a lot of guardrails for the conversation to be more useful for advancing AI and not so much of these other issues you attributed, what's happening in the AI field as well.

Speaker 0: Right, so this is actually a serious opportunity. I didn't use the right word, fun. I think it's an open opportunity to do some of the best innovation and conversational agents in the world.

Speaker 1: Absolutely.

Speaker 0: Why just universities?

Speaker 1: Why just universities? Because as I said, I really felt-

Speaker 0: Young minds.

Speaker 1: Young minds. It's also to, if you think about the other aspect of where the whole industry is moving with AI, there's a dearth of talent given the demands. So you do want universities to have a clear place where they can invent and research and not fall behind with that they can't motivate students. Imagine all grad students left to industry like us or faculty members, which has happened too. So this is a way that if you're so passionate about the field where you feel industry and academia need to work well, this is a great example and a great way for universities to participate.

So what do

Speaker 0: you think it takes to build a system that wins the Alexa Prize?

Speaker 1: I think you have to start focusing on aspects of reasoning that it is, there are still more lookups of what intents the customer is asking for and responding to those rather than really reasoning about the elements of the conversation. For instance, if you're playing, if the conversation is about games and it's about a recent sports event, there's so much context involved and you have to understand the entities that are being mentioned so that the conversation is coherent rather than you suddenly just switch to knowing some fact about a sports entity and you're just relaying that rather than understanding the true context of the game. If you just said, I learned this fun fact about Tom Brady rather than really say how he played the game the previous night, then the conversation is not really that intelligent. So you have to go to more reasoning elements of understanding the context of the dialogue and giving more appropriate responses, which tells you that we are still quite far because a lot of times it's more facts being looked up and something that's close enough as an answer, but not really the answer. So That is where the research needs to go more, an actual true understanding and reasoning.

And that's why I feel it's a great way to do it because you have an engaged set of users working to make, help these AI advances happen in this case.

Speaker 0: You mentioned customers there quite a bit, and there's a skill. What is the experience for the user that's helping? So just to clarify, this isn't, as far as I understand, the Alexa, so this skill is a standalone for the Alexa prize. I mean, it's focused on the Alexa prize. It's not you ordering certain things and Amazon.com or checking the weather or playing Spotify, right?

It's a separate skill. And so you're focused on helping that. I don't know, how do people, how do customers think of it? Are they having fun? Are they helping teach the system?

What's the experience like?

Speaker 1: I think it's both actually. And let me tell you how you invoke the skill. So all you have to say, Alexa, let's chat. And then the first time you say, Alexa, let's chat, it comes back with a clear message that you're interacting with 1 of those university social bots. And there's a clear, so you know exactly how you interact, right?

And that is why it's very transparent. You are being asked to help, right? And we have a lot of mechanisms where as we are in the first phase of feedback phase, then you send a lot of emails to our customers and then they know that the team needs a lot of interactions to improve the accuracy of the system. So we know we have a lot of customers who really want to help these university bots and they're conversing with that. And some are just having fun with just saying Alexa, let's chat and also some adversarial behavior to see whether, How much do you understand as a social bot?

Speaker 0: So I think we have a good healthy mix of all 3 situations. So what is the, if we talk about solving the Alexa challenge, the Alexa prize, what's the data set of really engaging pleasant conversations look like? Because if we think of this as a supervised learning problem, I don't know if it has to be, but if it does, maybe you can comment on that. Do you think there needs to be a data set of what it means to be an engaging, successful, fulfilling conversation.

Speaker 1: I think that's part of the research question here. This was, I think, we at least got the first part right, which is have a way for universities to build and test in a real world setting. Now you're asking in terms of the next phase of questions, which we are also asking, by the way, what does success look like from a optimization function? That's what you're asking in terms of, we as researchers are used to having a great corpus of annotated data and then making, then sort of tune our algorithms on those, right? And fortunately and unfortunately, in this world of Alexa Prize, that is not the way we are going after it.

So you have to focus more on learning based on live feedback. That is another element that's unique where just now I started with giving you how you ingress and experience this capability as a customer. What happens when you're done? So they ask you a simple question on a scale of 1 to 5, how likely are you to interact with this social bot again? That is a good feedback and customers can also leave more open-ended feedback.

And I think partly that to me is 1 part of the question you're asking, which I'm saying is a mental model shift that as researchers also, you have to change your mindset that this is not a DARPA evaluation or a NSF funded study and you have a nice corpus, this is where it's real world, you have real data.

Speaker 0: The scale is amazing. It's amazing. That's a beautiful thing. And then the customer, the user can quit the conversation at any time.

Speaker 1: Exactly. The user can, that is also a signal for how good you were at that point.

Speaker 0: So, and then on a scale of 1 to 5, 1 to 3, do they say how likely are you, or is it just a binary? 1 to 5. 1 to 5. Wow, okay. That's such a beautifully constructed challenge, okay.

See all Lex Fridman transcripts on Youtube

Rohit Prasad: Alexa Prize | AI Podcast Clips