This is a clip from a conversation with David Ferrucci from Oct 2019. New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. You can watch the full conversation here: https://www.youtube.com/watch?v=Whtt2H5_isM
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on iTunes:
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Note: I select clips with insights from these much longer conversation with the hope of helping make these ideas more accessible and discoverable. Ultimately, this podcast is a small side hobby for me with the goal of sharing and discussing ideas. For now, I post a few clips every Tue & Fri. I did a poll and 92% of people either liked or loved the posting of daily clips, 2% were indifferent, and 6% hated it, some suggesting that I post them on a separate YouTube channel. I hear the 6% and partially agree, so am torn about the whole thing. I tried creating a separate clips channel but the YouTube algorithm makes it very difficult for that channel to grow unless the main channel is already very popular. So for a little while, I'll keep posting clips on the main channel. I ask for your patience and to see these clips as supporting the dissemination of knowledge contained in nuanced discussion. If you enjoy it, consider subscribing, sharing, and commenting.

David Ferrucci led the team that built Watson, the IBM question-answering system that beat the top humans in the world at the game of Jeopardy. He is also the Founder, CEO, and Chief Scientist of Elemental Cognition, a company working engineer AI systems that understand the world the way people do.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

So 1 of the greatest accomplishments in the history of AI is Watson competing in the game of Jeopardy against humans.

And you were a lead in that, a critical part of that.

The game for us humans, human versus human.

Right, so it's to take a question and answer it.

It's really not, it's really to get a question and answer, but it's what we call a factoid question.

So this notion of like it really relates to some fact that few people would argue whether the facts are true or not.

Jeopardy kind of counts on the idea that these statements have factual answers.

And the idea is to, first of all, determine whether or not you know the answer, which is sort of an interesting twist.

So first of all, understand the question.

And that's a good point, because the questions are not asked directly, right?

They're all like, the way the questions are asked is nonlinear.

It's like, it's a little bit witty, it's a little bit playful sometimes.

Yeah, they're asked in, exactly, in numerous witty, tricky ways.

Exactly what they're asking is not obvious.

It takes inexperienced humans a while to go, what is it even asking?

And that's sort of an interesting realization that you have when somebody says, oh, Jeopardy is a question answering show.

And then you read it and you're still trying to process the question and the champions have answered and moved on.

They're 3 questions ahead by the time you figured out what the question even meant.

So there's definitely an ability there to just parse out what the question even is.

It's interesting historically though, if you look back at the Jeopardy games much earlier, you know, early games, the questions were much more direct.

They got sort of more and more interesting.

The way they asked them, that sort of got more and more interesting and subtle and nuanced and humorous and witty over time, which really required the human to kind of make the right connections in figuring out what the question was even asking.

So yeah, you have to figure out the questions even asking.

Then you have to determine whether or not you think you know the answer.

And because you have to buzz in really quickly, you sort of have to make that determination as quickly as you possibly can, otherwise you lose the opportunity to buzz in.

You may- Even before you really know if you know the answer.

I think a lot of humans will assume they'll look at, process it very superficially.

In other words, what's the topic, what are some keywords, and just say, do I know this area or not, before they actually know the answer.

Now some people who know all things, like Ken Jennings or something, or the more recent Big Jeopardy player, I mean, they'll just buzz in.

They'll just assume they know all about Jeopardy and they'll just buzz in.

You know, Watson, interestingly, didn't even come close to knowing all of Jeopardy, right?

Yeah, so for example, I mean, we had this thing called recall, which is like how many of all the Jeopardy questions, you know, how many could we even find, like find the right answer for, like anywhere?

Like could we come up with, if we had a big body of knowledge, something in the order of several terabytes, I mean, from a web scale, it was actually very small, but from a book scale, I was talking about millions of books, right?

So they're calling millions of books, encyclopedias, dictionaries, books, still a ton of information.

And for, I think it was only 85% was the answer anywhere to be found.

So you're already down at that level just to get started, right?

And so it was important to get a very quick sense of, do you think you know the right answer to this question?

So we had to compute that confidence as quickly as we possibly could.

So in effect, we had to answer it and at least spend some time essentially answering it and then judging the confidence that our answer was right and then deciding whether or not we were confident enough to buzz in.

And that would depend on what else was going on in the game, it could because there was a risk.

So like if you're really in a situation where I have to take a guess, I have very little to lose, then you'll buzz in with less confidence.

So that was accounted for the financial standings of the different competitors.

Correct, how much of the game was laughed, how much time was laughed, where you were in the standing, things like that.

How many hundreds of milliseconds that we're talking about here?

Do you have a sense of what is, like if, what's the target?

So, I mean, we targeted answering in under 3 seconds.

And buzzing in, so the decision to buzz in and then the actual answering, are those 2 different?

In fact, we had multiple stages, whereas we would say, let's estimate our confidence, which was sort of a shallow answering process.

And then ultimately decide to buzz in, and then we may take another second or something to kind of go in there and do that.

But by and large, we're saying like, we can't play the game, we can't even compete if we can't, on average, answer these questions in around 3 seconds or less.

So you stepped in, so there's these 3 humans playing a game, and you stepped in with the idea that IBM Watson would be 1 of, replace 1 of the humans and compete against 2.

Can you tell the story of Watson taking on this game?

Yeah, so the story was that it was coming up, I think, to the 10 year anniversary of Big Blue, not Big Blue.

IBM wanted to do sort of another kind of really fun challenge, public challenge, that can bring attention to IBM research and the kind of the cool stuff that we were doing.

I had been working in AI at IBM for some time.

I had a team doing what's called open domain factoid question answering, which is, you know, we're not going to tell you what the questions are.

We're not even going to tell you what they're about.

Can you go off and get accurate answers to these questions?

And it was an area of AI research that I was involved in.

And so it was a very specific passion of mine.

Language understanding had always been a passion of mine.

1 sort of narrow slice on whether or not you could do anything with language was this notion of open domain, meaning I could ask anything about anything.

Factoid, meaning it essentially had an answer and being able to do that accurately and quickly.

So that was a research area that my team had already been in.

And so completely independently, several IBM executives, like, what are we going to do?

And Ken Jennings was on his winning streak.

This was like, whatever it was, 2004, I think, was on his winning streak.

And someone thought, hey, that'd be really cool if the computer can play Jeopardy.

And so this was like in 2004, they were shopping this thing around.

And everyone was telling the research execs, no way, like this is crazy.

And we had some pretty senior people in the field and they're saying, no, this is crazy.

And it'll come across my desk and I was like, but that's kind of what I'm really interested in doing.

But there was such this prevailing sense of, this is nuts, we're not gonna risk IBM's reputation on this, we're just not doing it.

And this happened in 2004, it happened in 2005.

it was coming around again, and I was coming off of a, I was doing the open domain question answering stuff, but I was coming off a couple other projects.

I had a lot more time to put into this, and I argued that it could be done, and I argued it would be crazy not to do this.

So even though you argued for it, what's the confidence that you had yourself, privately, that this could be done?

We just told the story of how you tell stories to convince others.

What was your estimation of the problem at that time?

So I thought it was possible, and a lot of people thought it was impossible.

A reason why I thought it was possible was because I did some brief experimentation.

I knew a lot about how we were approaching open domain factoid question answering.

I said, this is gonna be hard for a lot of the points that we mentioned earlier.

Hard to interpret the question, hard to do it quickly enough, hard to compute an accurate confidence.

None of this stuff had been done well enough before.

But a lot of the technologies we're building were the kinds of technologies that should work.

But more to the point, what was driving me was, I was in IBM Research, I was a senior leader in IBM Research, and this is the kind of stuff we were supposed to do.

In other words, we were basically supposed to take things.

I mean, we were supposed to take things and say this is an active research area.

It's our obligation to kind of, if we have the opportunity, to push it to the limits.

And If it doesn't work, to understand more deeply why we can't do it.

And so I was very committed to that notion, saying, folks, this is what we do.

Why wouldn't we take this grand challenge and push it as hard as we can, at the very least, we'd be able to come out and say, here's why this problem is way hard.

Here's what we tried and here's how we failed.

So I was very driven as a scientist from that perspective.

And then I also argued based on what we did, a feasibility study, why I thought it was hard but possible.

And I showed examples of where it succeeded, where it failed, why it failed, and sort of a high-level architectural approach for why we should do it.

But for the most part, at that point, the execs really were just looking for someone crazy enough to say yes, because for several years at that point, everyone had said no, I'm not willing to risk my reputation and my career on this thing.

you dived right in, and yet, from what I understand, it was performing very poorly in the beginning.

So what were the initial approaches and why did they fail?

Well, there were lots of hard aspects to it.

I mean, 1 of the reasons why prior approaches that we had worked on in the past failed was because the questions were difficult to interpret.

Very often if the question was very direct, like what city, or even then it could be tricky, but what city or what person, often when it would name it very clearly, you would know that.

And if there were just a small set of them, in other words, we're gonna ask about these 5 types.

Like it's gonna be an answer, and the answer will be a city in this state or a city in this country.

The answer will be a person of this type, right?

But it turns out that in Jeopardy, there were like tens of thousands of these things.

And it was a very, very long tail, meaning it just went on and on.

And so even if you focused on trying to encode the types at the very top, like there's 5 that were the most, let's say 5 of the most frequent, you still cover a very small percentage of the data.

So you couldn't take that approach of saying, I'm just going to try to collect facts about these 5 or 10 types or 20 types or 50 types or whatever.

So That was like 1 of the first things, like, what do you do about that?

And so we came up with an approach toward that.

And we continued to improve our ability to handle that problem throughout the project.

The other issue was that, right from the outset, I said, we're not going to, I committed to doing this in 3 to 5 years.

But 1 of the things that that, putting that stake in the ground was, and I knew how hard the language understanding problem was, I said, we're not going to actually understand language to solve this problem.

We are not going to interpret the question and the domain of knowledge that the question refers to in reason over that to answer these questions.

At the same time, simple search wasn't good enough to confidently answer with a single correct answer.

That's such a great mix of innovation in practical engineering, 348.

So you're not trying to solve the general NLU problem.

You're saying, let's solve this in any way possible.

Oh, yeah, no, I was committed to saying, look, we're just solving the open domain question-answering problem.

We're using Jeopardy as a driver for that.

We could just like whatever, like just figure out what works because I wanna be able to go back to the academic science community and say, here's what we tried, here's what worked, here's what didn't work.

I don't wanna go in and say, oh, I only have 1 technology, I have a hammer, I'm only gonna use this.

I'm like, I'm gonna think out of the box, I'm gonna do whatever it takes.

And I also, there was another thing I believed.

I believed that the fundamental NLP technologies and machine learning technologies would be adequate.

And this was an issue of how do we enhance them?

So I had 1 researcher who had been working on question answering with me for a very long time who had said, we're gonna need Maxwell's equations for question answering.

And I said, if we need some fundamental formula that breaks new ground in how we understand language, we're screwed.

Like, I am not counting, my assumption is I'm not counting on some brand new invention.

What I'm counting on is the ability to take everything it has done before to figure out an architecture on how to integrate it well and then see where it breaks and make the necessary advances we need to make until this thing works.

Yeah, push it hard to see where it breaks and then patch it up.

I mean, that's how people change the world.

I mean, that's the Elon Musk approach with rockets, SpaceX, that's the Henry Ford and so on.

And I happen to be, in this case, I happen to be right.

But you kind of have to put a stake in it, so how are you gonna run the project?

So if you were to do, what's the brute force solution?

So you have a question, how would you search the possible space of answers?

Look, web search has come a long way even since then.

But at the time, first of all, I mean there are a couple of other constraints around the problem which is interesting.

So you couldn't go out to the web, you couldn't search the internet.

In other words, the AI experiment was, we want a self-contained device.

The device, If the device is as big as a room, fine, it's as big as a room, but we want a self-contained device.

You're not going out to the internet, you don't have a lifeline to anything.

So it had to kind of fit in a shoebox, if you will, or at least the size of a few refrigerators, whatever it might be.

But also, you couldn't just get out there.

You couldn't go off network, right, to kind of go.

But then we did, But the basic thing was go do a web search.

The problem was even when we went and did a web search, I don't remember exactly the numbers, but somewhere in the order of

of the time, the answer would be somewhere in the top 10 or 20 documents.

So first of all, that's not even good enough to play Jeopardy.

You know, in other words, even if you could pull the, even if you could perfectly pull the answer out of the top 20 documents, top 10 documents, whatever it was, which we didn't know how to do, but even if you could do that, and you knew it was right unless you had enough confidence in it, right?

So you'd have to pull out the right answer, you'd have to have confidence it was the right answer, and then you'd have to do that fast enough to now go buzz in, and you'd still only get 65% of them right, which doesn't even put you in the winner's circle.

But now the problem is, well, even if I had somewhere in the top 10 documents, how do I figure out where in the top 10 documents that answer is and how do I compute a confidence of all the possible candidates?

So it's not like I go in knowing the right answer and have to pick it.

I have a bunch of documents, somewhere in there is the right answer.

How do I, as a machine, go out and figure out which one's right?

So, and now how do I deal with the fact that I can't actually go out to the web?

First of all, if you pause on that, just think about it.

If you could go to the web, do you think that problem is solvable?

If you just pause on it, just thinking even beyond Jeopardy, Do you think the problem of reading text defined where the answer is?

Well, we solved that in some definition of solved given the Jeopardy challenge.

So how do you take a body of work on a particular topic and extract the key pieces of information.

So now forgetting about the huge volumes that are on the web, right?

So now we have to figure out, we did a lot of source research.

In other words, what body of knowledge is gonna be small enough but broad enough to answer a jabbery.

And we ultimately did find the body of knowledge that did that, I mean it included Wikipedia and a bunch of other stuff.

So like encyclopedia type of stuff, I don't know if you can speak

Encyclopedia, dictionaries, different types of semantic resources, unlike WordNet and other types of semantic resources like that, as well as like some web crawls.

In other words, where we went out and took that content and then expanded it based on producing statistical, statistically producing seeds, using those seeds for other searches, and then expanding that.

So using these expansion techniques, we went out and found enough content and we're like, okay, this is good.

And even up until the end, we had a thread of research that was always trying to figure out what content could we efficiently include.

mean, there's a lot of popular, like what is the church lady?

Where do you, I guess that's probably an encyclopedia.

But then we would take that stuff and we would go out and we would expand.

In other words, we go find other content that wasn't in the core resources and expand it.

The amount of content, we grew it by an order of magnitude, but still, again, from a web scale perspective, this is a very small amount of content.

We then took all that content, we pre-analyzed the crap out of it, meaning we parsed it, broke it down into all those individual words, and we did semantic, static and semantic parses on it, had computer algorithms that annotated it, and we indexed that in a very rich and very fast index.

So we have a relatively huge amount of, let's say the equivalent of, for the sake of argument, 2 to 5000000 bucks.

We've now analyzed all that, blowing up its size even more because now we have all this metadata.

And then we richly indexed all of that and by way in a giant in-memory cache.

So the infrastructure component there, if you could just speak to it, how tough it, I mean, I know 2000, maybe this is 2008, 9, you know, that's kind of a long time ago.

How hard is the infrastructure component, the hardware component?

We had something like, I forget exactly, but close to 3,000 cores completely connected.

So you had a switch where every CPU was connected to every other CPU.

And they were sharing memory in some kind of way.

And all this data was pre-analyzed and put into a very fast indexing structure that was all in memory.

And then we took that question, we would analyze the question, so all the content was now pre-analyzed.

So if I went and tried to find a piece of content, it would come back with all the metadata that we had pre-computed.

How do you shove that question, how do you connect the big stuff, the big knowledge base of the metadata that's indexed to the simple little witty confusing question?

Right, so therein lies the Watson architecture.

So we would take the question, we would analyze the question.

So which means that we would parse it and interpret it a bunch of different ways.

We'd try to figure out what is it asking about?

So we would come, we had multiple strategies to kind of determine what was it asking for.

That might be represented as a simple string, a character string, or something we would connect back to different semantic types that were from existing resources.

So anyway, the bottom line is we would do a bunch of analysis in the question.

And question analysis had to finish, and it had to finish fast.

So we do the question analysis because then from the question analysis, we would now produce searches.

See all Lex Fridman transcripts on Youtube

David Ferrucci: The Story of IBM Watson Winning in Jeopardy | AI Podcast Clips