The following is a conversation with Daniel Kahneman, winner of the Nobel Prize in Economics for his integration of economic science with the psychology of human behavior, judgment and decision-making.

He's the author of the popular book, Thinking Fast and Slow, that summarizes in an accessible way his research of several decades, often in collaboration with Amos Tversky, on cognitive biases, prospect theory, and happiness.

The central thesis of this work is the dichotomy between 2 modes of thought.

What he calls system 1 is fast, instinctive and emotional.

System 2 is slower, more deliberative and more logical.

The book delineates cognitive biases associated with each of these 2 types of thinking.

His study of the human mind and his peculiar and fascinating limitations are both instructive and inspiring for those of us seeking to engineer intelligence systems.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, give it 5 stars on Apple Podcasts, follow on Spotify, support it on Patreon, or simply connect with me on Twitter, Alex Friedman, spelled F-R-I-D-M-A-N.

I recently started doing ads at the end of the introduction.

I'll do 1 or 2 minutes after introducing the episode and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash app, the number 1 finance app in the app store.

I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin in just seconds.

Cash App also has a new investing feature.

You can buy fractions of a stock, say $1 worth, no matter what the stock price is.

Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC.

I'm excited to be working with Cash App to support 1 of my favorite organizations called FIRST, best known for their first robotics and Lego competitions.

They educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating, a charity navigator, which means the donated money is used to maximum effectiveness.

When you get Cash App from the App Store, Google Play, and use code LEXPODCAST, you'll get $10, and Cash App will also donate $10 to FIRST, which again, is an organization that I've personally seen inspire girls and boys to dream of engineering a better world.

And now, here's my conversation with Daniel Kahneman.

You tell a story of an SS soldier early in the war, World War II, in Nazi-occupied France and Paris, where you grew up.

He picked you up and hugged you and showed you a picture of a boy, maybe not realizing that you were Jewish.

So I told you I'm from the Soviet Union that was significantly impacted by the war as well, and I'm Jewish as well.

What do you think World War II taught us about human psychology broadly?

Well, I think the only big surprise is the extermination policy, genocide, by the German people.

That's, when you look back on it, and I think that's a major surprise.

It's a surprise that enough people willingly participated in that.

Now it's no longer a surprise, but it's changed many people's views, I think, about human beings.

Certainly, for me, the Ackman trial teaches you something because it's very clear that if it could happen in Germany, it could happen anywhere.

I think that what is certainly possible is you can dehumanize people so that you treat them not as people anymore, but as animals.

And the same way that you can slaughter animals without feeling much of anything.

It can, the same, and when you feel that, I think the combination of dehumanizing the other side and having uncontrolled power over other people.

I think that doesn't bring out the most generous aspect of human nature.

So that Nazi soldier, you know, he was a good man.

I mean, you know, and he was perfectly capable of killing a lot of people, and I'm sure he did.

But what did the Jewish people mean to Nazis?

So what, the dismissal of Jewish as worthy of...

Again, this is surprising that it was so extreme, but it's not 1 thing in human nature.

I don't want to call it evil, but the distinction between the in-group and the out-group, that is very basic.

The loyalty and affection towards in-group and the willingness to dehumanize the out-group, That is in human nature.

And that's what I think probably didn't need the Holocaust to teach us that.

But the Holocaust is a very sharp lesson of what can happen to people, what people can do.

So the effect of the in-group and the out-group.

You know, it's clear that those were people, you know, you could shoot them, you could, you know, they were not human.

They were not, there was no empathy, or very, very little empathy left.

So occasionally, you know, there might have been, and very quickly, by the way, the empathy disappeared, if there was initially, and the fact that everybody around you was doing it, that completely, the group doing it, and everybody shooting Jews, I think that makes it permissible.

Now, how much, whether it could happen in every culture or whether the Germans were just particularly efficient and disciplined so they could get away with it.

Are these artifacts of history or is it human nature?

You know, you put some people in a position of power relative to other people and then they become less human, they become different.

But in general in war, outside of concentration camps in World War II, it seems that war brings out darker sides of human nature, but also the beautiful things about human nature.

Well, you know, I mean, what it brings out is the loyalty among soldiers.

I mean, it brings out the bonding, male bonding I think is a very real thing that happens.

And there is a certain thrill to friendship and there is certainly a certain thrill to friendship under risk and to shared risk.

And so people have very profound emotions up to the point where it gets so traumatic that little is left.

So let's talk about psychology a little bit.

In your book, Thinking Fast and Slow, you describe 2 modes of thought, system 1, the fast, instinctive, and emotional 1, and system 2, the slower, deliberate, logical 1.

At the risk of asking Darwin to discuss theory of evolution, can you describe distinguishing characteristics for people who have not read your book of the 2 systems?

Well, I mean, the word system is a bit misleading, but it's, at the same time it's misleading, it's also very useful.

But what I call system 1, it's easier to think of it as a family of activities.

And primarily the way I describe it is there are different ways for ideas to come to mind.

And Some ideas come to mind automatically.

And the example, a standard example is 2 plus 2, and then something happens to you.

And in other cases, you've got to do something.

You've got to work in order to produce the idea.

My example, I always give the same pair of numbers as 27 times 14, I think.

You have to perform some algorithm in your head,

It's a very different, nothing comes to mind except something comes to mind, which is the algorithm.

And then it's work and it engages shorter memory and it engages executive function.

And it makes you incapable of doing other things at the same time.

So the main characteristic of system 2 is that there is mental effort involved and there is a limited capacity for mental effort.

Whereas system 1 is effortless essentially.

So you talk about there, you know, it's really convenient to talk about 2 systems, but you also mentioned just now and in general that there is no distinct 2 systems in the brain from a neurobiological, even from psychology perspective.

But why does it seem to, from the experiments you've conducted, there does seem to be kind of emergent 2 modes of thinking.

So at some point, these kinds of systems came into a brain architecture, maybe man will share it, or do you not think of it at all in those terms that it's all a mush and these 2 things just emerge.

You know, evolutionary theorizing about this is cheap and easy.

So it's the way I think about it is that it's very clear that animals have a perceptual system and that includes an ability to understand the world, at least to the extent that they can predict.

They can't explain anything, but they can anticipate what's going to happen, and that's a key form of understanding the world.

And my crude idea is that what I call System 2, well, System 2 grew out of this.

And there is language, and there is the capacity of manipulating ideas, and the capacity of imagining futures, and of imagining counterfactuals, things that haven't happened, and to do conditional thinking, and there are really a lot of abilities that, without language and without the very large brain that we have compared to others, would be impossible.

Now, System 1 is more like what the animals are, but system 1 also can talk.

I mean, it has language, it understands language.

I mean, you know, I'm not choosing every word as a deliberate process.

The words, I have some idea, and then the words come out.

And many of the experiments you've done is to show that listen, system 1 exists and it does speak for us and we should be careful about the voice it provides.

mean, you know, we have to trust it because it's the speed at which it acts.

System 2, if we depend on system 2 for survival, we wouldn't survive very long, because it's very slow.

I mean, many things depend on their being automatic.

1 very important aspect of System 1 is that it's not instinctive.

It contains skills that clearly have been learned.

So that skilled behavior like driving a car or speaking in fact, skilled behavior has to be learned.

And so it doesn't, you know, you don't come equipped with driving.

And you have to go through a period where driving is not automatic before it becomes automatic.

Yeah, you construct, I mean, this is where you talk about heuristic and biases, you make it automatic, you create a pattern, and then system 1 essentially matches a new experience against a previously seen pattern, And when that match is not a good 1, that's when all the mess happens, but most of the time it works, and so it's pretty-

Most of the time, the anticipation of what's going to happen next is correct.

And most of the time, the plan about what you have to do is correct.

So most of the time, everything works just fine.

What's interesting actually is that in some sense, system 1 is much better at what it does, and system 2 is at what it does.

That is, there is this quality of effortlessly solving enormously complicated problems, which clearly exists, so that a chess player, a very good chess player, all the moves that come to their mind are strong moves.

So all the selection of strong moves happens unconsciously and automatically and very, very fast.

So along this line of thinking, really what we are are machines that construct pretty effective system 1.

So we're not talking about humans, but if we think about building artificial intelligence systems, robots, do you think all the features and bugs that you have highlighted in human beings are useful for constructing AI systems?

So both systems are useful for perhaps instilling in robots?

What is happening these days is that actually what is happening in deep learning is more like a system 1 product than like a system 2 product.

I mean, deep learning matches patterns and anticipate what's going to happen, so it's highly predictive.

What deep learning doesn't have, and you know, many people think that this is a critical, it doesn't have the ability to reason, so it doesn't, there is no system to there.

But I think very importantly, it doesn't have any causality or any way to represent meaning and to represent real interaction.

So until that is solved, the, you know, what can be accomplished is marvelous and very exciting, but limited.

That's actually really nice to think of current advances in machine learning is essentially system 1 advances.

So how far can we get with just system 1?

If we think of deep learning and artificial intelligence systems as someone...

I mean, you know, it's very clear that deep mind has already gone way, way beyond what people thought was possible.

I think, I think the thing that has impressed me most about the developments in AI is the speed.

It's that things, at least in the context of deep learning, and maybe this is about to slow down, but things moved a lot faster than anticipated.

The transition from solving chess to solving Go, was, I mean, that's bewildering how quickly it went.

The move from AlphaGo to AlphaZero is sort of bewildering the speed at which they accomplished that.

Now, clearly, there are, so there are many problems that you can solve that way, but there are some problems for which you need something else.

Well, reasoning and also, you know, 1 of the real mysteries, psychologist Gary Marcus, who is also a critic of AI, I mean, he, what He points out, and I think he has a point, is that humans learn quickly.

So clearly there is a fundamental difference.

And what enables a machine to learn quickly, what you have to build into the machine, because it's clear that you have to build some expectations or something in the machine to make it ready to learn quickly.

I'm pretty sure that DeepMind is working on it, but If they have solved it, I haven't heard yet.

They're trying to actually, them and OpenAI are trying to start to get to use neural networks to reason.

So assemble knowledge, of course causality is, temporal causality is out of reach to most everybody.

You mentioned the benefits of system 1 is essentially that it's fast, allows us to function in the world.

You know, in a sense, I mean, there was the earlier phase of AI attempted to model reasoning, and they were moderately successful, but reasoning by itself doesn't get you much.

Deep learning has been much more successful in terms of what they can do.

But now, it's an interesting question, whether it's approaching its limits.

So I just talked to Gian Lacoon, you mentioned him.

So he thinks that the limits, we're not going to hit the limits with neural networks, that ultimately this kind of system 1 pattern matching will start to look like system 2 without significant transformation of the architecture.

So I'm more with the majority of the people who think that yes, neural networks will hit a limit in their capability.

On the 1 hand, I have heard him tell the Mississippis essentially that, you know, what they have accomplished is not a big deal, that they have just touched, that basically, you know, they can't do unsupervised learning in an effective way.

But you're telling me that he thinks that within the current architecture, you can do causality and reasoning?

So he's very much a pragmatist in a sense that's saying that we're very far away, that there's still, I think there's this idea that he says is we can only see 1 or 2 mountain peaks ahead and there might be either a few more after or thousands more after.

Right, but nevertheless it doesn't see a, the final answer not fundamentally looking like 1 that we currently have.

So neural networks being a huge part of that.

Yeah, I mean, that's very likely because pattern matching is so much of what's going on.

And you can think of neural networks as processing information sequentially.

Yeah, I mean, you know, there is an important aspect to, for example, you get systems that translate and they do a very good job, but they really don't know what they're talking about.

And for that, I'm really quite surprised.

For that, you would need an AI that has sensation, an AI that is in touch with the world.

self-awareness, and maybe even something resembles consciousness kind of ideas.

Certainly awareness of what's going on so that the words have meaning or can get, are in touch with some perception or some action.

What he refers to as grounding to the physical space.

So that's what we're talking about the same.

I mean, the grounding, Without grounding, then you get a machine that doesn't know what it's talking about, because it is talking about the world, ultimately.

The question, open question, is what it means to ground.

I mean, we're very human-centric in our thinking, but what does it mean for a machine to understand what it means to be in this world?

Does it need to have a finiteness like we humans have?

All of these elements, it's a very, it's an open question.

You know, I'm not sure about having a body, but having a perceptual system, having a body would be very helpful too, I mean, if you think about human mimicking human, but having a perception, that seems to be essential, so that you can build, you can accumulate knowledge about the world.

So you can imagine a human completely paralyzed, and there is a lot that the human brain could learn, you know, with a paralyzed body.

So if we got a machine that could do that, that would be a big deal.

And then the flip side of that, something you see in children and something in machine learning world is called active learning.

Maybe it is also in, is being able to play with the world.

How important for developing system 1 or system 2 do you think it is to play with the world?

Well, certainly a lot of what you learn is you learn to anticipate the outcomes of your actions.

I mean, you can see that, how babies learn it, you know, with their hands, how they learn, you know, to connect, you know, the movements of their hands with something that clearly is something that happens in the brain, and the ability of the brain to learn new patterns.

So, you know, it's the kind of thing that you get with artificial limbs, that you connect it and then people learn to operate the artificial limb, you know, really impressively quickly, at least from what I hear.

So we have a system that is ready to learn the world through action.

At the risk of going into way too mysterious of land, what do you think it takes to build a system like that?

Obviously, we're very far from understanding how the brain works, but How difficult is it to build this Mind of ours,

you know, I mean, I think that Jan the Coons answer that we don't know how many mountains there are I think that's a very good answer I think that you know, if you if you look at what cool and Ray Kurzweil is saying that strikes me as off the wall.

But I think people are much more realistic than that, where actually Demis Hassabis is and Yanis, and so the people who are actually doing the work fairly realistic, I think.

To maybe phrase it another way, from a perspective not of building it, but from understanding it, How complicated are human beings in the following sense?

You know, I work with autonomous vehicles and pedestrians, so we tried to model pedestrians.

How difficult is it to model a human being, their perception of the world, the 2 systems they operate under, sufficiently to be able to predict whether the pedestrian's gonna cross the road or not.

I'm fairly optimistic about that, actually, Because what we're talking about is a huge amount of information that every vehicle has and that feeds into 1 system, into 1 gigantic system.

And so anything that any vehicle learns becomes part of what the whole system knows.

And with a system multiplier like that, there is a lot that you can do.

So human beings are very complicated, but, and, you know, system is going to make mistakes, but human makes mistakes.

I think that they'll be able to, I think they are able to anticipate pedestrians, otherwise a lot would happen.

They're able to, you know, they're able to get into a roundabout and into traffic, so they must know both to expect or to anticipate how people will react when they're sneaking in.

And there's a lot of learning that's involved in that.

Currently, the pedestrians are treated as things that cannot be hit.

And they're not treated as agents with whom you interact in a game theoretic way.

So, it's a totally open problem and every time somebody tries to solve it, it seems to be harder than we think.

And nobody's really tried to seriously solve the problem of that dance, because I'm not sure if you've thought about the problem of pedestrians, but you're really putting your life in the hands of the driver.

You know, there is a dance, there's part of the dance that would be quite complicated, but for example, when I cross the street and there is a vehicle approaching, I look the driver in the eye, and I think many people do that.

And that's a signal that I'm sending, and I would be sending that machine to an autonomous vehicle and it had better understand it because it means I'm crossing.

So, and there's another thing you do that actually, so I'll tell you what you do because I've watched hundreds of hours of video on this, is when you step in the street, you do that before you step in the street.

And when you step in the street, you actually look away.

Now, what does that, what that's saying is, I mean, you're trusting that the car, who hasn't slown down yet, will slow down.

Yeah, and you're telling him, I'm committed.

I mean, this is like in a game of chicken.

So I'm committed, and if I'm committed, I'm looking away.

So the question is whether a machine that observes that needs to understand mortality.

Here I'm not sure that it's got to understand so much as it's got to anticipate.

So, and here, but you know, you're surprising me because here I would think that maybe you can anticipate without understanding because I think this is clearly what's happening in playing go or in playing chess, there's a lot of anticipation and there is 0 understanding.

So, I thought that you didn't need a model of the human and a model of the human mind to avoid hitting pedestrians.

And I have a follow-up question to see where your intuition lies.

Is it seems that almost every robot human collaboration system is a lot harder than people realize.

So do you think it's possible for robots and humans to collaborate successfully?

We talked a little bit about semi-autonomous vehicles like in the Tesla, Autopilot, but just in tasks in general.

If you think we talked about current neural networks being kind of system 1, Do you think those same systems can borrow humans for System 2 type tasks and collaborate successfully?

Well, I think that in any system where humans and the machine interact, that the human will be superfluous within a fairly short time.

That is, if the machine has advanced enough so that it can really help the human, then it may not need the human for a long time.

Now it would be very interesting if there are problems that for some reason the machine doesn't, cannot solve, but that people could solve.

Then you would have to build into the machine an ability to recognize that it is in that kind of problematic situation, and to call the human.

That cannot be easy without understanding.

That is, it must be very difficult to program a recognition that you are in a problematic situation without understanding the problem.

In order to understand the full scope of situations that are problematic, you almost need to be smart enough to solve

See all Lex Fridman transcripts on Youtube

Daniel Kahneman: Thinking Fast and Slow, Deep Learning, and AI | Lex Fridman Podcast #65