Vladimir Vapnik is the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times.

The associate lecture that Vladimir gave as part of the MIT Deep Learning series can be viewed here: https://www.youtube.com/watch?v=Ow25mjFjSmg

This episode is presented by Cash App. Download it & use code "LexPodcast":
Cash App (App Store): https://apple.co/2sPrUHe
Cash App (Google Play): https://bit.ly/2MlvP5w

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:55 - Alan Turing: science and engineering of intelligence
9:09 - What is a predicate?
14:22 - Plato's world of ideas and world of things
21:06 - Strong and weak convergence
28:37 - Deep learning and the essence of intelligence
50:36 - Symbolic AI and logic-based systems
54:31 - How hard is 2D image understanding?
1:00:23 - Data
1:06:39 - Language
1:14:54 - Beautiful idea in statistical theory of learning
1:19:28 - Intelligence and heuristics
1:22:23 - Reasoning
1:25:11 - Role of philosophy in learning theory
1:31:40 - Music (speaking in Russian)
1:35:08 - Mortality

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Vladimir Vapnik, part 2, the second time we spoke on the podcast.

He's the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning.

He was born in the Soviet Union, worked at the Institute of Controlled Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University.

His work has been cited over 200,000 times.

The first time we spoke on the podcast was just over a year ago, 1 of the early episodes.

This time, we spoke after a lecture he gave titled Complete Statistical Theory of Learning as part of the MIT series of lectures on deep learning and AI that I organized.

I'll release the video of the lecture in the next few days.

This podcast and lecture are independent from each other so you don't need 1 to understand the other.

The lecture is quite technical and math heavy, So if you do watch both, I recommend listening to this podcast first, since the podcast is probably a bit more accessible.

This is the Artificial Intelligence Podcast.

If you enjoy it, Subscribe on YouTube, give it 5 stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

As usual, I'll do 1 or 2 minutes of ads now and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash App, the number 1 finance app in the App Store.

Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Brokerage services are provided by Cash App Investing, a subsidiary of Square, and member SIPC.

Since Cash App allows you to send and receive money digitally, peer-to-peer, and security in all digital transactions is very important.

Let me mention the PCI data security standard, PCI DSS level 1, that Cash App is compliant with.

I'm a big fan of standards for safety and security And PCI DSS is a good example of that, where a bunch of competitors got together and agreed that there needs to be a global standard around the security of transactions.

Now, we just need to do the same for autonomous vehicles and AI systems in general.

So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to FIRST, 1 of my favorite organizations that is helping to advance robotics and STEM education for young people around the world.

And now, here's my conversation with Vladimir Vapnik.

You and I talked about Alan Turing yesterday a little bit.

And that he, as the father of artificial intelligence, may have instilled in our field an ethic of engineering and not science, seeking more to build intelligence rather than to understand it.

What do you think is the difference between these 2 paths of engineering intelligence and the science of intelligence?

Engineering is imitation of human activity.

You have to make a device which behave as human behave, have all the functions of human.

It does not matter how you do it, but to understand what is intelligence about is quite different problem.

So I think, I believe that it's somehow related to predicate we talked yesterday about.

He call it units, which can explain human behavior, at least in Russian tales, you look at Russian tales and derive from that, and then people realize that it's more wise than in Russian tales.

It is in TV, in movie serials, and so on and so on.

So you're talking about Vladimir Prop, who in 1928 published a book, Morphology of the Folk Tale, describing 31 predicates that have this kind of sequential structure that a lot of the stories, narratives follow in Russian folklore and in other content.

I'd like to talk about predicates in a focused way, but let me, if you'll allow me to stay zoomed out on our friend Alan Turing.

And he inspired a generation with the imitation game.

Do you think, if we can linger on that a little bit longer, Do you think we can learn, do you think learning to imitate intelligence can get us closer to understanding intelligence?

So why do you think imitation is so far from understanding?

So your goal is to create something, something useful.

And you can see how much things was done and I believe that it will be done even more.

You have self-driving cars and also this business.

It's more or less philosophical category.

I believe in scheme which starts from Plato that there exists world of ideas.

I believe that intelligence, it is world of ideas.

And when you combine them with reality things, it creates, as in my case, invariants, which is very specific.

And that's, I believe, the combination of ideas in way to constructing invariant is intelligence.

If you know predicate, and hopefully then not too much predicate exist.

For example, 31 predicate for human behavior, it is not a lot.

you can even call them predicates, 31 predicates to describe stories, narratives.

So you think human behavior, how much of human behavior, how much of our world, our universe, all the things that matter in our existence can be summarized in predicates of the kind that Propp was working with?

I think that we have a lot of formal behavior.

Because even in this example, which I gave you yesterday, You saw that predicate can be, 1 predicate can construct many different invariants depending on your data.

They're applying to different data and they give different invariants.

But my guess, I hope, that's why challenge about digital recognition, how much you need.

I think we'll talk about computer vision and 2D images a little bit in your challenge.

That's exactly, that's exactly about, no, that hopes to be exactly about the spirit of intelligence in the simplest possible way.

Yeah, absolutely, you should start the simplest way.

Well, there's an open question whether starting at the MNIST digit recognition is a step towards intelligence or it's an entirely different thing?

I think that to beat records using, say, 100, 200 times less examples, You need intelligence.

So let's, because you use this term, and it would be nice, I'd like to ask simple, maybe even dumb questions.

In terms of terms and how you think about it, What is a predicate?

But I believe that predicate for 2D images, 1 of them is symmetry.

At the simplest level, we're not being profound currently.

A predicate is a statement of something that is true.

Do you think of predicates as somehow probabilistic in nature or is this binary, this is truly constraints of logical statements about the world?

In my definition, the simplest predicate is function.

Function, and you can use this function to make inner product, that is predicate.

What's the input and what's the output of the function?

Input is x, something which is input in reality.

Say if you consider digit recognition, it pixel space, input.

But It can be any function from pixel space.

And you choose, and I believe that there are several functions which is important for understanding of images.

It's not so simple construction, as I described with the derivative, with all the stuff.

But another, I believe, I don't know how many, is how well-structurized is picture.

It is formal definition, say something heavy on the left corner, not so heavy in the middle and so on.

You describe in general concept of what you assume.

Concepts, some kind of universal concepts.

Yeah, but I don't know how to formalize this.

There's a million ways we can talk about this.

But it's hard to put them, just like you're saying now, it's hard to put them into words.

When critics in music, trying to describe music, they use predicate.

And not too many predicate, but in different combination.

But they have some special words for describing music.

But maybe there are critics who understand essence of what this image is about.

Do you think there exists critics who can summarize the essence of images, human beings?

The fundamental question I'm asking is do you think there exists a small set of predicates that will summarize images.

It feels to our mind like it does, that the concept of what makes a 2 and a 3 and a 4.

It describes some construction which allow you to create invariance.

And invariance, sorry to stick on this, but terminology.

Invariance, it is property of your image.

Say, I can say, looking at my image, it is more or less symmetric, and I can give you a value of symmetry, say, level of symmetry using this function which I gave yesterday.

And you can describe that your image has these characteristics exactly in the way how musical critics describe music.

So, but this is invariant, applied to specific data, to specific music to something.

I strongly believe in this Plato ideas that there exists world of predicate and world of reality and predicate and reality is somehow connected and you have to know that.

So you draw a line from Plato to Hegel to Wigner to today.

There's a world of ideas and a world of things as you talk about, and there's a connection.

And presumably the world of ideas is very small, and the world of things is arbitrarily big.

But they're all, what Plato calls them, like it's a shadow, the real world is a shadow from the world of thought.

In reality, you can realize this projection using these invariants because it is projection for on specific examples, which creates specific features of specific objects.

So the essence of intelligence is while only being able to observe the world of things, try to come up with a world of ideas.

Intelligent musical critics knows all this work and have a feeling about what they're doing.

I feel like that's a contradiction, intelligent music critics.

But I think music is to be enjoyed in all its forms.

The notion of critic, like a food critic.

Does emotion, there's certain elements of the human psychology, of the human experience, which seem to almost contradict intelligence and reason.

Like emotion, like fear, like love, all of those things, are those not connected in any way to the space of ideas?

I just want to be concentrated on very simple story, on digit recognition.

So you don't think you have to love and fear death in order to recognize digits?

It involves a lot of stuff which I never consider.

And I know that for digit recognition to get the records from small number of observations, you need predicate.

But not special predicate for this problem, but universal predicate which understand world of images.

But on the first step, they understand, say, world of handwritten digits, or characters, or something simple.

So like you said, symmetry is an interesting 1.

No, that's what I think 1 of the predicate is related to symmetry, the level of symmetry.

So you think symmetry at the bottom is a universal notion and there's degrees of a single kind of symmetry or is there many kinds of symmetries?

There is a symmetry, anti-symmetry, say, letter S.

And it could be diagonal symmetry, vertical symmetry.

Yeah, then the upper part and lower part in different directions.

But that's just like 1 example of symmetry, right?

Right, but there is a degree of symmetry.

If you play all this derivative stuff to do tangent distance, but whatever I describe, You can have a degree of symmetry and that is what describing reason of image.

It is the same as you will describe this image.

Think about digit S, it has anti-symmetry.

Digit C, symmetric, more or less look for symmetry.

Do you think such concepts like symmetry, predicates like symmetry, is it a hierarchical set of concepts?

Or are these independent, distinct predicates that we want to discover, some set of?

And you can, this idea of symmetry, make very general, like degree of symmetry.

The degree of symmetry can be 0, no symmetry at all.

Or degree of symmetry, say, more or less symmetrical, but you have 1 of this description and symmetry can be different.

As I told, horizontal, vertical, diagonal, and anti-symmetry is also a concept of symmetry.

I mean, symmetry is a fascinating notion, but...

I would like to concentrate on all I would like to know predicate for digit recognition.

Yes, but Symmetry is not enough for digit recognition,

It is not necessarily for digit recognition.

It helps to create invariant which you can use when you will have examples for digital cognition.

You have regular problem of digital cognition, you have examples of the first class or second class.

Plus you know that there exists concept of symmetry.

And you apply when you're looking for decision rule, you will apply concept of symmetry of this level of symmetry which you estimate from.

So let's talk, everything comes from weak convergence.

What is convergence, what is weak convergence, what is strong convergence?

You're converging, you would like to have a function.

The function which, say indicator function which indicate your digit 5,

So Classification means you will say whether this is a 5 or not, or say which of the 10 digits it is.

I can consider property of these examples, say symmetry, and I can measure level of symmetry for every digit.

And then I can take average and I, from my training data, and I will consider only functions of conditional probability, which I'm looking for my decision rule, which applying to digits will give me the same average as I observe on training data.

So actually, this is different level of description of what you want.

You show this predicate, show general property of all digits which you have in mind.

If you have in mind digit 3, it gives you property of digit 3 and you select as admissible set of function, only function, which keeps this property.

So you're immediately looking for smaller subset of function.

That's what you mean by admissible functions.

Which is still a pretty large, for the number 3, it's a large.

It is pretty large, but if you have 1 predicate.

But according to, there is a strong and weak convergence.

Strong convergence is convergence in function.

You're looking for the function on 1 function and you're looking for another function.

And square difference from them should be small.

If you take difference in any points, make a square, make an integral, and it should be small.

Suppose you have some function, any function.

So I would say, I say that some function converge to this function.

If integral from square difference between them is small.

That's the definition of strong convergence.

2 functions, the integral, the difference, is small.

But you have different convergence in functionals.

You take any function, you take some function phi, and take inner product, this function is f function.

So you say that set of functions converge in inner product to this function if this value of inner product converge to value f 0.

But V convergence requires that it converge for any function of Hilbert space.

If it converge for any function of Hilbert space.

If it converge for any function of Hilbert space, then you will say that this is weak convergence.

You can think that when you take integral, that is property, integral property of function.

For example, if you will take sine or cosine, it is coefficient of, say, Fourier expansion.

So if it converge for all coefficients of Fourier expansion, so under some condition, it converge to function you're looking for.

Convergence not point-wise, but integral property of function.

So weak convergence means integral property of functions.

When I talking about predicate, I would like to formulate which integral properties I would like to have for convergence.

So, and if I will take 1 predicate, it's function which I measure property.

If I will use 1 predicate and say, I will consider only function which give me the same value as this predicate, I selecting set of functions from functions which is admissible in the sense that function which I looking for in this set of functions because I checking in training data, it gives the same.

Yes, it always has to be connected to the training data in terms of...

See all Lex Fridman transcripts on Youtube

Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71