The following is a conversation with Vladimir Vapnik.

He's the co-inventor of the support vector machine, support vector clustering, VC theory, and many foundational ideas in statistical learning.

He was born in the Soviet Union and worked at the Institute of Control Sciences in Moscow.

Then in the United States, he worked at AT&T, NEC Labs, Facebook Research, and now is a professor at Columbia University.

His work has been cited over 170,000 times.

He has some very interesting ideas about artificial intelligence and the nature of learning, especially on the limits of our current approaches and the open problems in the field.

This conversation is part of MIT course on Artificial General Intelligence and the Artificial Intelligence podcast.

If you enjoy it, please subscribe on YouTube or rate it on iTunes or your podcast provider of choice.

Or simply connect with me on Twitter or other social networks at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Vladimir Vapnik.

Einstein famously said that God doesn't play dice.

You have studied the world through the eyes of statistics.

So, let me ask you in terms of the nature of reality, fundamental nature of reality.

And because we don't know some factors, which could be important, it looks like God play dice.

In philosophy, they distinguish between 2 positions, positions of instrumentalism, where you're creating theory for prediction, and position of realism, where you're trying to understand what God did.

Can you describe instrumentalism and realism a little bit?

For example, if you have some mechanical laws, What is that?

Is it law which true always and everywhere?

Or it is law which allow you to predict position of moving element?

What you believe, you believe that it is God's law, that God created the world, which obey to this physical law, or it is just law for predictions.

If you believe that this is law of God, and it's always true everywhere, that means that you're a realist.

So you're trying to really understand God's thought.

So the way you see the world is as an instrumentalist?

You know, I'm working for some models, models of machine learning.

So in this model, we can see setting and we try to solve, resolve the setting, to solve the problem.

And you can do it in 2 different ways, from the point of view of instrumentalists, and that's what everybody does now.

Because they say the goal of machine learning is to find the rule for classification.

That is true, but it is an instrument for prediction.

But I can say the goal of machine learning is to learn about conditional probability.

So how God played use, and is he play, what is probability for 1, what is probability for another given situation.

But for understanding, I need conditional probability.

So let me just step back a little bit first to talk about, you mentioned, which I read last night, the parts of the 1960 paper by Eugene Wigner, Unreasonable Effectiveness of Mathematics and Natural Sciences.

Made me feel, to be honest, to confess my own work in the past few years on deep learning, heavily applied, made me feel that I was missing out on some of the beauty of nature in the way that math can uncover.

So let me just step away from the poetry of that for a second.

How do you see the role of math in your life?

And does math for you have limits of what it can describe?

Some people say that math is language which uses God.

So, I believe that this article about effectiveness, unreasonable effectiveness of math, is that if you're looking at mathematical structures, they know something about reality.

And the most scientists from natural science, they're looking on equation and trying to understand reality.

If you try very carefully look on all equations which define conditional probability, you can understand something about reality more than from your fantasy.

So math can reveal the simple underlying principles of reality, perhaps.

But then when you discover them and look at them, you see how beautiful they are.

And it is surprising why people did not see that before you're looking on the equation and derive it from equations.

For example, I talked yesterday about least square method.

And people have a lot of fantasy how to improve least square method.

But if you look going step by step by solving some equations, you suddenly will get some term which, after thinking, you understand that it describes position of observation point.

In least square method, we throw out a lot of information.

We don't look in composition of point of observations, we're looking only on residuals.

But when you understood that, that's very simple idea, but it's not too simple to understand.

And you can derive this just from equations.

Allen So some simple algebra, a few steps will take you to something surprising that once you think about,

And that is proof that human intuition not to reach and very primitive.

And it does not see very simple situations.

But what about human, as opposed to intuition, ingenuity, the moments of brilliance?

So are you so, do you have to be so hard on human intuition?

Are there moments of brilliance in human intuition that can leap ahead of math and then the math will catch

I think that the best human intuition, it is putting in axioms, and then it is technical way of-

Yeah, But if they correctly take axioms, but axiom Polish during generations of scientists, and this is integral wisdom.

So that's beautifully put, But if you maybe look at, when you think of Einstein and special relativity, what is the role of imagination coming first there in the moment of discovery of an idea.

So there's obviously a mix of math and out of the box imagination there.

Whatever I did, I exclude any imagination.

Because whatever I saw in machine learning that comes from imagination, like features, like deep learning, they are not relevant to the problem.

When you're looking very carefully from mathematical equations, you're deriving very simple theory, which goes far beyond theoretically than whatever people can imagine.

It is just interpretation, it is just fantasy, but it is not what you need.

You don't need any imagination to derive, say, the main principle of machine learning.

When you think about learning and intelligence, maybe thinking about the human brain and trying to describe mathematically the process of learning, that is something like what happens in the human brain.

Do you think we have the tools currently?

Do you think we will ever have the tools to try to describe that process of learning?

You, it is not description what's going on.

You know, when guy invent microscope, Levenhuk, for the first time.

Only he got this instrument and nobody...

But he wrote report in London Academy of Science.

In his report, when he's looking at the blood, he looks everywhere, on the water, on the blood, on the skin.

But he described blood like fight between queen and king.

So he saw blood cells, red cells, and he imagined that it is army fighting each other.

And it was his interpretation of situation.

And he sent this report in the Academy of Science.

They very carefully looked because they believe that he's right.

And I believe the same can happen with brain.

Because the most important part, you know I believe in human language.

For example, People say that it is better than a thousand days of diligent studies 1 day with a great teacher.

But if I will ask you what a teacher does, nobody knows.

But we know from history and now from math and machine learning that teacher can do a lot.

Lex Doppelganger So what, from a mathematical point of view, is a great teacher?

Lex Doppelganger That's an open question.

He can introduce some invariants, some predicate for creating invariants.

How he doing it, I don't know, because teacher knows reality and can describe from this reality, a predicate invariants.

But he knows that when you're using invariant, you can decrease number of observations 100 times.

So, but, maybe try to pull that apart a little bit.

I think you mentioned like a piano teacher saying to the student, play like a butterfly.

I played piano, I played guitar for a long time.

And yeah, that's, there's, maybe it's romantic, poetic, but it feels like there's a lot of truth in that statement.

Like there is a lot of instruction in that statement.

The language itself may not contain this information.

What is the information being exchanged there?

What is the representation of that information?

I believe that it is sort of predicate, but I don't know.

That is exactly what intelligence in machine learning should be.

Because the rest is just mathematical technique.

I think that what was discovered recently is that there is 2 mechanisms of learning.

1 called strong convergence mechanism and weak convergence mechanism.

In weak convergence mechanism, you can use predicate.

That's what play like butterfly and it will immediately affect your playing.

You know there is English proverb, great.

If it looks like a duck, swims like a duck, and quacks like a duck, then it is probably a duck.

So you saw many ducks that you're training data.

So you have description of how looks integral, looks ducks.

Yeah, the visual characteristics of a duck, yeah.

Yeah, but you want, and you have model for the cognition doubts.

So you would like the theoretical description from the model to coincide with the empirical description, which you saw on the teletext there.

So about looks like a duck, it is general.

You can't say it plays chess like a duck.

And it is completely legal predicate, but it is useless.

So half teacher can recognize not useless predicate.

So up to now, we don't use this predicate in existing machine learning.

But in this English proverb, they use only 3 predicate.

Looks like a duck, swims like a duck, and quacks like a duck.

So you can't deny the fact that swims like a duck and quacks like a duck has humor in it, has ambiguity.

But that means that you know ducks, you know different birds, you know animals, and you derive from this that it is relevant to say swim like a duck.

So underneath, in order for us to understand swims like a duck, it feels like we need to know millions of other little pieces of information.

Which we pick up along the way, you don't think so.

There doesn't need to be this knowledge base, in those statements carries some rich information that helps us understand the essence of DUC.

How far are we from integrating predicates?

You know that when you consider complete theory of machine learning, so what it does, you have a lot of functions, and then you're talking, It looks like a duck.

From training data, you recognize, like, expected duck should look.

Then you remove all functions which does not look like you think it should look from training date.

So you decrease amount of function from which you pick up 1.

Then you give a second predicate and again decrease the set of functions.

And after that you pick up the best function you can find.

Because your predicates aren't very good?

Yeah, that means the predicate's very good.

Because every predicate is invented to decrease admissible set of functions.

So you talk about admissible set of functions and you talk about good functions.

Vladimir Zaytsev So admissible set of function is set of function which has small capacity or small diversity, small VC dimension, which contain good function inside.

way, for people who don't know, VC, you're the V in the VC.

So how would you describe to a layperson what VC theory is?

you have a machine, so a machine capable to pick up 1 function from the admissible set of functions.

But set of admissible functions can be big, say contain all continuous functions and useless.

You don't have so many examples to pick up functions.

Small, We call it capacity, but maybe better call it diversity.

So not very different function in the set.

It's an infinite set of functions, but not very diverse.

When VC dimension is small, you need small amount of training data.

So the goal is to create admissible set of functions which have small VC dimension and contain good function.

Then you should, then you will be able to pick up the function using small amount of observations.

Dossier Is creating a set of admissible functions that has a small VC dimension.

And then you've figured out a clever way of picking up...

No, that is goal of learning, which I formulated yesterday.

Statistical learning theory does not involve in creating admissible set of function.

In classical learning theory, everywhere, 100% in textbook, the set of function, admissible set of function is given.

But this is science about nothing, because the most difficult problem to create admissible set of functions, given, say, a lot of functions, continuum set of functions, create admissible set of functions, that means that it has finite VC dimensions, small VC dimensions, and contain good functions.

What is the process of creating this admissible set of functions?

Yeah, You're looking of properties of training data and properties means that you have some function and you just count what is the average value of the function on training data.

You have a model and what is the expectation of this function on the model.

So the problem is about how to pick up functions.

But because when I'm talking about say, duck does not jump, so you don't ask question, jump like a duck, because it is trivially does not jumping and doesn't help you to recognize jump.

But you know something, which question to ask.

And you're asking if it swims like a germ, like a duck.

But looks like a duck at this general situation.

Looks like, say, guy who has this illness, is this disease, it is legal.

So there is a general type of predicate looks like and special type of predicate which related to this specific problem.

And that is intelligence part of all this business.

Incorporating the specialized predicates.

What do you think about deep learning as a neural network, these arbitrary architectures as helping accomplish some of the tasks you're thinking about?

What are the weaknesses and what are the possible strengths?

Everything, like deep learning, like features.

1 of the greatest books is Churchill's book about history of Second World War.

And he's starting this book describing that in old time, When war is over, so the great kings, they gathered together, almost all of them were relatives, and they discussed what should be done, how to create peace.

And when happened the First World War, the general public came in power.

And they were so greedy that robbed Germany.

And it was clear for everybody that it is not peace.

That peace will last only 20 years because they were not professionals.

There are mathematicians who are looking for the problem from a very deep point of view, mathematical point of view.

And there are computer scientists, who mostly do not know mathematics.

And they invented a lot of blah, blah, blah interpretations like deep learning.

If you like to say piecewise linear function, say that and do it in class of piecewise linear function.

And then they try to prove advantage of that through interpretations, which mostly wrong.

And when it's not enough, they appeal to brain, which they know nothing about that.

Nobody knows what's going on in the brain.

So I think that more reliable, look on math.

Try to understand that there is not only 1 way of convergence, which is a strong way of convergence.

There is a weak way of convergence, which requires predicate.

And if you will go through all this stuff, you will see that you don't need deep learning.

Even more, I would say 1 of the theories, which called the representative theory.

It says that optimal solution of mathematical problem which is described learning is on shadow network, not on deep learning.

Absolutely, so in the end, what you're saying is exactly right.

The question is, you have no value for throwing something on the table, playing with it, not math.

It's like in neural network where you said throwing something in the bucket or the biological example and looking at kings and queens with the cells or the microscope.

You don't see value in imagining the cells or kings and queens and using that as inspiration and imagination for where the math will eventually lead you.

You think that interpretation basically deceives you in a way that's not productive.

of learning and especially discussion about deep learning, it is discussion about interpretation, not about things, about what you can say about things.

That's right, but aren't you surprised by the beauty of it?

So, not mathematical beauty, but the fact that it works at all.

Or you were criticizing that very beauty, our human desire to interpret, to find our silly interpretations in these constructs.

How do you feel about the success of a system like AlphaGo at beating the game of Go?

Using neural networks to estimate the quality of a board and the quality of the position.

That is your interpretation, quality of the board.

The fact is a neural network system, doesn't matter, a learning system that we don't, I think, mathematically understand that well, beats the best human player, does something that was thought impossible.

That means that it's not a very difficult problem.

See all Lex Fridman transcripts on Youtube

Vladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5