This is a talk by Josh Tenenbaum for course 6.S099: Artificial General Intelligence. This class is free and open to everyone. Our goal is to take an engineering approach to exploring possible paths toward building human-level intelligence for a better world.

INFO:
Course website: https://agi.mit.edu
Contact: agi@mit.edu
Playlist: https://goo.gl/tC9bHs

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

He's a professor here at MIT, leading the computational cognitive science group.

Among many other topics in cognition and intelligence, he is fascinated with the question of how human beings learn so much from so little.

And how these insights can lead to build AI systems that are much more efficient at learning from data.

I'm excited to be part of what looks like really quite a very impressive lineup, especially starting after today.

And it's, I think, quite a great opportunity to get to see perspectives on artificial intelligence from many of the leaders in industry and other entities working on this great quest.

So I'm going to talk to you about some of the work that we do in our group, but also I'm going to try to give a broader perspective reflective of a number of MIT faculty, especially those who are affiliated with the Center for Brains, Minds, and Machines.

So you can see up there on my affiliation.

Academically, I'm part of Brain and Cognitive Science or Course 9.

I'm also part of CSAIL, but I'm also part of the Center for Brains, Minds, and Machines, which is an NSF-funded center, Science and Technology Center, which really stands for the bridge between the science and the engineering of intelligence.

It literally straddles Vassar Street in that we have CSAIL and BCS members.

We also have partners at Harvard and other academic institutions.

And again, what we stand for, I wanna try to convey some of the specific things we're doing in the center and where we want to go with a vision that really is about jointly pursuing the science, the basic science of how intelligence arises in the human mind and brain, and also the engineering enterprise of how to build something increasingly like human intelligence in machines.

And we deeply believe that these 2 projects have something to do with each other and are best pursued jointly.

Now it's a really exciting time to be doing anything related to intelligence or certainly to AI for all the reasons that, you know, brought you all here.

We have all these ways in which AI is kind of finally here.

We finally live in the era of something like real practical AI.

Or for those who've been around for a while and have seen some of the Rises and falls, AI is back in a big way.

But from my perspective, and I think maybe this reflects why we distinguish what we might call AGI from AI, we don't really have any real AI, basically.

We have what I like to call AI technologies, which are systems that do things we used to think that only humans could do, and now we have machines that do them, often quite well, maybe even better than any human who's ever lived, like a machine that plays Go.

But none of these systems, I would say, are truly intelligent.

None of them have anything like common sense.

None of them have anything like the flexible general purpose intelligence that each of you might use to learn every 1 of these skills or tasks.

Each of these systems had to be built by large teams of engineers working together, often for a number of years, often at great cost to somebody who's willing to pay for it, and each of them just does 1 thing.

So AlphaGo might beat the world's best, but it can't drive to the match, or even tell you what Go is.

It can't even tell you that Go is a game, because it doesn't even know what a game is.

What is it that makes every 1 of your brains, maybe you can't beat the world's best in Go, but any 1 of you can get behind the wheel of a car.

I think of this because my daughter is gonna turn 16 tomorrow.

If she lived in California, she'd have a driver's license.

It's a little bit down the line for us here in Massachusetts.

But she didn't have to be specially engineered by billion dollar startups.

And she got really into chess recently, and now she's taught herself chess by playing just a handful of games basically.

And she can do any 1 of these activities and any 1 of us can.

I'll talk about the focus for us in our research and a lot of us again in CBMM is summarized here.

What drives the successes right now in AI, especially in industry, in all these AI technologies, is many, many things.

But where the progress has been made most recently and what's getting most of the attention is of course deep learning, but other kinds of machine learning technologies which essentially represent the maturation of a decades-long effort to solve the problem of pattern recognition.

That means taking data and finding patterns in the data that tell you something you care about, like how to label a class or how to predict some other signal.

And it's reasonable to say that deep learning as a technology has really made great strides on pattern recognition and maybe even has come in close to solving the problems of pattern recognition.

But intelligence is about many other things.

In particular, it's about modeling the world.

And think about all the activities that a human does to model the world that go beyond just say recognizing patterns in data, but actually trying to explain and understand what we see for instance.

Or to be able to imagine things that we've never seen, that never seen, maybe even very different from anything we've ever seen, but might want to see, and then to set those as goals, to make plans and solve problems needed to make those things real.

Or thinking about learning, again, some kinds of learning can be thought of as pattern recognition, if you're learning sufficient statistics or weights in a neural net that are used for those purposes.

But many activities of learning are about building out new models, right?

Either refining, reusing, improving old models, or actually building fundamentally new models as you've experienced more of the world.

And then think about sharing our models, communicating our models to others, modeling their models, learning from them.

All these activities of modeling, these are at the heart of human intelligence and it requires a much broader set of tools.

So I want to talk about the ways we're studying these activities of modeling the world and something in a pretty non-technical way about what are the kind of tools that allow us to capture these abilities.

Now I think it's I want to be very honest up front and to say this is just the beginning of a story right.

When you look at deep learning successes That itself is a story that goes back decades.

I'll say a little bit about that history in a minute.

But where we are now is just looking forward to a future when we might be able to capture these abilities at a really mature engineering scale.

And I would say we are far from being able to capture all the ways in which humans richly, flexibly, quickly build models of the world at the kind of scale that say Silicon Valley wants, either big tech companies like Google or Microsoft or IBM or Facebook or small startups, right?

And I think what I wanna talk to you about here is 1 route for trying to get there.

And this is the route that CBMM stands for.

The idea that by reverse engineering how intelligence works in the human mind and brain, that will give us a route to engineering these abilities in machines.

When we say reverse engineering, we're talking about science, but doing science like engineers.

This is our fundamental principle, that if we approach cognitive science and neuroscience like an engineer, where the output of our science isn't just a description of the brain or the mind in words, but in the same terms that an engineer would use to build an intelligent system, then that will be both the basis for a much more rigorous and deeply insightful science, but also direct translation of those insights into engineering applications.

Now, I said before I talk a little about history.

Again, if part of what brought you here is deep learning, and I know even if you've never heard of deep learning before, which I'm sure is unlikely, you saw some, you know, a good spectrum of that in the overview session last night.

It's really interesting and important to look back on the history of where did techniques for deep learning come from, or reinforcement learning.

Those are the 2 tools in the current machine learning arsenal that are getting the most attention.

Things like back propagation, or end-to-end stochastic gradient descent, or temporal difference learning, or Q-learning.

Maybe some of you have read these original papers.

Here's the original paper by Rumelhart, Hinton, and colleagues in which they introduced the back propagation algorithm for training multi-layer perceptrons, multi-layer neural networks.

Here's the original perceptron paper by Rosenblatt which introduced the one-layer version of that architecture and the basic perceptron learning algorithm.

Here's the first paper on sort of the temporal difference learning method for reinforcement learning from Sutton and Bartow.

Here's the original Boltzmann machine paper, also by Hinton and colleagues, which you know again is a, those of you who don't know that architecture, think of a kind of probabilistic, undirected, multilayer perceptron.

Or, for example, before there were LSTMs, if you know about current recurrent neural network architecture, earlier, much simpler versions of the same idea were proposed by Jeff Ellman and his Simple Recurrent Networks.

The reason I want to put up the original papers here is for you to look at both when they were published and where they were published.

So if you look at the dates, you'll see papers going back to the 80s, but even the 60s or even the 1950s.

Most of them were published in psychology journals.

So the journal Psychological Review, if you don't know it, is like the leading journal of theoretical psychology and mathematical psychology.

Okay, or Cognitive Science, the journal of the Cognitive Science Society.

Or the back prop paper was published in Nature, which is a general interest science journal, but by people who are mostly affiliated with the Institute for Cognitive Science in San Diego.

So what you see here is already a long history of scientists thinking like engineers.

These are people who are in psychology or cognitive science departments and publishing in those places, but by formalizing even very basic insights about how humans might learn or how brains might learn in the right kind of math, that led to, of course, progress on the science side, but it led to all the engineering that we see now.

We needed, of course, lots of innovations and advances in computing, hardware and software systems.

But this is where the basic math came from, And it came from doing science like an engineer.

So what I want to talk about in our vision is what does the future of this look like?

If we were to look 50 years into the future, what would we be looking back on now or over this time scale?

Well, here's a long-term research roadmap that reflects some of my ambitions and some of our center's goals and many others too.

We'd like to be able to address basic questions, fundamental questions of what it is to be and to think like a human.

Questions for example of consciousness or meaning in language or real learning.

Questions like even beyond the individual, like questions of culture or creativity.

And for each of these, there are basic scientific questions, right?

How do we become aware of the world and ourselves in it?

Starts with perception, but it really turns into awareness, awareness of yourself and of the world and what we might call consciousness, right?

Or how does a word start to have a meaning?

What really is a meaning and how does a child grasp it?

What do baby's brains actually start with?

Are they blank slates or do they start with some kind of cognitive structure?

And then what does real learning look like?

These are just some of the questions that we're interested in working on.

Or when we talk about culture we mean how do you learn all the things you didn't directly experience but that somehow you got from the accumulation of knowledge in society over many generations.

Or how do you ever think of new ideas or answers to new questions?

How do you think of the new questions themselves?

These are all key activities of human intelligence when we talk about how we model the world, where our models come from, what we do with our models.

And If we could get machines that could do these things, well, again, on the bottom row, think of all the actual real engineering payoffs.

Now, in our center, in both my own activities and a lot of what my group does these days and what a number of other colleagues in the Center for Brains, Minds, and Machines do, as well as very broadly people in BCS and CSAIL, 1 place where we work on the beginnings of these problems in the near term, this is the long term, like think 50 years, okay?

Maybe shorter, maybe longer, I don't know, but think well beyond 10 years.

But in the short term, 5 to 10 years, a lot of our focus is around visual intelligence.

Again, we can build on the successes of deep networks and a lot of pattern recognition and machine vision.

It's a good way to put these ideas into practice.

When we look at the actual brain, the visual system in the brain, in the human and other mammalian brains, for example, is really very clearly the best understood part of the brain.

And at a circuit level, it's the part of the brain that's most inspired current deep learning and neural network systems.

But even there, there's things which we still don't really understand like engineers.

So here's an example of a basic problem in visual intelligence that we and others in the center are trying to solve.

Look around you and you feel like there's a whole world around you and there is a whole world around you.

But what the actual sense data that's coming in through your eyes looks more like this photograph here where you can see there's a crowd scene, but it's mostly blurry except for a small region of high resolution in the center.

So that corresponds biologically to what part of the image is in your fovea.

That's the central region of cells in the retina, where you have really high resolution visual data.

The size of your fovea is roughly like if you hold out your thumb at arm's length, it's a little bit bigger than that, but not much bigger.

Most of the image, in terms of the actual information coming in in a bottom-up sense to your brain is really quite blurry.

But somehow by looking at just 1 part and then by saccading around or making a few eye movements, you get a few glimpses, each not much bigger than the size of your thumb at arm's length.

Somehow you stitch that information together into what feels like and really is a rich representation of the whole world around you.

And when I say around you, I mean literally around you.

Without turning around, nobody's allowed to turn around, ask yourself what's behind you.

Now the answer is going to be different for different people depending on where you're sitting, right?

For most of you, you might think, well, there's, I think there's a person pretty close behind me, right?

You know you're in a crowded auditorium, although you haven't seen that person, you know that they're there, right?

For people in the very back row, you know, there isn't a person behind you and you're conscious of being in the back row, right?

You might be conscious that there's a wall right behind you.

But now for the people who are in the room, not in the very back, think about how far behind you is the back, like where's the nearest wall behind you?

So we can, maybe we can call out, try a little demonstration.

So I don't know, I'm pointing to someone there.

Well, I could have been pointing at you, but I'm pointing to someone behind you.

All right, so how far is the nearest wall?

Nope, you can't turn around, you've blown your chance.

Without turning around, okay, so you were, okay, do you see I'm pointing to you there with the tie?

Okay, so without turning around, How far is the nearest wall behind you?

Without turning around, OK, so you were laughing.

OK, do you see I'm pointing to you there with the tie?

So without turning around, how far is the nearest wall behind you?

OK, well, I mean, that might be about right.

So yeah, since I didn't grow up in the metric system, I barely know, but yeah, I mean, the point is that like, Each of you is surely not exactly right, but you're certainly within an order of magnitude.

And I guess if we actually tried to measure, my guess is you're probably right within 50% or less, often maybe just 20% error.

I mean, Even if it's not, what did you say, 20 meters?

Even if it's not 20 meters, it's probably closer to 20 meters than it is to 5 or 10 meters, and it is to 50 meters.

You haven't turned around in a while, right?

But some part of your brain is tracking the whole world around you, right?

I mean, I don't know if it's 200 or 300, but it's not a thousand, I don't think so.

So you track these things, and you use them to plan your actions.

Okay, so again, think about how instantly, effortlessly, and very reliably, okay, your brain computes all these things, so the people and objects around you, and it's not just approximations.

Certainly when we're talking about what's behind you in space, there's a lot of imprecision.

But when it comes to reaching for things right in front of you, very precise shape and physical property estimates needed to pick up and manipulate objects.

And then when it comes to people, it's not just the existence of the people, but something about what's in their head.

You track whether someone's paying attention to you when you're talking to them, what they might want from you, what they might be thinking about you, what they might be thinking about other people.

So when we talk about visual intelligence, this is the whole stuff we're talking about.

And you can start to see how it turns into basic questions, I think, of what we might call the beginnings of consciousness, or at least our awareness of ourself in the world, and of ourselves as a self in the world, but also other aspects of higher level intelligence and cognition that are not just about perception, like symbols, to describe even to ourselves what's around us and where we are and what we can do with it.

You have to go beyond just what we would normally call the stuff of perception to say the thoughts in somebody's head and your own thoughts about that.

So what we've been doing in CBMM is trying to develop an architecture for visual intelligence.

And I'm not gonna go into any of the details of how this works, and this is just notional.

It's like just a sketch from a grant proposal of what we say we wanna do.

But it's based on a lot of scientific understanding of how the brain works.

There are different parts of the brain that correspond to these different modules in our architecture, as well as some kind of emerging engineering way to try to capture at the software and maybe even hardware levels how these modules might work.

So we talk about a sort of an early module of visual or perceptual stream, which is like bottom-up visual or other perceptual input, that's the kind of thing that is pretty close to what we currently have in, say, deep convolutional neural networks.

But then we talk about some kind of, the output of that isn't just pattern class labels, but what we call the cognitive core, core cognition.

So again an understanding of space and objects, their physics, other people, their minds, that's the real stuff of cognition that has to be the output of perception.

But somehow we have to have, this is what we call the brain OS in this picture.

We have to get there by stitching together the bottom up inputs from a glimpse here, a glimpse here, a little bit here and there, and accessing prior knowledge that comes from our memory systems to tell us how to stitch these things together into the really core cognitive representations of what's out there in the world.

And then if we're going to start to talk about it in language or to build plans on top of what we have seen and understood, that's where we talk about symbols coming into the picture.

The building blocks of language and plans and so on.

So now we might say, well, OK, this is an architecture that is brain inspired and cognitively inspired, and we're planning to turn into real engineering.

Again, I know this is a question you considered in the first lecture, maybe the engineering toolkit that's currently been making a lot of progress in, let's say, industry, maybe that's good enough.

Maybe, you know, let's take deep learning, but to stand for a broader set of modern pattern recognition-based and reinforcement learning-based tools and say, okay, well maybe that can scale up to this.

I'm happy in the question period if people want to debate this.

I think that it's not, when I say no, I don't mean like it can't happen or it won't happen, what I mean is the highest value, the highest expected route right now is to take this more science-based reverse engineering approach.

And that at least if you follow the current trajectory that industry incentives especially optimize for, It's not even really trying to take us to these things.

So think about, for example, a case study of visual intelligence that is in some ways, as pattern recognition, very much of a success.

It's again been mostly driven by industry.

It's something that if you read in the news or even play around with in certain publicly available data sets, feels like we've made great progress.

And this is an aspect of visual intelligence, which is sometimes called image captioning, or mapping images to text.

Basically, there's been a bunch of systems.

Google's AI can now capture images almost as well as humans.

A couple of years ago, I think, there were something like 8 papers all released onto archive around the same time from basically all the major industry computer vision groups, as well as a couple of academic partners, which all driven by basically the same data set produced by some Microsoft researchers and other collaborators, trained a combination of deep convolutional neural networks, state of the art visual pattern recognition, with recurrent neural networks, which had recently been developed for basically kinds of neural statistical language modeling, glued them together and produced a system which made very impressive results in a big training set and a held-out test set, where the goal was to take an image and write a sentence like a short sentence caption that that would seem like the kind of way a human would describe that image.

And these systems you know surpassed human level accuracy on the held out test set from a big training set.

But what you can see when you really dig into these things is there's often a lot of what I would call Dataset overfitting.

It's not overfitting to the training set, but it's overfitting to whatever are the particular characteristics of this dataset You know wherever it came from.

Certain set of photographs and certain ways of captioning them.

Okay, which even a big dataset It's not about quantity.

It's more about the quality, the nature of what people are doing.

So 1 way to test this system is to apply it to what seems like basically the same problem, but not within a certain curated or built data set.

And there's a convenient Twitter bot that lets you do this.

So there's something called the PickDeskBot, which takes 1 of the state-of-the-art industry AI captioning systems, a very good 1.

Again, this is not meant to, I'm not trying to critique these systems for what they're trying to do.

I'm just trying to point out what they don't really even try to do.

And just every couple of hours, it takes a random image from the web, captions it, and uploads the results to Twitter.

And a couple of months ago, when I prepared a first version of this talk, I just took a few days in the life of this Twitter bot.

I didn't take every single image, but I took, you know, most of the images in a way that was meant to be representative of the successes and the kinds of failures that such a system will make.

So we can go through this and it's a little bit entertaining and I think quite informative.

So here's just a somewhat random sample of a few days in the life of 1 of these caption bots.

So here we have a picture of a person holding, fortunately my screen is very small here and I can't read up there So maybe you'll have to tell me what it says.

So you have a person holding a cell phone.

Well, it's not a person holding a cell phone.

It's a person holding some kind of machine.

I don't even know what that is, but it's some kind of musical instrument, right?

A group of people on a field playing football.

I would call that an A result, maybe even A+.

Here's a group of people standing on top of a mountain.

There's a mountain, but as far as I can tell, there's no people.

See all Lex Fridman transcripts on Youtube

MIT AGI: Building machines that see, learn, and think like people (Josh Tenenbaum)