Oriol Vinyals is the Research Director and Deep Learning Lead at DeepMind. Please support this podcast by checking out our sponsors:
- Shopify: https://shopify.com/lex to get 14-day free trial
- Weights & Biases: https://lexfridman.com/wnb
- Magic Spoon: https://magicspoon.com/lex and use code LEX to get $5 off
- Blinkist: https://blinkist.com/lex and use code LEX to get 25% off premium

EPISODE LINKS:
Oriol's Twitter: https://twitter.com/oriolvinyalsml
Oriol's publications: https://scholar.google.com/citations?user=NkzyCvUAAAAJ
DeepMind's Twitter: https://twitter.com/DeepMind
DeepMind's Instagram: https://instagram.com/deepmind
DeepMind's Website: https://deepmind.com
Papers:
1. Gato: https://deepmind.com/publications/a-generalist-agent
2. Flamingo: https://deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model
3. Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165
4. Emergent Abilities of Large Language Models: https://arxiv.org/abs/2206.07682
5. Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
0:34 - AI
15:31 - Weights
21:50 - Gato
56:38 - Meta learning
1:10:37 - Neural networks
1:33:02 - Emergence
1:39:47 - AI sentience
2:03:43 - AGI

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

At which point is the neural network a being versus a tool?

The following is a conversation with Aureal Vinales, his second time in the podcast.

Aureal is the Research Director and Deep Learning Lead at DeepMind and 1 of the most brilliant thinkers and researchers in the history of artificial intelligence.

To support it, please check out our sponsors in the description.

And now, dear friends, here's Ariel Vinales.

You are 1 of the most brilliant researchers in the history of AI, working across all kinds of modalities.

Probably the 1 common theme is it's always sequences of data.

So we're talking about languages, images, even biology and games as we talked about last time.

In your lifetime, will we be able to build an AI system that's able to replace me as the interviewer

in this conversation in terms of ability to ask questions that are compelling to somebody listening.

And then further question is, are we close, will we be able to build a system that replaces you as the interviewee in order to create a compelling conversation?

I think partly I would say, do we want that?

I really like when we start now with very powerful models, interacting with them and thinking of them more closer to us.

The question is, if you remove the human side of the conversation, is that an interesting, you know, is that an interesting artifact?

I've seen, for instance, last time we spoke, Lex, was we were talking about StarCraft.

And creating agents that play games involves self-play.

But ultimately, what people care about was how does this agent behave when the opposite side is a human.

So without a doubt, we will probably be more empowered by AI.

Maybe you can source some questions from an AI system.

I mean, that even today, I would say it's quite plausible that with your creativity, you might actually find very interesting questions that you can filter.

We call this cherry picking sometimes in the field of language.

And likewise, if I had now the tools on my side, I could say, look, you're asking this interesting question.

From this answer, I like the words chosen by this particular system that created a few words.

Completely replacing it feels not exactly exciting to me.

Although in my lifetime, I think way, I mean, given the trajectory, I think it's possible that perhaps there could be interesting, maybe self-play interviews, as you're suggesting, that would look or sound quite interesting and probably would educate or you could learn a topic through listening to 1 of these interviews, at a basic level at least.

So you said it doesn't seem exciting to you, but what if exciting is part of the objective function the thing is optimized over?

So there's probably a huge amount of data of humans, if you look correctly, of humans communicating online, and there's probably ways to measure the degree of, you know, as they talk about engagement.

So you can probably optimize the question that's most created an engaging conversation in the past.

So actually, if you strictly use the word exciting, there is probably a way to create a optimally exciting conversations that involve AI systems.

I think maybe looping back a bit to games and the game industry, when you design algorithms, you're thinking about winning as the objective, right?

But in fact, when we discussed this with Blizzard, the creators of StarCraft in this case.

I think what's exciting, fun, if you could measure that and optimize for that, that's probably why we play video games or why we interact or listen or look at cat videos or whatever on the internet.

So it's true that modeling reward beyond the obvious reward functions we're used to in reinforcement learning is definitely very exciting.

And again, there is some progress actually into a particular aspect of AI which is quite critical, which is, for instance, is a conversation or is the information truthful, right?

So you could start trying to evaluate these from excerpts from the internet, right, that has lots of information.

And then if you can learn a function automated ideally so you can also optimize it more easily, then you could actually have conversations that optimize for non-obvious things such as excitement.

And then I would say in that case, it would definitely be fun exercise and quite unique to have at least 1 side that is fully driven by an excitement reward function.

But obviously there would be still quite a lot of humanity in the system, both from who is building the system, of course, and also, ultimately, if we think of labeling for excitement, those labels must come from us, because it's just hard to have a computational measure of excitement.

As far as I understand, there's no such thing.

Well, as you mentioned truth also, I would actually venture to say that excitement is easier to label than truth, or is perhaps has lower consequences of failure.

But there is perhaps the humanness that you mentioned, that's perhaps part of a thing that could be labeled.

And that could mean An AI system that's doing dialogue, that's doing conversations, should be flawed, for example.

Like that's the thing you optimize for, which is have inherent contradictions by design, have flaws by design.

Maybe it also needs to have a strong sense of identity.

So it has a backstory it told itself that it sticks to.

It has memories, not in terms of how the system is designed, but it's able to tell stories about its past.

It's able to have mortality and fear of mortality in the following way, that it has an identity and if it says something stupid and gets canceled on Twitter, that's the end of that system.

So it's not like you get to rebrand yourself.

So maybe that, the high stakes nature of it, because like you can't say anything stupid now, or because you'd be canceled on Twitter.

And that there's, there's stakes to that.

And that I think part of the reason that makes it interesting.

And then you have a perspective, like you've built up over time that you stick with, and then people can disagree with you.

So holding that perspective strongly, holding sort of maybe a controversial, at least a strong opinion, all of those elements, it feels like they can be learned because it feels like there's a lot of data on the internet of people having an opinion.

And then combine that with a metric of excitement, you can start to create something that, as opposed to trying to optimize for sort of grammatical clarity and truthfulness, the factual consistency over many sentences, you optimize for the humanness.

And there's obviously data for humanness on the internet.

So I wonder if there's a future where that's part, I mean, I sometimes wonder that about myself.

I'm a huge fan of podcasts, and I listen to some podcasts, and I think, what is interesting about this?

The same way you watch other games, like you said, watch, play StarCraft or have Magnus Carlsen play chess.

So I'm not a chess player, so but it's still interesting to me.

That's the stakes of it, maybe the end of a domination of a series of wins.

I don't know, there's all those elements somehow connect to a compelling conversation.

And I wonder how hard is that to replace?

Because ultimately all of that connects to the initial proposition of how to test whether an AI is intelligent or not with the Turing test, which I guess my question comes from a place of the spirit of that test.

Yes, I actually recall I was just listening to our first podcast where we discussed Turing test.

So I would say from a neural network, you know, AI builder perspective, there's, you know, usually you try to map many of these interesting topics you discussed to benchmarks, and then also to actual architectures on how these systems are currently built, how they learn, what data they learn from, what are they learning, right?

We're talking about weights of a mathematical function.

And then looking at the current state of the game, maybe what do we need leaps forward to get to the ultimate stage of all these experiences, lifetime experience, fears, like words that currently, barely we're seeing progress just because what's happening today is you take all these human interactions, it's a large vast variety of human interactions online, And then you're distilling these sequences, right?

Going back to my passion, like sequences of words, letters, images, sound, there's more modalities here to be at play.

And then you're trying to just learn a function that will be happy, that maximizes the likelihood of seeing all these through a neural network.

Now I think there's a few places where the way currently we train these models would clearly like to be able to develop the kinds of capabilities you say.

1 is the lifetime of an agent or a model.

So you learn from this data offline, right?

So you're just passively observing and maximizing this, you know, it's almost like a mountains, like a landscape of mountains.

And then everywhere there's data that humans interacted in this way, you're trying to make that higher and then, you know, lower where there's no data.

And then these models generally don't then experience themselves.

And then we're putting them to then generate data when we interact with them.

The experience they actually experience when they could maybe be optimizing or further optimizing the weights.

So to be clear, and again, mapping to AlphaGo, AlphaStar, we train the model And when we deploy it to play against humans, or in this case, interact with humans, like language models, they don't even keep training, right?

They're not learning in the sense of the weights that you've learned from the data, they don't keep changing.

Now, there's something a bit more, feels magical, but it's understandable if you're into neural net, which is, well, they might not learn in the strict sense of the words, the way it's changing, maybe that's mapping to how neurons interconnect and how we learn over our lifetime.

But it's true that the context of the conversation that takes place with when you talk to these systems, it's held in their working memory, right?

It's almost like you start a computer, It has a hard drive that has a lot of information.

You have access to the internet, which has probably all the information, but there's also a working memory where these agents, as we call them, or start calling them, build upon.

I mean, right now we're talking to be concrete about 2000 words that we hold, and then beyond that, we start forgetting what we've seen.

So you can see that there's some short-term coherence already, right?

With when you said, I mean, it's a very interesting topic, having sort of a mapping, an agent to like have consistency.

Then, you know, if you say, oh, what's your name?

It could remember that, but then it might forget beyond 2000 words, which is not that long of context, if we think even of these podcast books are much longer.

So technically speaking, there's a limitation there, super exciting from people that work on deep learning to be working on, But I would say we lack maybe benchmarks and the technology to have this lifetime-like experience of memory that keeps building up.

However, the way it learns offline is clearly very powerful, right?

So you asked me 3 years ago, I would say, oh, we're very far.

I think we've seen the power of this imitation again on the internet scale that has enabled this to feel like at least the knowledge, the basic knowledge about the world now is incorporated into the weights.

And in fact, as I said, we don't even train them when we're talking to them, other than their working memory, of course, is affected.

So That's the dynamic part, but they don't learn in the same way that you and I have learned, right?

When, from basically when we were born and probably before.

So lots of fascinating, interesting questions you asked there.

I think the 1 I mentioned is this idea of memory and experience versus just kind of observe the world and learn its knowledge, which I think for that, I would argue, lots of recent advancements that make me very excited about the field.

And then the second maybe issue that I see is all these models, we train them from scratch.

That's something I would have complained 3 years ago or 6 years ago or 10 years ago.

And it feels if we take inspiration from how we got here, how the universe evolved us, and we keep evolving, it feels that is a missing piece, that we should not be training models from scratch every few months, that there should be some sort of way in which we can grow models much like as a species and many other elements in the universe is building from the previous sort of iterations.

And that from a just purely neural network perspective, even though we would like to make it work.

It's proven very hard to not throw away the previous weights, this landscape we learned from the data, and refresh it with a brand new set of weights, given maybe a recent snapshot of this dataset we trained on, etc.

So that feels like something is missing, fundamentally.

We might find it, but it's not very clear how it will look like.

There's many ideas and it's super exciting as well.

When you're approaching a new problem in machine learning, you're going to come up with an architecture that has a bunch of weights, and then you initialize them somehow, which in most cases is some version of random.

So that's what you mean by started from scratch and it seems like it's a waste every time you solve the game of Go and chess, StarCraft, protein folding, like surely there's some way to reuse the weights as we grow this giant database of neural networks that have solved some of the toughest problems in the world.

How to learn extract what's generalizable, or at least has a chance to be, and throw away the other stuff.

And maybe the neural network itself should be able to tell you that.

Like what, yeah, how do you, what ideas do you have for better initialization of weights?

Maybe stepping back, if we look at the field of machine learning, but especially deep learning, right?

At the core of deep learning, there's this beautiful idea that is a single algorithm can solve any task, right?

So it's been proven over and over with more increasing set of benchmarks and things that were thought impossible that are being cracked by this basic principle.

That is, you take a neural network of uninitialized ways, so like a blank computational brain, then you give it, in the case of supervised learning, a lot, ideally, of examples of, hey, here is what the input looks like and the desired output should look like this.

I mean, image classification is very clear example, images to maybe 1 of a thousand categories, that's what ImageNet is like, but many, many, if not all problems can be mapped this way.

And then there's a generic recipe, right, that you can use.

And this recipe with very little change, and I think that's the core of deep learning research, right?

That what is the recipe that is universal, that for any new given task, I'll be able to use without thinking, without having to work very hard on the problem at stake.

We have not found this recipe, but I think the field is excited to find less tweaks or tricks that people find when they work on important problems specific to those, and more of a general algorithm, right?

So at an algorithmic level, I would say we have something general already, which is this formula of training a very powerful model, a neural network on a lot of data.

And in many cases, you need some specificity to the actual problem you're solving.

Protein folding being such an important problem has some basic recipe that is learned from before, right?

Like transformer models, graph neural networks, ideas coming from NLP, like something called BERT that is a kind of loss that you can emplace to help the model.

Knowledge distillation is another technique, right?

We still had to find some particular things that were specific to alpha fold, right?

That's very important because protein folding is such a high value problem that as humans, we should solve it no matter if we need to be a bit specific.

And it's possible that some of these learnings will apply then to the next iteration of this recipe that deep learners are about.

But it is true that so far, the recipe is what's common, but the weights you generally throw away, which feels very sad.

Although maybe in the last, especially in the last 2, 3 years, and when we last spoke, I mentioned this area of meta-learning, which is the idea of learning to learn.

That idea and some progress has been had, starting, I would say, mostly from GPT-3 on the language domain only, in which you could conceive a model that is trained once, and then this model is not narrow in that it only knows how to translate a pair of languages or only knows how to assign sentiment to a sentence, this actually, you could teach it by a prompting, it's called.

And this prompting is essentially just showing it a few more examples, almost like you do show examples, input-output examples, algorithmically speaking, to the process of creating this model.

But now you're doing it through language, which is very natural way for us to learn from 1 another.

I tell you, hey, you should do this new task.

Maybe you ask me some questions, and now you know the task, right?

You didn't need to retrain it from scratch.

And we've seen these magical moments almost in this way to do few-shot prompting through language on language-only domain.

And then in the last 2 years, we've seen this expanded to beyond language, adding vision, adding actions and games, lots of progress to be had.

But this is maybe, if you ask me like about how are we going to crack this problem?

This is perhaps 1 way in which you have a single model.

The problem of this model is it's hard to grow in weights or capacity, but the model is certainly so powerful that you can teach it some tasks, right?

In this way that I teach you, I could teach you a new task now, if we were all at a text-based task or a classification, a vision style task.

But it still feels like more breakthroughs should be had, but it's a great beginning, right?

We have an idea that this maybe is the way we want to benchmark progress towards AGI.

And I think in my view, that's critical to always have a way to benchmark the community sort of converging to this overall, which is good to see.

And then This is actually what excites me in terms of also next steps for deep learning is how to make these models more powerful, how do you train them, how to grow them if they must grow, should they change their weights as you teach it, task or not.

There's some interesting questions, many to be answered.

Yeah, you've opened the door to a bunch of questions I wanna ask, but let's first return to your tweet and read it like a Shakespeare.

You wrote, Gato is not the end, It's the beginning.

And then you wrote meow and then an emoji of a cat.

First, can you explain the meow and the cat emoji?

And second, can you explain what Gato is and how

I mean, Thanks for reminding me that we're all exposing on Twitter.

1 of the greatest AI researchers of all time, meow and cat emoji.

Can you imagine touring and tweeting meow and cat?

So yeah, the tweet is important, actually.

Gato's not the end, Gato's the beginning.

I mean, I probably was just explaining a bit where the field is going, But let me tell you about Gato.

So first, the name Gato comes from maybe a sequence of releases that DeepMind had that named, like used animal names to name some of their models that are based on this idea of large sequence models.

Initially, they're only language, but we are expanding to other modalities.

So we had Gopher, Chinchilla, these were language only, And then more recently we released Flamingo, which adds vision to the equation.

And then Gato, which adds vision and then also actions in the mix, right?

As we discuss actually actions, especially discrete actions like up, down, left, right.

I just told you the actions, but they're words.

So you can kind of see how actions naturally map to sequence modeling of words, which these models are very powerful.

So Gato was named after, I believe, I can only from memory, right?

These, you know, these things always happen with an amazing team of researchers behind.

So before the release, we had a discussion about which animal would we pick, right?

And I think because of the word general agent, right?

And this is a property quite unique to gato.

We kind of were playing with the GA words and then, you know, gato is- Kind

And gato is obviously a Spanish version of cat.

I had nothing to do with it, although I'm from Spain.

Oh, how do you, wait, sorry, how do you say cat in Spanish?

I think you say it the same way, but you write it as M-I-A-U.

All right, so then how does the thing work?

So you said general is, so you said language, vision, and action.

How does this, Can you explain what kind of neural networks are involved?

And maybe what to you are some beautiful ideas within this system?

Yeah, so maybe the basics of GATO are not that dissimilar from many, many work that comes.

So here is where the sort of the recipe, I mean, hasn't changed too much.

There is a transformer model that's the kind of recurrent neural network that essentially takes a sequence of modalities, observations that could be words, could be vision, or could be actions.

And then its own objective that you train it to do when you train it is to predict what the next anything is.

And anything means what's the next action.

If this sequence that I'm showing you to train is a sequence of actions and observations, then you're predicting what's the next action and the next observation, right?

So you think of these really as a sequence of bytes, right?

So take any sequence of words, a sequence of interleaved words and images, a sequence of maybe observations that are images and moves in Atari up, down, left, right.

And these, you just think of them as bytes and you're modeling what's the next byte gonna be like.

And you might interpret that as an action and then play it in a game, or you could interpret it as a word and then write it down if you're chatting with the system and so on.

So GATO basically can be thought as inputs, images, text, video, actions.

It also actually inputs some sort of proprioception sensors from robotics because robotics is 1 of the tasks that it's been trained to do.

And then at the output, similarly, it outputs words, actions.

That's just by design, we decided not to go that way for now.

That's also in part why it's the beginning, because there's more to do clearly.

But that's kind of what the gato is, is this brain that essentially you give it any sequence of these observations and modalities and it outputs the next step.

And then off you go, you feed the next step into and predict the next 1 and so on.

Now, it is more than a language model because even though you can chat with Gato, like you can chat with Chinchilla or Flamingo, it also is an agent, right?

So that's why we call it A of Gato, like the word, the letter A, and also it's general.

It's not an agent that's been trained to be good at only StarCraft or only Atari or only Go, it's been trained on a vast variety of data sets.

What makes it an agent, if I may interrupt,

Yes, so when we call it, I mean, it's a good question, right?

See all Lex Fridman transcripts on Youtube

Oriol Vinyals: Deep Learning and Artificial General Intelligence | Lex Fridman Podcast #306