- Jan LeCun, chief AI scientist at META and professor at NYU, believes that the concentration of power in proprietary AI systems is a bigger danger than security concerns.
- He advocates for open sourcing AI development and criticizes those who warn about the existential threat of AGI.
- LeCun argues that autoregressive LLMs, like GPT-4 and LLAMA 2 and 3, are not the path to superhuman intelligence because they lack essential characteristics of intelligent behavior.
- He believes that intelligence needs to be grounded in reality and that language alone does not contain enough wisdom and knowledge to construct a world model and understanding of the physical world.
- LeCun highlights the complexity of the real world and the limitations of AI in tasks that humans take for granted, such as driving a car or performing domestic chores.

- The piece discusses the limitations of current language models in understanding and reasoning about the world
- It explains the challenges of training language models to handle visual data and intuitive physics
- The focus is on the inability of language models to plan and predict actions based on a deep understanding of the world
- The piece also explores the limitations of generative models in predicting the evolution of the world based on actions
- The overall conclusion is that current models are not capable of achieving a complete understanding of the world and its dynamics

- The speaker and colleagues have been trying to develop a model to predict a distribution over all possible frames in a video for 10 years
- Predicting details in video is difficult due to the complexity and richness of information
- Attempts to train systems to reconstruct images from corrupted versions have failed
- Joint embedding predictive architecture (JEPA) is a new approach that shows promise in training systems to predict abstract representations of inputs
- Non-contrastive methods have been developed to prevent system collapse
- JEPA differs from generative architectures like LLMs in that it focuses on predicting abstract representations rather than all the pixels in the input
- JEPA is seen as a first step towards advanced machine intelligence

- The JEPA system is designed to extract easily predictable information from inputs
- It aims to eliminate non-predictable details, allowing for a higher level of abstraction in representation
- Self-supervised algorithms learn more from redundant data, making visual data more suitable for training than text
- Combining self-supervised training on visual and language data is possible, but currently being done with cheating methods
- Non-contrastive techniques like distillation and iJEPA are used to train the system to predict representations from corrupted inputs
- The VJPa system applies the same idea to video, masking a temporal tube and predicting the representation of the full video from the partially masked video.

- The system has been developed to learn representations of video, allowing for accurate classification of actions in the video.
- The representation also allows the system to determine if a video is physically possible or impossible based on object movements and appearances.
- The system could potentially be used for planning and predicting outcomes based on actions and world models.
- Hierarchical planning and detailed muscle control may require a different architecture or experience of the physical world.

- The speaker discusses how most plans people use are learned and not invented by themselves
- They explain the need for higher level abstractions in representation to link low level actions
- They discuss the success of autoregressive LLMs and self-supervised learning in language understanding and translation
- The speaker emphasizes the limitations of these systems and the importance of understanding what they cannot do
- They also talk about the success and challenges of self-supervised learning in image and video representation learning

- The speaker argues that generative models cannot learn good representations of the real world
- They discuss the limitations of language models in understanding common sense reasoning and knowledge
- The speaker explains how errors accumulate in large language models, leading to nonsensical answers
- They emphasize the vast amount of prompts that language models have not been trained on, leading to potential nonsensical outputs

Jan LeCun, chief AI scientist at META and professor at NYU, advocates for open sourcing AI development and criticizes warnings about the existential threat of AGI. He argues that autoregressive LLMs lack essential characteristics of intelligent behavior and that intelligence needs to be grounded in reality. The limitations of current language models in understanding and reasoning about the world are discussed, and a new approach called JEPA is seen as a first step towards advanced machine intelligence. The system aims to predict representations from corrupted inputs and has potential applications in planning and predicting outcomes based on actions and world models. The speaker also emphasizes the limitations of language models in understanding common sense reasoning and knowledge, and the accumulation of errors in large language models leading to nonsensical outputs."""

- The speaker discusses how certain prompts can cause language models to produce unexpected or nonsensical answers
- They highlight the limitations of autoregressive language models (LLMs) in reasoning and planning
- They propose a blueprint for future dialogue systems that involves planning the answer through optimization before turning it into text
- This approach would involve minimizing a cost function to measure the quality of the answer in abstract representation space
- The speaker emphasizes the need for systems to think about and plan their answers before producing text, similar to human reasoning and planning

- The speaker discusses the process of creating a system to measure whether an answer is good for a question
- They explain the optimization process and the use of gradient-based inference
- They also discuss training energy-based models and the use of contrastive and non-contrastive methods
- The speaker recommends abandoning generative models in favor of joint embedding architectures and energy-based models

- The speaker recommends abandoning probabilistic models in favor of energy-based models and contrastive methods in favor of regularized methods.
- They suggest minimizing the use of reinforcement learning (RL) and using it only when planning doesn't yield the predicted outcome, to adjust the world model or the critic.
- The speaker discusses reinforcement learning with human feedback and its transformational effect on large language models.
- They argue that AI systems will become increasingly dominant in mediating our interactions with the digital world, and emphasize the importance of open source and diversity in AI systems to avoid bias and control by a small number of companies.

- The speaker discusses the need for diverse AI systems and the potential for open source platforms to achieve this diversity
- Open source platforms allow for fine-tuning of AI systems for different languages, cultures, and values
- The speaker addresses the challenge of making money from open source platforms and suggests various business models
- The potential for bias in AI systems is discussed, with the speaker emphasizing the difficulty of achieving complete unbiasedness due to differing perspectives and the risk of sacrificing factual accuracy in the process

- The speaker discusses the challenges of creating unbiased AI systems, particularly in image generation
- Mark Andreessen highlights the difficulties big tech companies face in producing generative AI products
- The discussion emphasizes the need for diversity and open source models to address these challenges
- The limitations of language-based instructions in creating physical objects are also mentioned
- The future of the LLAMA AI system is discussed, including improvements and the potential for planning and reasoning capabilities
- The speaker acknowledges the uncertainty of when these advancements will be integrated into the product line

- The speaker discusses the development of world models based on training from video, with similar work at DeepMind and UC Berkeley.
- They express excitement about the potential for human-level intelligence with systems that can understand the world, remember, plan, and reason.
- The speaker emphasizes that achieving AGI will take at least a decade, and possibly much longer, due to the complexity of intelligence and the need for hardware innovation.
- They push back against AI doomers who imagine catastrophic scenarios of AI escaping or controlling humans, stating that these assumptions are mostly false.

- The speaker discusses the limitations of autoregressive language models and proposes a blueprint for future dialogue systems involving planning the answer through optimization before turning it into text
- They emphasize the need for systems to think about and plan their answers before producing text, similar to human reasoning and planning
- The speaker discusses the challenges and potential of open source platforms for achieving diversity in AI systems
- They address the potential for bias in AI systems and the difficulties faced by big tech companies in producing generative AI products
- The future of the LLAMA AI system, including improvements and potential for planning and reasoning capabilities, is discussed
- The speaker expresses excitement about the potential for human-level intelligence with systems that can understand the world, remember, plan, and reason, but acknowledges that achieving AGI will take at least a decade
- They push back against AI doomers who imagine catastrophic scenarios of AI escaping or controlling humans, stating that these assumptions are mostly false.

- The emergence of superintelligence will not be a sudden event, but a gradual process of creating increasingly intelligent systems with guardrails in place
- Intelligent AI systems will not have the desire to dominate or compete with humans, as the desire to dominate is specific to social species
- Designing guardrails for AI systems will be a progressive and iterative process, similar to the development of turbojet safety
- AI systems designed to control or manipulate people will be countered by AI assistants, leading to a race where no system is way ahead
- The development of AI will be continuous and disseminated quickly within industry, rather than a sudden breakthrough discovery like the Manhattan Project for nuclear weapons

- The tech industry lacks secrecy and information circulates quickly
- There is a fear of new technology and its impact on society
- Open source platforms are needed to avoid concentration of power in AI systems
- The diversity of AI systems is important for the preservation of democracy
- The future of robots as effective collaborators with humans is being reinvigorated by progress in humanoid robots by Tesla's Optimus team

- The emergence of humanoid robots has reinvigorated the robotics industry
- Companies like Boston Dynamics and Figure AI are leading the way
- The main issue in robotics is the lack of systems that can understand and plan actions in the world
- The hope is that advancements in AI will lead to progress in robotics
- The future of robotics in the home is promising, but still faces challenges in tasks like cooking and cleaning
- The potential for AI to make humanity smarter and more efficient is a source of hope for the future

- The speaker believes that AI will make humanity smarter and compares it to the invention of the printing press.
- He discusses the potential impact of AI on the job market and believes that it will not cause mass unemployment.
- He also discusses the importance of open source AI and the empowerment of humanity through technology.
- The conversation ends with a quote from Arthur C. Clarke about discovering the limits of the possible.

- The emergence of superintelligence will be a gradual process with guardrails in place
- AI systems will not desire to dominate humans
- Designing guardrails for AI will be a progressive process
- Open source platforms are needed to avoid concentration of power in AI systems
- Progress in humanoid robots is reinvigorating the robotics industry
- AI has the potential to make humanity smarter and more efficient
- AI will not cause mass unemployment
- Open source AI is important for empowering humanity through technology

Yann LeCun is the Chief AI Scientist at Meta, professor at NYU, Turing Award winner, and one of the most influential researchers in the history of AI. Please support this podcast by checking out our sponsors:
- HiddenLayer: https://hiddenlayer.com/lex
- LMNT: https://drinkLMNT.com/lex to get free sample pack
- Shopify: https://shopify.com/lex to get $1 per month trial
- AG1: https://drinkag1.com/lex to get 1 month supply of fish oil

TRANSCRIPT:
https://lexfridman.com/yann-lecun-3-transcript

EPISODE LINKS:
Yann's Twitter: https://twitter.com/ylecun
Yann's Facebook: https://facebook.com/yann.lecun
Meta AI: https://ai.meta.com/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:18 - Limits of LLMs
13:54 - Bilingualism and thinking
17:46 - Video prediction
25:07 - JEPA (Joint-Embedding Predictive Architecture)
28:15 - JEPA vs LLMs
37:31 - DINO and I-JEPA
38:51 - V-JEPA
44:22 - Hierarchical planning
50:40 - Autoregressive LLMs
1:06:06 - AI hallucination
1:11:30 - Reasoning in AI
1:29:02 - Reinforcement learning
1:34:10 - Woke AI
1:43:48 - Open source
1:47:26 - AI and ideology
1:49:58 - Marc Andreesen
1:57:56 - Llama 3
2:04:20 - AGI
2:08:48 - AI doomers
2:24:38 - Joscha Bach
2:28:51 - Humanoid robots
2:38:00 - Hope for the future

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else.

What works against this is people who think that for reasons of security, we should keep AI systems under lock and key, because it's too dangerous to put it in the hands of everybody.

That would lead to a very bad future in which all of our information diet is controlled by a small number of companies who proprietary systems.

I believe that people are fundamentally good and so if AI, especially open source AI, can make them smarter, it just empowers the goodness in humans.

And in fact, a lot of doomers are doomers because they don't think that people are fundamentally good.

The following is a conversation with Jan LeCun.

He is the chief AI scientist at META, professor at NYU, Turing Award winner, and 1 of the seminal figures in the history of artificial intelligence.

He and META AI have been big proponents of open sourcing AI development and have been walking the walk by open sourcing many of their biggest models, including LLAMA 2 and eventually LLAMA 3.

Also, Jan has been an outspoken critic of those people in the AI community who warn about the looming danger and existential threat of AGI.

He believes the AGI will be created 1 day, but it will be good.

It will not escape human control, nor will it dominate and kill all humans.

At this moment of rapid AI development, this happens to be somewhat a controversial position.

And so it's been fun seeing Yan get into a lot of intense and fascinating discussions online as we do in this very conversation.

To support it, please check out our sponsors in the description.

You've had some strong statements, technical statements, about the future of artificial intelligence recently, throughout your career actually, but recently as well.

You've said that autoregressive LLMs are not the way we're going to make progress towards superhuman intelligence.

These are the large language models like GPT-4, like Lama 2 and 3 soon and so on.

How do they work and why are they not going to take us all the way?

The first is that there is a number of characteristics of intelligent behavior.

For example, the capacity to understand the world, understand the physical world, The ability to remember and retrieve things.

The ability to reason and the ability to plan.

Those are 4 essential characteristics of intelligent systems or entities, humans, animals.

LLMs can do none of those, Or they can only do them in a very primitive way.

They don't really understand the physical world, they don't really have persistent memory, they can't really reason and they certainly can't plan.

And so, if you expect a system to become intelligent just without having the possibility of doing those things, you're making a mistake.

That is not to say that autoregressive LLMs are not useful.

That they're not interesting, that we can't build a whole ecosystem of applications around them.

Of course we can, but as a path towards human level intelligence, they're missing essential components.

And then there is another tidbit or fact that I think is very interesting.

Those headlines are trained on enormous amounts of text, basically the entirety of all publicly available text on the internet, right?

That's typically on the order of 10 to the 13 tokens.

Each token is typically 2 bytes, so that's 2 10 to the 13 bytes as training data.

It would take you or me 170,000 years to just read through this at 8 hours a day.

So it seems like an enormous amount of knowledge, right?

But then you realize it's really not that much data.

If you talk to developmental psychologists and they tell you a four-year-old has been awake for 16,000 hours in his or her life, And the amount of information that has reached the visual cortex of that child in 4 years is about 10 to the 15 bytes.

And you can compute this by estimating that the optical nerve carry about 20 megabytes per second, roughly.

And so 10 to the 15 bytes for a four-year-old versus 2 times 10 to the 13 bytes for 170,000 years worth of reading.

What that tells you is that through sensory input, we see a lot more information than we do through language.

And that despite our intuition, most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language.

Everything that we learn in the first few years of life, and certainly everything that animals learn, has nothing to do with language.

So it would be good to maybe push against some of the intuition behind what you're saying.

So it is true there's several orders of magnitude more data coming into the human mind much faster and the human mind is able to learn very quickly from that, filter the data very quickly.

You know, Somebody might argue your comparison between sensory data versus language, that language is already very compressed.

It already contains a lot more information than the bytes it takes to store them if you compare it to visual data.

So there's a lot of wisdom in language, there's words and the way we stitch them together, it already contains a lot of information.

So is it possible that language alone already has enough wisdom and knowledge in there to be able to, from that language, construct a world model and understanding of the world, an understanding of the physical world that you're saying LLM's lack?

So it's a big debate among philosophers and also cognitive scientists, like whether intelligence needs to be grounded in reality.

I'm clearly in the camp that yes, intelligence cannot appear without some grounding in some reality.

It doesn't need to be, you know, a physical reality.

It could be simulated, but the environment is just much richer than what you can express in language.

Language is a very approximate representation of our percepts and our mental models, right?

I mean, there's a lot of tasks that we accomplish where we manipulate a mental model of the situation at hand and that has nothing to do with language.

Everything that's physical, mechanical, whatever, when we build something, when we accomplish a task, a model task of grabbing something, etc.

We plan our action sequences, and we do this by essentially imagining the result of the outcome of a sequence of actions that we might imagine.

That requires mental models that don't have much to do with language.

And that's, I would argue, most of our knowledge is derived from that interaction with the physical world.

So a lot of my colleagues who are more interested in things like computer vision are really on that camp that AI needs to be embodied, essentially.

And then other people coming from the NLP side or maybe have some other motivation don't necessarily agree with that.

And the complexity of the world is hard to imagine.

It's hard to represent all the complexities that we take completely for granted in the real world that we don't even imagine require intelligence, right?

This is the old Moravec paradox from the pioneer of robotics, hence Moravec, who said, you know, how is it that with computers, it seems to be easy to do high-level complex tasks like playing chess and solving integrals and doing things like that, whereas the thing we take for granted that we do every day, like, I don't know, learning to drive a car or, you know, grabbing an object.

We have LLMs that can pass the bar exam, so they must be smart.

But then They can't learn to drive in 20 hours like any 17 year old.

They can't learn to clear out the dinner table and fill up the dishwasher like any 10 year old can learn in 1 shot.

Like, you know, what, What are we missing?

What type of learning or reasoning architecture or whatever are we missing that basically prevent us from having level 5 start-up in cars and domestic robots?

Can a large language model construct a world model that does know how to drive and does know how to fill a dishwasher, but just doesn't know how to deal with visual data at this time?

So it can operate in a space of concepts.

So yeah, that's what a lot of people are working on.

And the more complex answer is you can use all kinds of tricks to get an LLM to basically digest visual representations of images, or video, or audio for that matter.

And a classical way of doing this is you train a vision system in some way.

And we have a number of ways to train vision systems, either supervised, semi-supervised, self-supervised, all kinds of different ways.

That will turn any image into a high-level representation, basically a list of tokens that are really similar to the kind of tokens that typical LLM takes as an input.

And then you just feed that to the LLM in addition to the text.

And you just expect the LLM to kind of, you know, during training to kind of be able to use those representations to help make decisions.

I mean, it's been work along those lines for quite a long time.

I mean, there are LLMs that have some vision extension, but they're basically hacks in the sense that those things are not like trained end-to-end to handle, to really understand the world.

They're not trained with video, for example.

They don't really understand intuitive physics, at least not at the moment.

So you don't think there's something special to you about intuitive physics, about sort of common sense reasoning about the physical space, about physical reality.

That to you is a giant leap that LLMs are just not able to do.

We're not going to be able to do this with the type of LLMs that we are working with today.

And there's a number of reasons for this, but the main reason is the way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by blank markers, and you train a genetic neural net to predict the words that are missing.

And if you build this neural net in a particular way so that it can only look at words that are to the left of the 1 it's trying to predict.

Then what you have is a system that basically is trying to predict the next word in a text, right?

So then you can feed it a text, a prompt, and you can ask it to predict the next word.

It can never predict the next word exactly.

And so what it's going to do is produce a probability distribution over all the possible words in your dictionary.

In fact, it doesn't predict words, it predicts tokens that are kind of subword units.

And so It's easy to handle the uncertainty in the prediction there because there's only a finite number of possible words in the dictionary and you can just compute a distribution over them.

Then what the system does is that it picks a word from that distribution.

Of course, there's a higher chance of picking words that have a higher probability within that distribution.

So you sample from that distribution to actually produce a word.

And then you shift that word into the input.

And so that allows the system not to predict the second word, right?

And once you do this, you shift it into the input, et cetera.

That's called autoregressive prediction, which is why those LLMs should be called autoregressive LLMs, but we just call them LLMs.

And there is a difference between this kind of process and a process by which, before producing a word, when you talk, when you and I talk, you and I are bilingual, we think about what we're gonna say, and it's relatively independent of the language in which we're gonna say it.

When we talk about, I don't know, let's say a mathematical concept or something,

kind of thinking that we're doing and the answer that we're planning to produce is not linked to whether we're gonna say it in French or Russian or English.

Chomsky just rolled his eyes, but I understand.

So you're saying that there's a bigger abstraction that goes before language and maps onto language.

Right, it's certainly true for a lot of thinking that we do.

Is that obvious that we don't, like you're saying your thinking is same in French as it is in English?

Like if there's a probability distribution.

Well, it depends what kind of thinking, right?

If it's just, if it's like producing puns, I get much better in French than English about that.

Is there an abstract representation of puns?

Like, is your humor an abstract, like when you tweet, and your tweets are sometimes a little bit spicy, is there an abstract representation in your brain of a tweet before it maps onto English?

There is an abstract representation of imagining the reaction of a reader to that text.

Or you start with laughter and then figure out how to make that happen.

Or figure out a reaction you want to cause and then figure out how to say it, right?

But that's like really close to language.

But think about like a mathematical concept or imagining something you want to build out of wood or something like this, right?

The kind of thinking you're doing is absolutely nothing to do with language, really.

Like it's not like you have necessarily like an internal monologue in any particular language.

You're imagining mental models of the thing, right?

I mean, if I ask you to imagine what this water bottle will look like if I rotate it 90 degrees, that has nothing to do with language.

So clearly there is a more abstract level of representation in which we do most of our thinking and we plan what we're going to say.

If the output is uttered words, as opposed to an output being muscle actions, right?

They just produce 1 word after the other, instinctively if you want.

It's a bit like the subconscious actions where you're distracted, you're doing something, you're completely concentrated, and someone comes to you and asks you a question, and you kind of answer the question, you don't have time to think about the answer, but the answer is easy, so you don't need to pay attention.

It doesn't think about its answer, really.

It retrieves it because it's accumulated a lot of knowledge, so it can retrieve some things, but it's going to just spit out 1 token after the other without planning the answer.

But you're making it sound just 1 token after the other, 1 token at a time generation is bound to be simplistic.

But if the world model is sufficiently sophisticated, that 1 token at a time, the most likely thing it generates is a sequence of tokens is going to be a deeply profound thing.

assumes that those systems actually possess an internal world model.

So it really goes to the, I think the fundamental question is, can you build a really complete world model, not complete, but 1 that has a deep understanding of the world?

Yeah, so can you build this, first of all, by prediction?

Can you predict, can you build it by predicting words?

And the answer is most probably no, because language is very poor in terms of weak or low bandwidth, if you want.

There's just not enough information there.

So building world models means observing the world and understanding why the world is evolving the way it is.

And then the extra component of a world model is something that can predict how the world is going to evolve as a consequence of an action you might take, right?

So what model really is, here is my idea of the state of the world at time t, here is an action I might take.

What is the predicted state of the world at time t plus 1?

Now that state of the world does not need to represent everything about the world.

It just needs to represent enough that's relevant for this planning of the action, but not necessarily all the details.

You're not going to be able to do this with generative models.

So a generative model is trained on video, and we've tried to do this for 10 years.

You take a video, show a system a piece of video, and then ask it to predict the reminder of the video.

Basically, predict what's going to happen.

1 frame at a time, do the same thing as sort of the auto-aggressive LLMs do, but for video.

Either 1 frame at a time or a group of frames at a time.

But yeah, a large video model, if you want.

The idea of doing this has been floating around for a long time and at FAIR, some of my colleagues and I have been trying to do this for about 10 years.

And you can't really do the same trick as with LLMs because you know LLMs as I said, you can't predict exactly which word is going to follow a sequence of words, but you can predict a distribution over words.

Now, if you go to video, what you would have to do is predict a distribution over all possible frames in a video.

And we don't really know how to do that properly.

We do not know how to represent distributions over high-dimensional continuous spaces in ways that are useful.

And the reason we can do this is because the world is incredibly more complicated and richer in terms of information than text.

Video is high dimensional and continuous.

So if I take a video of this room, and the video is a camera panning around, there is no way I can predict everything that's gonna be in the room as I pan around.

The system cannot predict what's gonna be in the room as the camera is panning.

Maybe it's going to predict this is a room where there is a light and there is a wall and things like that.

It can't predict what the painting on the wall looks like or what the texture of the couch looks like.

So there's no way I can predict all those details.

So the way to handle this is, 1 way possibly to handle this, which we've been working for a long time, is to have a model that has what's called a latent variable.

And the latent variable is fed to a neural net and it's supposed to represent all the information about the world that you don't perceive yet and that you need to augment the system for the prediction to do a good job at predicting pixels, including the fine texture of the carpet on the couch and the painting on the wall.

That has been a complete failure, essentially.

We tried just straight neural nets, we tried GANs, we tried you know VAEs, all kinds of regularized autoencoders, we tried many things.

We also tried those kind of methods to learn good representations of images or video that could then be used as input to, for example, an image classification system.

All the systems that attempt to predict missing parts of an image or video from a corrupted version of it, basically.

So, take an image or a video, corrupt it or transform it in some way, and then try to reconstruct the complete video or image from the corrupted version, and then hope that internally the system will develop good representations of images that you can use for object recognition, segmentation, whatever it is.

That has been essentially a complete failure.

That's the principle that is used for LLMs, right?

Is it that it's very difficult to form a good representation of an image, like a good embedding of all the important information in the image?

Is it in terms of the consistency of image to image to image to image that forms the video?

Like what is the, if we do a highlight reel of all the ways you failed, what's that look like?

Okay, so the reason this doesn't work is, first of all, I have to tell you exactly what doesn't work, because there is something else that does work.

So the thing that does not work is training the system to learn representations of images by training it to reconstruct a good image from a corrupted version of it.

And we have a whole slew of techniques for this that are variants of denoising autoencoders.

Something called MAE, developed by some of my colleagues at FAIR, Maxed Autoencoder.

So it's basically like the LLMs or things like this, where you train the system by corrupting text, except you corrupt images, you remove patches from it, and you train a gigantic neural net to reconstruct.

And you know they're not good because if you now train the same architecture, but you train it supervised with label data, with textual descriptions of images, etc.

And the performance on recognition tasks is much better than if you do this self-supervised pre-training.

Okay, but the fact that you train the system to reconstruct images does not lead it to produce to learn good generic features of images.

What are these architectures that you're so excited about?

Okay, so now instead of training a system to encode the image and then training it to reconstruct the full image from a corrupted version.

See all Lex Fridman transcripts on Youtube

Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416