Livestream powered by baiosphere – the bavarian ai network

How could machines learn as efficiently as humans and animals? How could machines learn to reason and plan? How could machines learn representations of percepts and action plans at multiple levels of abstraction, enabling them to reason, predict, and plan at multiple time horizons?

Prof. Dr. Yann LeCun, Chief AI Scientist for Meta AI Research and Silver Professor at the Courant Institute of Mathematical Sciences at New York University will propose a possible path towards autonomous intelligent agents, based on a new modular cognitive architecture and a somewhat new self-supervised training paradigm. The centerpiece of the proposed architecture is a configurable predictive world model that allows the agent to plan.  Behavior and learning are driven by a set of differentiable intrinsic cost functions.  The world model uses a new type of energy-based model architecture called H-JEPA (Hierarchical Joint Embedding Predictive Architecture). H-JEPA learns hierarchical abstract representations of the world that are simultaneously maximally informative and maximally predictable. 

The event is organized by Prof. Dr. Gitta Kutyniok, Bavarian AI-Chair for Mathematical Foundations of Artificial Intelligence at the Ludwig-Maximilians-Universität München, the professorship is funded by the Hightech Agenda Bayern. She is spokesperson of the CAS Research Focus "Next Generation AI" at the Center for Advanced Studies (CAS) at LMU and LMU-Director of the Konrad Zuse School of Excellence for Reliable AI (relAI). The following ecosystem-partners support the event: Center for Advanced Studies (CAS) at LMU, baiosphere – the bavarian ai network, BAdW – Bayerische Akademie der Wissenschaften, bidt – Bayerisches Forschungsinstitut für Digitale Transformation, MCML – Munich Center for Machine Learning, Konrad Zuse School of Excellence in Reliable AI (relAI).

You you you you you you you you you Good afternoon.

Good afternoon, dear ladies and gentlemen.

As president of the Bavarian Academy of Science, I would like to welcome you all to our talk focusing on current developments of advanced computing technologies.

We are very proud to host this event in the Royal Residence, the center of Munich, which houses the main office of the Bavarian Academy of Science.

Founded in 1759, the academy functions today as a community of scholars, a non-university research institution, and a communication interface between the Bavarian scientific community, society, and policy makers.

With the scholars, the Academy provides a powerful interdisciplinary network of very established scientists.

This network of excellence interacts very closely with all Bavarian research institutions and political decision makers and represents an important part of the science communication with the public.

Research activities range from the composer Richard Strauss to the study of climate change in the Alps, from baroque ceiling paintings to quantum physics.

The longer-term basic research spans from natural science and technology to humanities and social studies.

The Academy research project actively leverages the latest digital technologies.

The Leibniz Supercomputer Center, also part of our academy, serves as an important infrastructural support for digital activities at Bavarian universities.

Today, we will hear an important contribution to the rapidly growing discussions about artificial intelligence from Yann LeCun, who is, according to the Time magazine, 1 of the hundred worldwide leading AI pioneers.

He's professor of NYU and chief AI scientist at Meta, Northern California in New York, I just learned.

The title of his talk, From Machine Learning to Autonomous Intelligence, artificial intelligence, as you all know, is a key topic of our time, not only in research and industry, but also in the broader society.

This becomes evident as a large interest, and thank you for all coming to this meeting and sharing this experience.

We are very pleased as Bavarian Academy together with the AI Agency to be part of Biosphere.

Biosphere is the official network of all AI activities in Bavaria.

We partner in this important task to advance AI science in Bavaria in close cooperation with the Center for Advanced Studies at LMU, the Bavarian Research Institute for Digital Transformation, the Munich Center for Machine Learning, and the Konrad Zuse School of Excellence in Reliable AI.

The event today is part of this successful cooperation and we're looking very broadly forward to learn more about recent progress from advanced computing to autonomous intelligence.

I hereby hand over to Professor Thomas Seidel, member of the Bavarian AI Council, Chair of Database Systems and Data Mining, Director of Munich Center for Machine Learning.

To all of you, I wish a very insightful and informative afternoon.

Thank you, president Schweiger, for this nice introduction and a warm welcome also from my side to all of you, particularly to you, Jan Leckart, to be here today.

1 is I'm a member of the Bavarian AI Council.

We are 20 members appointed by the Bavarian state government as part of their high tech agenda from universities, research institutions, and also companies in the field to advise the government on AI strategies and actions, particularly in steering the Bavarian AI agency.

Klimke, which promotes the Bavarian AI network, which we call Biosphere.

So the Biosphere comprises a variety of strong AI players in Bavaria, also universities, research institutions from all over Bavaria, as well as a lot of the global companies we have here, including Google, Microsoft, IBM, but also the locally sitting global players, Siemens, BMWs, several insurances, Munich, Realianz, and so on, but also many regional small and medium enterprises and start-ups in the field of AI.

So the second head I am aware today is I'm 1 of the 4 directors of the Munich Center for Machine Learning, 1 of the co-directors, Daniel Gremmes, also here.

And this is a consortium of LMU and TUM funded by the BMBF, the Bavarian High Tech Agenda, with around 50 PIs in machine learning and IA3 junior research groups recently established around 200 doctoral students.

And our focus is on foundations of machine learning where we have several players including Gita Kutyniok from the mathematical part and statistics and computer sciences there, then perception, we are particularly strong in computer vision here in Munich and in natural language processing, the 2 big things where humans and computers interact and a lot of domain-specific things.

It's not on research, but also on transfer activities, fostering the collaboration network together with the biosphere, outreach to the general public, things like that.

If you're interested, we also have openings, of course, so this is that part.

So I'm sure we get fully inspired by your presentation and between your presentation is now, the next is Dr.

Ladies and gentlemen, may I also welcome you warmly on behalf of the Center for Advanced Studies at LMU and let me briefly say a few words about this institution and how it comes into play.

The Center for Advanced Studies at LMU was founded 15 years ago to provide a forum for precisely those research questions that cannot be tackled by only 1 discipline.

This was intended to take account of an increasingly diversifying but also specializing body of research that is becoming more and more disparate, not only in terms of content, but also in terms of space in Munich.

You can just think of the campus in Oberschleisheim, Overgleiching, to Neuried in the far south.

So the Center for Advanced Studies offers the place where these centrifugal forces can be bundled.

And for what topic does this task play a more important role than for artificial intelligence, which is spread across most faculties of LMU and other universities.

It was therefore a great pleasure for us when Gita Kutin-York, freshly appointed at our university, approached us and asked whether a format could be found that would network research on AI at LMU and are able to discuss overarching issues together.

It wasn't long before a so-called interdisciplinary CAS research focused entitled Next Generation AI was born, bringing together researchers from 14 faculties working on the topic of artificial intelligence.

Over the course of 2 years, a wide variety of lectures, workshops and conferences was organized and held and we were thrilled by the spirit that emerged of that group.

That's why we look forward to Professor Lequin's lecture today with both a smile and a tear, as it marks the formal conclusion of the research focus.

We are honored and grateful that Professor Lequin is going to give the lecture in this framework today.

Yes, also a warm welcome from my side to everyone here on site and also to everyone who participates via live stream.

It is wonderful to have so many people with us here this afternoon.

And a great thanks to all of our cooperation partners for actually making this lecture possible.

And here I would like to particularly thank Dr.

Annette Meyer and the Center for Advanced Studies of the Ludwig Maximilians University, München, Professor Dr.

Markus Schweiger and the Bavarian Academy of Sciences and Humanities, that this event can take place here in the academy in these really beautiful rooms.

Michael Klimke from the Biosphere, the Bavarian Eye Network, which made the live stream for the event possible, and also Dr.

Christoph Egle from the Bavarian Research Institute for Digital Transformation.

And now it's my great pleasure and honor to welcome Professor Jan LeCun.

Thank you very much for accepting our invitation and for coming to Munich for this lecture today.

Professor LeCun is Chief AI Scientist at Meta and the Silver Professor of Computer science at New York University.

He started his career with a PhD in computer science at Sorbonne University in Paris and then moved to the US where he became the head of the image processing research department at the famous Bell Labs, the AT&T Bell Laboratories.

Then after intermediate stations, he joined New York University in 2003, and he also became there the founding director of the NYU Center for Data Science in 2012.

His groundbreaking work includes among many others the development of convolutional neural networks, which are the state of the art for basically any problem in particular imaging sciences and computer vision.

And a particularly particular convolutional neural network architecture is also named by him, the so-called LeNet, which in a sense also promoted the impressive development of deep learning and AI as we experience it today.

His contributions are honored by numerous awards, many more than I could name here.

Let me just mention that he's a member of the US National Academy of Sciences and the National Academy of engineering.

He received various honorary degrees, for instance from EPFL, received the IEEE, Neural Network Pioneer Award and in 2019 the Turing Award which is typically referred to as the Nobel Prize of Computing.

And just a few weeks ago, we already heard it, the Time magazine, and conglations to that, has selected him as 1 of the 100 most influential people in AI worldwide.

And he also repeatedly contributes to the public debate about AI with also controversial proclamations, for example on the current craze around large language models.

How could machines learn as efficiently as humans and animals?

How could machines learn to reason and plan?

In his lecture, Professor Yan LeCun will now talk about a possible path towards autonomous intelligent agents based on a new modular cognitive architecture.

Thank you very much Thank you very much for the introduction.

And Thank you very much for inviting me and for coming here so numerous.

I did not call Comme des Charnettes Lunette.

This was my Lab director at Bell Labs who gave it that name.

I would never have done this but It's a good name Okay, it's a long title and a long subtitle objective driven AI this is what I call this I used to give this talk with the title Autonomous Machine Intelligence, and it scares people.

You know, they say, you mean machines that will be autonomous?

We're not going to be able to control them.

So I changed the name to Objective Driven AI because that's really more accurate.

And they're really kind of systems, it's an aspiration, it's not something that we've done, it's something that we should do.

And there are systems that could of course learn, remember, reason, plan, have common sense, be steerable, controllable, safe, and have the same kind of learning abilities and intelligence that we observe in animals and humans.

So let me start by a little bit of the state of the art, okay?

Because there's a lot of debates today about about AI and a lot of people are afraid of AI It's understandable whenever there is Technological revolution people are afraid of the unknown and AI is promising to be a big revolution, so people are afraid.

So let's first talk about the benefits before we talk about the risks.

And the benefits of AI are numerous already today, and there is even more coming in medicine, particularly in imaging, diagnosis, assistant, treatment protocol, drug design, things like this, very promising research.

In transportation, every car sold in the European Union today has to come with what's called an automatic emergency braking system, a system that will automatically stop the car if there is an obstacle in front of it and the driver does not react.

And that uses convolutional nets, by the way, In all the systems that I know in fact Germany was kind of a and Bavaria in particular was a pioneer in this.

Some of the early systems of this type were developed by Demler-Benz.

So driving assistance, autonomous driving, energy storage and management, things like that.

Environmental monitoring and protection, I'm going to say a few words about this.

Content information and management, This is probably the biggest use of AI today And of course in industry manufacturing Information systems quality control, etc.

A lot of applications are expected also in things like education for personalized education connecting people with each other with translation, telepresence, augmented reality, virtual reality.

And then enormous applications in science, biology and genomics, neuroscience, physics, particularly physics of disordered systems, complex systems, very large-scale simulations, chemistry, material science, very promising area for AI.

So this will really, and of course you know we've been talking a lot about creation, like creating art.

AI is essentially enabling a lot more people to be creative, people who don't necessarily have the technique, the underlying technique for producing art.

So AI will affect every aspect of human activity.

This is a video that was put together by my colleagues at Meta a couple years ago.

This is already sort of aging, if you want.

We chose the capability of computer vision system as of about 2 years ago.

So we can have systems that detect objects, and put frames around them, give them a name.

They can track human bodies and figure out in what pose they are, densely actually, so that's actually very useful for all kinds of applications.

And more interestingly, we can have systems that perform what's called semantic segmentation, which means isolating every object, marking them with kind of a mask, and then giving them a name for a category.

And this works for a very fine-grained category, For example, the species of a bird or plant or something of that type.

It's not like computer vision is completely solved.

In fact, if it was solved, we wouldn't have the large conference that takes place in Paris next week called ICCV.

But there's been a huge amount of advances there.

And a lot of advances in AI, but no advances in my slides for some reason.

Okay, my presentation refuses to advance.

Okay, I mentioned medicine, So certainly medical imaging is an area where a lot of work is going on.

This is some work by some of my colleagues at NYU that use 3D image recognition, not just 2D.

In some cases, this is actually 2D, but that use various techniques to detect, for example, tumors in mammograms or particular things in MRI and other types of images.

Some projects that took place a few years ago, which was a collaboration between the NYU radiology department and people at FAIR, Meta's Fundamental AI Research Lab, which essentially allows to accelerate the data collection for an MRI by a factor of 4 without degrading the image quality.

So instead of having to lie down in an MRI machine for 40 minutes or something, you can reduce this to 10 minutes and have the same quality of images.

And that's thanks to deep learning, essentially.

What's interesting today is that the favorite model that neuroscientists use to explain how the brain works, use artificial neural nets.

So the best explanation for what we observe using functional MRI data in the visual cortex of humans and animals are actually models that are essentially convolutional net models.

And that's kind of a closing the circle because the architecture of convolutional net is actually inspired by the architecture of the visual cortex, classic work in neuroscience from the 1960s.

The similar work also in language understanding, this is a recent paper in science by some of my colleagues from Meta actually, and they tried to figure out if the current large language models that everybody is playing with explain what we observe in the brain when people are asked to kind of remember or understand a story?

And the answer is, sort of, but not really.

It doesn't work nearly as well as convolutional net models for vision.

So what that means is that we're missing something that those models probably are not sufficient to explain what the brain does when we understand language.

I mentioned some applications in science, in particle physics in particular, high energy physics, to kind of make models of particle collisions and things of that type, image processing to discover exoplanets.

Some estimate says that about 12% of all physics papers today actually mention AI as a tool that he used, which is astonishing in just a relatively short time.

And in large-scale simulations, sort of universe-scale simulation that could sort of validate or invalidate certain theories about dark matter and things like this.

So very fascinating work and applications.

This is a very interesting project that was started by some of my colleagues at FAIR, by Larry Zitnick in particular, called the Open Catalyst Project.

And you can actually participate if you want.

And The idea of that project is that we could solve climate change if we had a good, efficient, scalable way of storing energy.

If we had a good way of storing energy, we could cover a small desert with solar panels and produce enough energy to power Europe or the entire planet.

The problem is you have to have a way of storing energy, which is why renewables today, despite the decisions of the German government to go all out on it.

You can't control whenever there is wind or sun.

And so you need another source of energy when there is no sun or no wind.

And for that, you need to be able to store energy and ship it wherever it's needed.

The best way to store energy is in the form of hydrogen or maybe methane.

And the best way to do this is by separating hydrogen from oxygen from water.

So take some water, put 2 electrodes, and then separate hydrogen from oxygen.

Problem with this is that it's either scalable if you use catalysts to do this, like platinum.

Sorry, it's either efficient if you use catalysts like platinum or it's scalable but not efficient.

So the big question is could we design compounds, new catalysts that would facilitate this reaction so that it's efficient, but does not require exotic materials like platinum so that it's scalable.

And the idea there is that you do a lot of chemical simulation, that's called DFT simulation, of water onto various compounds, and then you generate the data using simulation and also using experiments.

You put that data, you make it available, and then you ask people, can you train the machine learning system to figure out what the underlying rule is so that we can use it to design new materials that might have the same effect but be cheap.

OK, now What's important to realize is that the progress we've seen over the last few years in AI and machine learning are due to a set of techniques that we call self-supervised learning, which I'm sure many of you here in the room have heard about.

And Essentially, self-supervised learning would be a set of techniques that allows a system to be trained to represent the data, the world, without requiring labelled data, without requiring sort of manual human intervention to produce the data.

So perhaps the best success of this idea, which I've been advocating for a long time, is in the context of natural language understanding.

So the way all NLP systems are trained today, whether they are LLMs of the types that we play with or others, is the following.

You take a piece of text, a sequence of words, And you remove some of the words you you mask you mask them you blank them out you replace them by a blank marker Okay, you corrupt essentially the input and you put it at the input of a large neural net You train this very large neural net usually usually a transformer architecture to predict the words that are missing.

In the process of doing so, the system has to extract representations of the text that contain the semantics, the syntax, the grammar, everything.

They are what's called tokens, which are essentially sub-word units.

So in most languages, words have a prefix and a root and a suffix.

And you need to kind of separate those for those systems to work properly Otherwise your dictionary of words would be gigantic and then in German you have to do it because you can have words that are long Like this there are you know, but so So there is no choice.

You have to break up words into sub word units, you know in tokens.

And so you train the system, and this is the so-called BERT model, if you want, or idea.

You don't need any other data than the text and Once you've pre trained that system you can use the internal representation produced by the system as input to a subsequent Task a downstream task like let's say translation hate speech detection You know summarization whatever So that's the general idea of self-supervised learning, fill in the blanks.

Have a big piece of data, corrupt it in some way, and then train some big neural net to fill in the blanks, or recover the original data.

A particularly stunning example of this, which I'm not going to go into the technical details of, but I will later, is a system that came out of my colleagues in Paris, at Fair Paris, called Dino V2.

You can think of it as a foundation model for vision.

So it's a system that is trained to extract features from images such that those features can be used for anything you want, whether it's classification, fine-grained classification, depth estimation, semantic segmentation, instance retrieval.

So the same kind of application that I showed in the video, but basically with very little supervision.

And it basically, because it's pre-trained on enormous amounts of data, just training a very shallow head to solve any particular 1 of those problems actually beats the state of the art for depth estimation or classification or whatever.

You can actually play with it interactively.

And these are some examples of visualization of what the features that are extracted are.

It's kind of a, you know, colorful representation of the, like, different feature vectors are represented by different colors.

This is actually kind of each color is like a principal component if you know what that is.

So those are you know examples on sort of typical images.

And people have started to use this for all kinds of stuff, for biological image analysis, for astronomy, for environmental protection.

So that's the next example I'm gonna show you.

So this is a project by someone on the team, Camille Coupry, and a large collection of collaborators.

And what she did was use those Dino V2 features and trained a relatively small system on top of it to tell what the height of the trees are from a satellite image.

So we have lots of satellite images on the entire world at half meter resolution, you can get this from a satellite imaging companies, and for some areas there is LIDAR data which tells you how tall the trees are.

So you use that to train the system and then you can apply it to the entire world.

And what that tells you is how much how much carbon is captured by the trees If you know roughly what the height of the tree is, you know, roughly how much carbon is Captured in the tree.

That's super important to know like, you know, should we protect forests?

There's publications on this where, you know, everything is detailed and everything.

Another success of self-supervised learning of the type that I showed for natural language processing, where you remove some of the words, is in biology, proteomics particularly.

So a protein is a sequence of amino acids and We know hundreds of millions of them So you take a sequence of amino acids you remove some of the amino acids and you train some gigantic neural net to predict The amino acids that are missing the system kind of learns to represent sequences of amino acids that that constitute proteins and then you use that representation as input to a system that predicts the confirmation of that protein how it folds, whether it can stick to another protein, a particular location.

So there's a famous work by our colleagues at DeepMind, AlphaFold, but the this idea of using pre-trained transformers for protein was actually first published by my colleagues at FAIR.

They're actually no longer at FAIR, now they have left FAIR to create a startup around this idea.

Thousands of research groups around the world are using this kind of data.

There's actually an atlas of folded protein, continues 600 million proteins or something like that, with the structure that is predicted.

It's called the ESM Metagenomic Atlas, and at esmatlas.com.

A very big tool for biologists that really may change completely the way we do drug design and understand the mechanisms of life.

Another very impressive project here that required a lot of effort is a project called No Language Left Behind, again from FAIR, a collection of people from the various sites of FAIR.

And this is a system that can translate 200 languages in any direction.

And when you look at what those languages are, it's a lot of languages.

Most of them we never heard of in, you know, obscure corners of the world.

But it's important for people to be able to preserve that culture that, you know, they can speak their language and basically be understood using automatic translation.

So what's interesting about this is that there are 40,000 directions for translation, but the data only covers 2,400 of those pairs among the 40,000.

Despite that, because we train a giant transformer to represent language regardless of the language, the system takes advantage of the similarities between language families to actually kind of extract a multilingual, language-independent representation of language which allows the system to do translation in any direction, including for directions it's never been trained on.

It's a pretty small model by today's standard, only 54 billion parameters.

The same team now has another project called Seamless, which was announced a few weeks ago that can do speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation, as well as speech recognition, speech synthesis, et cetera.

Speech to speech is interesting because it can do translation for languages that are not written directly from speech to speech.

That system can handle a thousand languages, which is really impressive.

So applications of deep learning that are less visible, perhaps, is that deep learning or AI connects people to knowledge and they connect people to each other.

The biggest deployment of machine learning today is probably in social networks and online services, like search engines.

And if you take deep learning out of Google or Meta or Microsoft companies crumble, they literally are built around it So deep learning helps us deal with the information deluge for doing things like search and retrieval ranking question answering things like this, but And that requires machine to understand content, of course for translation, which is very useful for people who are not literate for example or people who are blind or Visually impaired so there's 3000000000 people in the world today who can't use technology because they basically can't read.

So here's the biggest use of AI today, filtering out illegal and dangerous content.

This is something that's very hard to do.

But to tell you, to give you an idea of how much progress AI has made, those idea of pre-training transformers and stuff like that, the proportion of hate speech that Facebook was able to take down automatically 5 years ago was about 20% to 25%.

It was using fairly simple machine learning techniques and LPM methods of the types that were common 5 years ago.

And then self-supervised pre-trained transformers happened.

And That number went to 95% last year And it's just progressing AI.

So a lot of people that we hear talk about AI, who generally don't know much about AI, actually, tell you about all the dangers of AI, that AI is going to destroy democracy because of disinformation and things like that.

What they don't understand is that AI is actually the solution to those problems.

And it's already the case that doing content moderation on social networks makes massive use of the latest advancements in AI.

And the people who try to corrupt that system are not nearly as sophisticated in terms of their AI.

See all Ludwig-Maximilians-Universität München transcripts on Youtube

From Machine Learning to Autonomous Intelligence – AI-Talk by Prof. Dr. Yann LeCun