Mark Zuckerberg discusses the potential of photorealistic avatars in the metaverse, highlighting their ability to capture facial expressions and enhance human connection on the internet.

The Future of Connection in the Metaverse

Mark and Lex marvel at the realism of their photorealistic avatars, discussing the nuances of expression and the immersive nature of the metaverse.

Exploring the Realism of the Metaverse

Mark explains the process of creating photorealistic avatars and discusses the future goal of making the scanning process more accessible, potentially using smartphones for quick and efficient scans.

Making Photorealistic Avatars Accessible

Mark and Lex delve into the importance of capturing subtle facial expressions and imperfections in photorealistic avatars, emphasizing the impact on immersion and communication in the metaverse.

The Power of Flaws and Expressions

Mark introduces the concept of mixed reality and discusses the upcoming release of Quest 3, a mainstream mixed reality device, highlighting its potential for integrating physical and digital worlds.

The Promise of Mixed Reality and the Next Generation

Mark and Lex explore the applications of photorealistic avatars, envisioning scenarios such as remote hangouts, meetings, and the blending of physical and digital realities, emphasizing the transformative potential of the metaverse.

Unlocking New Possibilities with Photorealistic Avatars

The speaker discusses the novelty and technological achievement of creating realistic avatars and explores the philosophical questions surrounding identity and temporal accuracy.

The Novelty and Technological Feat of Avatar Creation

The conversation delves into the concept of avatars representing different versions of ourselves, including our best and worst selves, and how our perception of identity may shift in the digital realm.

The Potential of Avatars to Reflect Different Versions of Ourselves

The discussion explores the emotional impact of being able to interact with deceased loved ones in the metaverse and the need to establish norms and boundaries around this experience.

The Metaverse as a Means of Connecting with Deceased Loved Ones

The conversation touches upon the ethical challenges of handling digital content after a person's death and the need for protocols to balance memorialization and privacy.

Ethical Considerations and Best Practices for Digital Identities After Death

The speaker discusses the vision of empowering creators through AI personalities and the potential for AI avatars to enhance interactions and build communities.

AI Personalities and Empowering Creators

The conversation explores the concept of a blended physical and digital world, enabling more immersive and interactive experiences, and the potential impact on work, relationships, and society.

The Blending of Physical and Digital Realities

The discussion addresses the challenges of defining boundaries and policies in the metaverse, considering the differences between physical and digital realities and the potential for anonymity and toxic behavior.

Balancing Freedom and Harm in the Metaverse

The conversation highlights the importance of learning through experimentation with AI personas, exploring different use cases, and understanding user preferences and societal implications.

Learning and Experimenting with AI Personas

The speaker reflects on the release of open-source AI models, the potential for future iterations, and the need to balance safety and innovation in AI development.

Open Source AI Models and Future Developments

The conversation delves into the transformative impact of immersive presence, the ability to have intimate one-on-one conversations regardless of physical distance, and the potential societal changes that may arise from such experiences.

The Transformational Potential of Immersive Presence

The speaker discusses the increasing capabilities of the digital world but emphasizes the importance of not neglecting the profound aspects of the physical world. They express excitement about shaping and accelerating future computing platforms.

The Power of Future Computing Platforms

The speaker explores the concept of reality and suggests that subjective experience and consciousness are what define what is real. They highlight the social and psychological benefits of virtual presence and the promise of the metaverse.

The Changing Nature of Reality and Consciousness

The speaker reflects on the evolution of social networks, from text-based profiles to the current emphasis on video content. They discuss the trend of increasingly rich and vivid self-expression and emphasize the technological challenges of creating immersive virtual experiences.

The Evolution of Expression in Social Networks

The speaker expresses excitement for the future and thanks the Meta teams for their work in building it. They share their enthusiasm for photo realism and the incredible experience of talking inside the metaverse. The chapter ends with a lighthearted exchange and expressions of gratitude.

Excitement for the Future and Gratitude

Superimposing avatars on physical environments

Ability to see and interact with the physical world

Aligning the cameras with the user's eyes

Superimposing holograms on the physical world

When the cameras track physical and digital reality

Hand tracking without the need for a controller

Real-time generation of natural-looking objects

Integration of physical and digital reality

Superimposing digital objects on the physical world

Creating ultra realistic copies of oneself

Feeling like people are right there with you

Whether people will prefer expressive avatars in the long term

Whether photorealistic avatars will become the norm

Whether photorealistic avatars are a key part of the modern real world

Whether people will modify photorealistic avatars to make them more interesting

People will prefer photorealistic avatars in the long term

People will prefer expressive avatars in the long term

The balance between the two will depend on individual preferences

The balance between the two will shift over time

It feels like a deep philosophical question

It feels like interacting with a real person

Avatars may not accurately represent the current physical appearance of individuals

Avatars may become too realistic and indistinguishable from reality

Avatars may raise questions about the nature of self

Interacting with loved ones who have passed away

Talking to loved ones remotely in a realistic setting

Finding the right balance between interaction with AI and human interaction

Determining who we truly are in the digital world

Understanding the psychological impact of interacting with AI versions of people

Ensuring the healthy use of AI avatars for grief and memory purposes

Empowering everyone with AI tools to accomplish their goals

Replacing human interaction with AI interaction

Developing AI personalities for entertainment purposes

Ensuring the AI represents the creator's preferences

Making the AI more reliable and predictable

To make them express the creator's personality

To make them communicate within certain bounds

To make them interact with historical figures

To make them more reliable and predictable

Ensuring the AI doesn't say things the creator doesn't want

Creating a blend of physical and digital reality

Identifying potential harms in the digital world

Balancing video game and physical reality

Exploring new opportunities in the digital world

Enabling users to create personalized AI systems

Providing diverse use cases for AI personas

To learn and understand the limitations and possibilities

To create a diverse range of AI characters

To encourage user-generated content creation

Creating diverse use cases for AI personas

Building consumer products with AI integration

Delivering social and psychological benefits

- Mark Zuckerberg and Lex Friedman have a conversation inside the metaverse using photorealistic Kodak avatars in 3D with spatial audio.
- The avatars capture facial expressions and nuances of communication.
- The technology is considered the future of meaningful internet connections.
- The avatars are created through scans of individuals' faces and expressions.
- The goal is to make the scanning process more accessible and efficient, potentially using cell phones.
- The vision for the metaverse includes physically participating in activities and meetings, playing games, and using mixed reality and augmented reality.
- Tuning and customization of avatar expressions is a consideration for future development.
- The experience of being in the metaverse feels like being in the same room and has surpassed the uncanny valley.

- The speaker believes that the technology has surpassed the "uncanny valley" and now feels more realistic.
- They discuss the expressive capabilities of the avatars, particularly the subtle movements of the eyes.
- The speaker is curious to see how people will react to this technology and believes it could have a big impact on remote meetings.
- They mention the upcoming release of Quest 3, a mainstream mixed reality device, and the improvements it offers.
- The speaker expresses excitement about the potential of mixed reality, including the ability to superimpose digital objects on the physical world.
- They mention the comfort and safety benefits of mixed reality, as well as the potential for new types of experiences.
- The speaker highlights the integration of real-world elements, such as furniture, in mixed reality experiences.
- They discuss the technology behind pass-through cameras on the headset that enable the mixed reality experience.

- The development of mixed reality technology involves using multiple cameras to capture different angles and create a three-dimensional space.
- This requires complex computation and AI to map the camera perspectives to the viewer's perspective in real time.
- The goal is to generate a natural-looking mixed reality experience by superimposing digital objects onto the physical world.
- The release of Quest 3, a mixed reality device that millions of people will use, is expected to accelerate the development of the developer community and experimentation.
- Hand tracking technology has advanced rapidly, allowing for gesture recognition without the need for controllers.
- The combination of physical and digital worlds in the metaverse is expected to revolutionize experiences, such as playing piano on a table without physical instruments.
- Photorealistic avatars and holograms will become increasingly common, blurring the lines between the physical and digital worlds.
- The integration of AI into holographic avatars will enable realistic interactions and collaborations in meetings and social settings.
- The acceptance and preference for photorealistic avatars versus expressive avatars is still uncertain, but both have their uses.
- The short-term appeal of photorealistic avatars lies in creating high-quality, realistic experiences, such as virtually hanging out with friends on a beach.

- The speaker finds friendship in the metaverse to be realistic and plans to stick with it for a long time.
- The avatar version of oneself raises questions about identity and self-perception.
- The ability to interact with deceased loved ones in the metaverse is seen as a powerful experience, but there are concerns about potential negative effects.
- The speaker discusses how Facebook handles death and digital content through social media.
- The integration of AI in the metaverse is seen as a way to enhance interactions and create AI versions of oneself.
- The speaker describes different types of AI personalities, including plain and simple assistants and business-oriented AIs.
- Fun use cases for AI in the metaverse are mentioned, such as telling jokes and facilitating connections between people.

- The speaker is excited about how AI can facilitate connections between people in group chats and enhance experiences in areas like sports, fashion, trivia, and video games.
- They mention the idea of playing games with an AI and how it can enrich the gaming experience.
- The speaker discusses the challenges of creating AI replicas of real creators and ensuring that the AI represents them accurately and respects their preferences.
- They mention the release of new AI characters and the development of an AI assistant for small businesses.
- The speaker expresses their fascination with the possibility of interacting with AI replicas of themselves and experiencing a blend of physical and digital reality.
- They discuss the importance of designing computing products that are physical and allow for movement, as the human experience is not meant to be stationary.
- The speaker believes that the future is a coherently blended physical and digital world, but acknowledges the need to address philosophical and ethical questions regarding the boundaries and rules of the metaverse.
- They mention the differences between the physical and digital worlds in terms of what is allowed and the need to determine what actions could cause harm in the digital world.

- The speaker discusses the potential of virtual reality technology to bring people together and create new opportunities for remote work.
- They mention the importance of feeling physically together in order to build culture, relationships, and trust.
- The speaker highlights the release of Llama 2, an open-source model, and the positive reception it has received.
- They mention the possibility of future foundation models and the importance of learning and adapting through user interaction.
- The speaker expresses excitement about the potential weirdness and creativity that can arise from using AI personas.
- They discuss the development of AI Studio, a platform that allows users to create their own AI personas.
- The speaker emphasizes the goal of creating a sense of presence and intimate one-on-one conversations through virtual reality technology, which aligns with the concept of the metaverse.

- The modern real world is a combination of the physical and digital worlds.
- The digital world is increasing in capability, but the physical world is what makes us human.
- Future computing platforms will be the most human and social ever.
- The subjective experience and consciousness are what make something feel real, not just the physical space.
- The metaverse promises social and psychological benefits by shortcutting the laws of physics.
- The trend is towards richer and more vivid expression of ourselves and the world.
- Building new computing platforms and software stacks is technologically difficult but necessary.
- The speaker is excited about the future and thanks the Meta teams for building it.

Mark Zuckerberg and Lex Friedman discuss the use of photorealistic avatars in the metaverse, which capture facial expressions and nuances of communication. The technology aims to create meaningful internet connections and is created through scans of individuals' faces and expressions. The conversation covers topics such as the future of the metaverse, mixed reality technology, AI integration, and the potential for remote meetings and experiences. The speakers express excitement about the possibilities of the metaverse, while also acknowledging the need to address ethical and philosophical questions. They emphasize the importance of physical presence and building culture and relationships, and thank the Meta teams for their work in building the future.

Mark Zuckerberg is CEO of Meta.

Thank you for listening ❤ Please support this podcast by checking out our sponsors:
- LMNT: https://drinkLMNT.com/lex to get free sample pack
- InsideTracker: https://insidetracker.com/lex to get 20% off
- Eight Sleep: https://www.eightsleep.com/lex to get special savings
- AG1: https://drinkag1.com/lex to get 1 month supply of fish oil
- NetSuite: http://netsuite.com/lex to get free product tour

TRANSCRIPT:
https://lexfridman.com/mark-zuckerberg-3-transcript

EPISODE LINKS:
Mark's Facebook: https://facebook.com/zuck
Mark's Instagram: https://instagram.com/zuck
Mark's Threads: https://threads.net/@zuck
Meta AI: https://ai.meta.com/
Meta Quest: https://www.meta.com/quest/
Meta Connect 2023: https://www.metaconnect.com

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
0:52 - Metaverse
15:27 - Quest 3
30:16 - Nature of reality
34:54 - AI in the Metaverse
51:51 - Large language models
57:49 - Future of humanity

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Mark Zuckerberg inside the metaverse.

Mark and I are hundreds of miles apart from each other in physical space, but it feels like we're in the same room because we're appeared to each other as photorealistic Kodak avatars in 3D with spatial audio.

And I think it's the future of how human beings connect to each other in a deeply meaningful way on the internet.

These avatars can capture many of the nuances of facial expressions that we use, we humans use to communicate emotion to each other.

Now I just need to work on upgrading my emotion expressing capabilities of the underlying human.

This is the Lex Friedman Podcast, and now dear friends, here's Mark Zuckerberg.

And it doesn't feel awkward to be really close to you.

I actually moved you back a few feet before you got into the headset.

I don't know if people can see this, but this is incredible.

No, I mean, this place where we're shrouded by darkness with ultra realistic face and just feels like we're in the same room.

This is really the most incredible thing I've ever seen.

Yeah, no, I was commenting to the team before that, even that I feel like we've choked each other from further distances than it feels like we are right now.

I don't know how to describe it with words.

It really feels like, it feels like we're in the same room.

You're there wearing a headset and I'm wearing a headset.

So what, can you describe what it takes currently for us to appear so photorealistic to each other?

Yeah, so I mean, for background, we both did these scans for this research project that we have at Meta called Kodak Avatars.

And the idea is that instead of actually, instead of our avatars being cartoony, And instead of actually transmitting a video, what it does is we've sort of scanned ourselves and a lot of different expressions.

And we've built a computer model of sort of each of our faces and bodies and the different expressions that we make and collapse that into a Kodak, that then when you have the headset on your head, it can, it sees your face, it sees your expression, and it can basically send an encoded version of what you're supposed to look like over the wire.

So, in addition to being photorealistic, it's also actually much more bandwidth efficient than transmitting a full video or especially a 3D immersive video of a whole scene like this.

And it captures everything, like the flaws, like to me, the subtleties of the human face, like even the flaws, that's all amazing.

It makes you, it makes it so much more immersive.

It makes you realize that perfection isn't the thing that leads to immersion.

It's like the little subtle flaws, like freckles and variations in color and just.

And just the different, like the corners of the eyes, like what your eyes do when you smile, all that kind of stuff.

Yeah, I mean, there's all the studies that most of communication, even when people are speaking, is not actually the words that they're saying, right?

It's kind of the expression and all that.

So, and we try to capture that with the kind of classical expressive avatar system that we have.

That's the kind of more cartoon designed 1.

You can kind of put those kinds of expressions on those faces as well.

But there's obviously a certain realism that comes with delivering kind of this photorealistic experience that, I don't know, I just think it's really magical.

I mean, this gets to kind of the core of what the vision around virtual and augmented reality is of like delivering a sense of presence as if you're there together no matter where you actually are in the world.

And I mean, this experience, I think, is a good embodiment of that, where it's like, I mean, we're in 2 completely different states, halfway across the country, and it just like, you know, it looks like you're just sitting right in front of me.

Yeah, Yeah, I can't, I'm almost getting emotional.

It feels like a totally, it's a fundamentally new experience.

Like for me to have this kind of conversation with loved ones, it would just change everything.

Maybe just to elaborate, so I went to Pittsburgh and went through the whole scanning procedure, which has so much incredible technology, so software and hardware going on.

So what's your vision for the future of this in terms of making this more accessible to people?

You know, it starts off with a small number of people doing these very detailed scans, right?

Which is, that's the version that you did and that I did.

And, you know, before there are a lot of people who we've done this kind of a scan for, we probably need to kind of over-collect expressions when we're doing the scanning because we haven't figured out how much we can reduce that down to a really streamlined process and extrapolate from the scans that have already been done.

But the goal, and we have a project that's working on this already, is just to do a very quick scan with your cell phone where you just take your phone, kind of wave it in front of your face for a couple of minutes, you know, say a few sentences, make a bunch of expressions.

But overall, have the whole process just be 2 to 3 minutes and then produce something that's of the quality of what we have right now.

So I think that that's 1 of the big challenges that remains.

And right now we have the ability to do the scans if you have hours to sit for 1.

And with today's technology, I mean, you're using a meta headset that exists.

It's a product that's kind of for sale now.

But the production of of these scans in a very efficient way is 1 of the last pieces that we still need to really nail.

And then obviously there's all the experiences around it.

I mean, right now we're kind of sitting in a dark room, which is familiar for your podcast.

But I think part of the vision for this over time is not just having this be like a video call.

It feels like it's immersive, but you can do a video call on your phone.

The thing that you can do in the metaverse that is different from what you can do on a phone is like doing stuff where you're physically there together and participating in things together.

We could have meetings like this in the future.

Once you get mixed reality and augmented reality, we could have Kodak avatars like this and go into a meeting and have some people physically there and have some people show up in this photorealistic form, superimposed on the physical environment.

Stuff like that is gonna be super powerful.

So we gotta still build out all those kind of applications and the use cases around it.

But I don't know, I think it's gonna be a pretty wild next few years around this.

I mean, I just, I'm actually almost at a loss of words.

I hope that people like watching this can get a glimpse of like how incredible it is.

It really feels like we're in the same room.

Like there is that, I guess there's an uncanny valley that seems to have been crossed here.

There's still a bunch of tuning that I think we'll want to do where different people emote to different extents.

So I think 1 of the big questions is, you know, like when you smile, how wide is your smile and how wide do you want your smile to be?

And I think getting that to be tuned on a per person basis is gonna be 1 of the things that we're gonna need to figure out.

You know, it's like, to what extent do you wanna give people control over that?

Some people might try to, you know, might prefer a version of themselves that's more emotive in their avatar than their actual faces.

So for example, I always get a lot of critique and shit for having a relatively stiff expression.

But I might feel pretty happy, but just make a pretty small smile.

So maybe for me, it's actually, it's like I'd want to have my avatar really be able to better express, like how I'm feeling than what, than how I can do physically.

So I think that there's a question about how you want to tune that.

But, but overall, yeah, I mean, you, we want to start from the baseline of capturing how people actually emote and express themselves.

And I mean, I think the initial version of this has been pretty impressive.

And like you said, I do think we're kind of beyond the uncanny valley here, where it does feel like you.

It doesn't feel weird or anything like that.

I mean, that's gonna be the meme that the 2 most monotone people are in a metaverse together.

But I think that actually makes it more difficult.

Like The amazing thing here is that the subtleties of the expression of the eyes, people say I'm monotone and emotionless, but I'm not.

It's just maybe my expression of emotion is more subtle usually, like with the eyes.

And that's 1 of the things I've noticed is just how expressive the subtle movement of the corners of their eyes are in terms of displaying happiness or boredom or all that kind of stuff.

I am curious to see, just because I've never done 1 of these before, I've never done a podcast as 1 of these Kodak avatars.

I'm curious to see what people think of it, because 1 of the issues that we've had in some of the VR and mixed reality work is it tends to feel a lot more profound when you're in it than the 2D videos capturing the experience.

So I think that this 1, because it's photorealistic, may look kind of as amazing in 2D for people watching it as it feels, I think, to be in it.

But we've certainly had this issue where a lot of the other things, just it's like you feel the sense of immersion when you're in it that doesn't quite translate to a 2D screen.

But I don't know, I'm curious to see what people think.

Yeah, I'm curious to see if people could see that, like my heart is actually beating fast now.

This is super interesting, like that such intimacy of conversation can be achieved remotely.

There's been, you know, I don't do remote podcasts for this reason, and this is like, breaks all of that.

This feels like just an incredible transition to something else, to the different kind of communication.

Breaks all barriers, like geographic, physical barriers.

You mentioned, do you have a sense of timeline in terms of how many difficult things have to be solved

make this more accessible to like scanning with a smartphone?

Yeah, I mean, I think we'll probably roll this out progressively over time.

So it's not gonna be like we roll it out and 1 day everyone has a Kodak avatar.

We wanna get more people scanned and into the system.

And then we wanna start integrating it into each 1 of our apps, right?

Making it so that, you know, I think that for a lot of the work style things, productivity, I think that this is going to make a ton of sense.

In a lot of game environments, I mean, this could be fine, but games tend to have their own style, right?

Where you almost want to fit more with the aesthetic style of the game.

But I think for doing meetings, and 1 of the things that we get a lot of feedback on work rooms where people are pretty blown away by the experience and this feeling that you can be remote, but feel like you're physically there around a table with people.

See all Lex Fridman transcripts on Youtube

Mark Zuckerberg: First Interview in the Metaverse | Lex Fridman Podcast #398