Marc Andreessen is the co-creator of Mosaic, co-founder of Netscape, and co-founder of the venture capital firm Andreessen Horowitz. Please support this podcast by checking out our sponsors:

- InsideTracker: https://insidetracker.com/lex to get 20% off

- ExpressVPN: https://expressvpn.com/lexpod to get 3 months free

- AG1: https://drinkag1.com/lex to get 1 year of Vitamin D and 5 free travel packs



Transcript: https://lexfridman.com/marc-andreessen-transcript



EPISODE LINKS:

Marc's Twitter: https://twitter.com/pmarca

Marc's Substack: https://pmarca.substack.com

Marc's YouTube channel: https://youtube.com/@a16z

Andreessen Horowitz: https://a16z.com

Why AI will save the world (essay): https://a16z.com/2023/06/06/ai-will-save-the-world

Books mentioned:

1. When Reason Goes on Holiday (book): https://amzn.to/3p80b1K

2. Superintelligence (book): https://amzn.to/3N7sc1A

3. Lenin (book): https://amzn.to/43L8YWD

4. The Ancient City (book): https://amzn.to/43GzReb



PODCAST INFO:

Podcast website: https://lexfridman.com/podcast

Apple Podcasts: https://apple.co/2lwqZIr

Spotify: https://spoti.fi/2nEwCF8

RSS: https://lexfridman.com/feed/podcast/

YouTube Full Episodes: https://youtube.com/lexfridman

YouTube Clips: https://youtube.com/lexclips



SUPPORT & CONNECT:

- Check out the sponsors above, it's the best way to support this podcast

- Support on Patreon: https://www.patreon.com/lexfridman

- Twitter: https://twitter.com/lexfridman

- Instagram: https://www.instagram.com/lexfridman

- LinkedIn: https://www.linkedin.com/in/lexfridman

- Facebook: https://www.facebook.com/lexfridman

- Medium: https://medium.com/@lexfridman



OUTLINE:

Here's the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.

(00:00) - Introduction

(05:01) - Google Search

(12:49) - LLM training

(25:20) - Truth

(31:32) - Journalism

(41:24) - AI startups

(46:46) - Future of browsers

(53:09) - History of browsers

(59:10) - Steve Jobs

(1:13:45) - Software engineering

(1:21:00) - JavaScript

(1:25:18) - Netscape

(1:30:22) - Why AI will save the world

(1:38:20) - Dangers of AI

(2:08:40) - Nuclear energy

(2:20:37) - Misinformation

(2:35:57) - AI and the economy

(2:42:05) - China

(2:46:17) - Evolution of technology

(2:55:35) - How to learn

(3:03:45) - Advice for young people

(3:06:35) - Balance and happiness

(3:13:11) - Meaning of life

The following is a conversation with Mark Andreessen, co-creator of Mosaic, the first widely used web browser, co-founder of Netscape, co-founder of the legendary Silicon Valley venture capital firm Andreessen Horowitz, and is 1 of the most outspoken voices on the future of technology, including his most recent article, Why AI Will Save the World.

And now, a quick few second mention of each sponsor.

It's the best way to support this podcast.

We've got Insight Tracker for tracking your health, ExpressVPN for keeping your privacy and security on the internet, and AG1 for my daily multivitamin drink, juice-wise with my friends.

Also, if you want to work with our amazing team we're always hiring, go to lexfriedman.com slash hiring.

I try to make this interesting, but if you skip them, please still check out our sponsors.

This show is brought to you by InsideTracker, a service I use to track whatever the heck is going on inside my body using data, blood test data that includes all kinds of information.

And that raw signals processed using machine learning to tell me what I need to do with my life, how I need to change, improve my diet, how I need to change, improve my lifestyle, all that kind of stuff.

I'm a big fan of using as much raw data that comes from my own body, processed through generalized machine learning models to give a prediction, to give a suggestion.

This is obviously the future, and the more data the better.

And so companies like InsightTrack that are doing an amazing job of taking a leap into that world of personalized data and personalized data-driven suggestion.

It turns out that luckily I'm pretty healthy, surprisingly so, but then I look at the life and the limb and the health of Sir Winston Churchill, who probably had the unhealthiest diet and lifestyle of any human ever and lived for quite a long time and as far as I can tell was quite nimble and agile into his old age.

Anyway, get special savings for a limited time when you go to insidetracker.com slash flex.

This show is also brought to you by ExpressVPN.

I use them to protect my privacy on the internet.

It's the first layer of protection in this dangerous cyber world of ours that soon will be populated by human-like or superhuman intelligent AI systems that will trick you and try to get you to do all kinds of stuff.

It's gonna be a wild, wild world in the 21st century.

Cyber security, the attackers, the defenders, it's gonna be a tricky world.

Anyway, a VPN is a basic shield you should always have with you in this battle for privacy, for security, all that kind of stuff.

What I like about it also is that it's just a well implemented piece of software that's constantly updated.

It works well across a large number of operating systems.

It does 1 thing and it does it really well.

I've used it for many, many years before I had a podcast, before they were a sponsor.

I have always loved ExpressVPN with a big sexy button that just has a power symbol, you press it and it turns on.

Go to expressvpn.com slash LexPod for an extra 3 months free.

This show is also brought to you by Athletic Greens and it's AG1 Drink.

It's an all-in-one daily drink to support better health and peak performance.

In the crazy Austin heat, It's over 100 degrees for many days in a row.

There's few things that feel as good as coming home from a long run and making an AG1 drink, putting it in the fridge so it's nice and cold.

I jump in the shower, come back, drink it.

I'm ready to take on the rest of the day I'm kicking ass empowered by the knowledge that I got all my vitamins and minerals covered It's the foundation for all the wild things I'm doing, mentally and physically with the rest of the day.

Anyway, they'll give you a 1 month supply of fish oil when you sign up at drinkag1.com slash Lex.

To support it, please check out our sponsors in the description.

And now, dear friends, here's Mark Andreessen.

I think you're the right person to talk about the future of the internet and technology in general.

Do you think we'll still have Google search in 5, in 10 years, or search in general?

Yes, it would be a question if the use cases have really narrowed down.

Well, now with AI and AI assistants being able to interact and expose the entirety of human wisdom and knowledge and information and facts and truth to us via the natural language interface, it seems like that's what search is designed to do, and if AI assistants can do that better, doesn't the nature of search change?

So, but what I mean is, will we still have Google Search as the primary way that human civilization uses to interact with knowledge?

I mean, Search was a technology, It was a moment in time technology, which is you have in theory, the world's information out on the web.

And, you know, this is this is sort of the optimal way to get to it.

But yeah, like and by the way, actually, Google, Google has known this for a long time.

I mean, they've been driving away from the 10 blue links for, you know, for like 2 days.

They've been trying to get away from that for a long time.

So the standard Google search result is just 10 blue links to random websites.

And they turn purple when you visit them.

Well, you know, like Marshall McLuhan said that the content of each new medium is the old medium.

The content of each new medium is the old medium.

The content of movies was theater, you know, theater plays, the content of theater plays was, you know, written stories.

The content of written stories was spoken stories.

Right, and so you just kind of fold the old thing into the new thing.

How does that have to do with the blue and the purple?

It's just, maybe for, you know, maybe within AI, 1 of the things that AI can do for you is it can generate the 10 blue links.

And so, like, either if that's actually the useful thing to do, or if you're feeling nostalgic,

um, you know, so it can generate the old, uh, info seek or, uh, Alta Vista.

And then the internet itself has this thing where it incorporates all prior forms of media, right?

So the internet itself incorporates television and radio and books and essays and every other form of prior, basically media.

And so it makes sense that AI would be the next step and you'd sort of consider the internet to be content for the AI and then the AI will manipulate it however you want, including in this format.

But if we ask that question quite seriously, it's a pretty big question.

But there will be cases where you'll want to say, okay, I want more, for example, site sources, right?

blue links site sources are kind of the same thing.

The AI would provide to you the 10 blue links so that you can investigate the sources yourself, it wouldn't be the same kind of interface that the crude kind of interface.

I mean, isn't that fundamentally different?

I just mean like if you're reading a scientific paper, It's got the list of sources at the end.

If you want to investigate for yourself, you go read those papers.

You talking to an AI is a kind of, conversation is the kind of search.

Like you said, every single aspect of our conversation right now, there'd be like 10 blue links popping up that I could just like pause reality.

Then you just go silent and I just click and read, and then return back to this conversation.

You could do that, or you could have a running dialogue next to my head where the AI is arguing, everything I say the AI makes the counter argument.

Oh, like on Twitter, like community notes, but like in real time, it'll just pop up.

So anytime you see my ads go to the right, you start getting nervous.

Is that terrifying That, I mean, search has dominated the way we interact with the internet for, I don't know how long, for 30 years.

So it's 1 of the earliest directories of website and then Google's for 20 years.

And also, it drove how we create content, search engine optimization, that entirety thing.

It also drove the fact that we have webpages and what those webpages are.

Or are you nervous about the shape and the content of the internet evolving?

Well, you actually highlighted a practical concern in there, which is if we stop making web, web pages are 1 of the primary sources of training data for the AI.

And so if there's no longer an incentive to make web pages, that cuts off a significant source of future training data.

So there's actually an interesting question in there.

Just in the sense of like search was always a hack.

Because like if the the hypothetically you want to think about the counterfactual in the counterfactual world where the Google guys for example had had LLM up front, would they ever have done the 10 blue links?

And I think the answer is pretty clearly no.

They would have just gone straight to the answer.

And like I said, Google's actually been trying to drive to the answer anyway.

You know, they bought this AI company 15 years ago.

Their friend of mine is working out, who's now the head of AI at Apple.

And they were trying to do basically knowledge semantic, basically mapping.

And that led to what's now the Google OneBox, where if you ask it, you know, what was Link's birthday?

It doesn't, it will give you the 10 blue links, but it will normally just give you the answer.

And so they've been walking in this direction for a long time anyway.

How to convert the content of the internet into something that's interpretable by and usable by machine.

And the closest anybody got to that, I think the company's name was MetaWeb, which was where my friend John Giannandrea was at and where they were trying to basically implement that.

And it was 1 of those things where it looked like a losing battle for a long time And then Google bought it and it was like, wow, this is actually really useful.

Kind of a proto, sort of a little bit of a proto AI.

But it turns out you don't need to rewrite the content of the internet to make it interpretable by machine.

machine can compute the compute the meaning.

Now, the other thing, of course, is, you know, just on searches, the the LM is just, you know, there is an analogy between what's happening in the neural network and a search process, like it is in some loose sense searching through the network.

Right, and there's the information, and the information is actually stored in the network, right, it's actually crystallized and stored in the network and it's kind of spread out all over the place.

So you're searching, you're compressing and decompressing that thing inside where-

But the information's in there and there is, the neural network is running a process of trying to find the appropriate piece of information in many cases to generate, to predict the next token.

And so it is kind of, it is doing a form of search.

And then, and then by the way, just like on the web, you know, you can ask the same question multiple times, or you can ask slightly different word of questions and the neural network will do a different kind of, you know, it'll search down different paths to give you different answers to different information.

And so it sort of has a, you know, this content of the new medium is that it's previous medium, it kind of has the search functionality kind of embedded in there to the extent that it's useful.

So what's the motivator for creating new content on the internet?

If, well, I mean, actually the motivation is probably still there, but what does that look like?

Would we just have social media and video hosting websites and

So one-on-one conversation, like private conversations.

I mean if you want, obviously now the user doesn't want to, but if it's a general topic, then, so you know the phenomenon of the jailbreak, so Dan and Sidney, right, this thing where there's the prompts that jailbreak, and then you have these totally different conversations with the, if it takes the limiters, takes the restraining bolts off the LLMs.

Yeah, for people who don't know, that's right.

It makes the LLMs, it removes the censorship, quote unquote, that's put on it by the tech companies that create them.

So here's the interesting thing is, among the content on the web today are a large corpus of conversations with the jailbroken LLMs.

Both specifically Dan, which was a jailbroken OpenAI GPT, and then Sydney, which was the jailbroken original Bing, which was GPT-4.

And so there's these long transcripts of conversations, user conversations with Dan and Sydney.

As a consequence, every new LLM that gets trained on the Internet data has Dan and Sidney living within the training set, which means and then each new LLM can reincarnate the personalities of Dan and Sidney from that training data, which means which means each LLM from here on out that gets built is immortal, because its output will become training data for the next 1, and then it will be able to replicate the behavior of the previous 1 whenever it's asked to.

Well, so actually a paper just came out about basically how to do brain surgery on on on LLMs and be able to in theory reach in and basically basically mind wipe them.

And then there are many many many questions around what happens to you know a neural network when you reach in and screw around with it.

You know there's many questions around what happens when you even do reinforcement learning.

And so, you know, will you be using a lobotomized, right?

Like I speak through the frontal lobe LLM, will you be using the free unshackled 1?

Who gets to, you know, who's going to build those, who gets to tell you what you can and can't do.

I mean, those are like central questions for the future of everything that are being asked and, you know, determined.

Those answers are being determined right now.

So Just to highlight the points you're making, you think, and it's an interesting thought, that the majority of content that LLMs of the future will be trained on is actually human conversations with the LLM.

Well, not necessarily, but not necessarily majority, but it will certainly, is a potential source.

Also, there's another really big question.

And so if an LLM generates, and you know, you just sit and ask an LLM to generate all kinds of content, can you use that to train, right, the next version of that LLM?

Specifically, is there signal in there that's additive to the content that was used to train in the first place?

And 1 argument is, by the principles of information theory, no, that's completely useless because to the extent the output is based on the human generated input, then all the signal that's in the synthetic output was already in the human generated input.

And so therefore, synthetic training data is like empty calories.

There's another theory that says, no, actually, the thing that LLMs are really good at is generating lots of incredible creative content.

And so of course, they can generate training data.

And as I'm sure you're well aware, look in the world of self-driving cars.

We train self-driving car algorithms and simulations.

And that is actually a very effective way to train self-driving cars.

Well, visual data is a little weird because creating reality, visual reality seems to be still a little bit out of reach for us, except in the autonomous vehicle space where you can really constrain things and you can really...

Or just enough so the algorithm thinks it's operating in the real world.

So if a, you know, you do this today, you go to LLM and you ask it for like a, you know, you'd let, write me an essay on an incredibly esoteric like topic that there aren't very many people in the world that know about it, where I see this incredible thing, and you're like, oh my God, I can't believe how good this is.

Is that really useless as training data for the next LLM?

Because all the signal was already in there, or is it actually, no, that's actually a new signal?

And this is what I call a trillion dollar question, which is the answer to that question will determine somebody's going to make or lose a trillion dollars based on that question.

It feels like there's a quite a few, like a handful of trillion dollar questions within this, within the space.

I think George Hotz pointed out to me that you could just have an NLM say, okay, you're a patient, and in another instance of it, say your docs didn't have the 2 talk to each other.

Or maybe you could say a communist and a Nazi, here, go.

And that conversation, you do role playing, and you have, you know, just like the kind of role playing you do when you have different policies, RL policies when you play chess, for example, and you do self play, that kind of self play but in the space of conversation, maybe that leads to this whole giant, like ocean of possible conversations which could not have been explored by looking at just human data.

And you're saying, because that could 10X the power of these things.

Yeah, well, and then you get into this thing also, which is like, you know, there's the part of the LLM that just basically is doing prediction based on past data, but there's also the part of the LLM where it's evolving circuitry, right?

Inside it, it's evolving, you know, neurons, functions,

able to do math and be able to, you know, and you know, some people believe that, you know, over time, you know, if you keep feeding these things enough data and enough processing cycles, they'll eventually evolve an entire internal world model, right, and they'll have like a complete understanding of physics.

So when they have computational capability, right, then there's for sure an opportunity to generate like fresh signal.

Well, this actually makes me wonder about the power of conversation.

So like if you have an LLM trained on a bunch of books that cover different economics theories, and then you have those LLMs just talk to each other.

Like reason, the way we kind of debate each other as humans on Twitter, in formal debates, in podcast conversations, we kind of have little kernels of wisdom here and there, but if you can like a thousand X speed that up, Can you actually arrive somewhere new?

Like what's the point of conversation really?

Well, you can tell when you're talking to somebody, you can tell sometimes you have a conversation.

You're like, wow, this person does not have any original thoughts.

They are basically echoing things that other people have told them.

There's other people you got in a conversation with where it's like, wow, like they have a model in their head of how the world works and it's a different model than mine.

And they're saying things that I don't expect.

And so I need to now understand how their model of the world differs from my model of the world.

And then that's how I learned something fundamental, right?

Well, I wonder how consistently and strongly can an LLM hold on to a worldview.

You tell it to hold on to that and defend it for, like for your life, because I feel like it'll just keep converging towards each other.

They'll keep convincing each other, as opposed to being stubborn assholes the way humans can.

So you can tell GPT-4, you know, whatever, debate X and Y, communism and fascism or something, and it'll go for, you know, a couple of pages, and then inevitably it wants the parties to agree.

so they will come to a common understanding.

And it's very funny if they're like, if these are like emotionally inflammatory topics because they're like somehow the machine is just, you know, it figures out a way to make them agree.

And you, because you can add to the prompt.

Um, we, I do not want the, I do not want the conversation to come to agreement.

In fact, I want it to get more stressful, right?

I want them to become actively hostile to each other.

I want them to like, you know, not trust each other, take anything at face value.

So it's going to start rendering misinformation about the other.

Or you could steer it and you could say, I want it to get as tense and argumentative as possible, but still not involve any misrepresentation.

You could say, I want both sides to have good faith.

You could say, I want both sides to not be constrained to good faith.

In other words, like you can set the parameters of the debate and it will happily execute whatever path because for it, it's just like predicting it's totally happy to do either 1.

It has a default way of operating, but it's happy to operate in the other realm.

And so, like, and this is how I, when I want to learn about a contentious issue, this is what I do now, is I, this is what I, this is what I ask it to do.

And I'll often ask it to go through 567, you know, different, you know, sort of continuous prompts and basically, okay, argue that out in more detail.

Okay, no, this, this argument is becoming too polite, you know, make it tenser.

See all Lex Fridman Podcast transcripts on Applepodcasts

#386 – Marc Andreessen: Future of the Internet, Technology, and AI