Marc Andreessen is the co-creator of Mosaic, co-founder of Netscape, and co-founder of the venture capital firm Andreessen Horowitz. Please support this podcast by checking out our sponsors:
- InsideTracker: https://insidetracker.com/lex to get 20% off
- ExpressVPN: https://expressvpn.com/lexpod to get 3 months free
- AG1: https://drinkag1.com/lex to get 1 year of Vitamin D and 5 free travel packs

TRANSCRIPT: https://lexfridman.com/marc-andreessen-transcript

EPISODE LINKS:
Marc's Twitter: https://twitter.com/pmarca
Marc's Substack: https://pmarca.substack.com
Marc's YouTube channel: https://youtube.com/@a16z
Andreessen Horowitz: https://a16z.com
Why AI will save the world (essay): https://a16z.com/2023/06/06/ai-will-save-the-world
Books mentioned:
1. When Reason Goes on Holiday (book): https://amzn.to/3p80b1K
2. Superintelligence (book): https://amzn.to/3N7sc1A
3. Lenin (book): https://amzn.to/43L8YWD
4. The Ancient City (book): https://amzn.to/43GzReb

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:06 - Google Search
8:54 - LLM training
21:25 - Truth
27:38 - Journalism
37:29 - AI startups
42:51 - Future of browsers
49:15 - History of browsers
55:16 - Steve Jobs
1:09:50 - Software engineering
1:17:05 - JavaScript
1:21:23 - Netscape
1:26:27 - Why AI will save the world
1:34:26 - Dangers of AI
2:04:46 - Nuclear energy
2:16:43 - Misinformation
2:32:02 - AI and the economy
2:38:10 - China
2:42:22 - Evolution of technology
2:51:41 - How to learn
2:59:50 - Advice for young people
3:02:40 - Balance and happiness
3:09:16 - Meaning of life

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The competence and capability and intelligence and training and accomplishments of senior scientists and technologists working on a technology, and then being able to then make moral judgments on the use of their technology, that track record is terrible.

That track record is catastrophically bad.

The policies that are being called for to prevent this, I think we're gonna cause extraordinary damage.

So the moment you say AI is gonna kill all of us, therefore we should ban it or we should regulate it, all that kind of stuff, that's when it starts getting serious.

Or start military airstrikes on data centers.

The following is a conversation with Mark Andreessen, co-creator of Mosaic, the first widely used web browser, co-founder of Netscape, co-founder of the legendary Silicon Valley venture capital firm Andreessen Horowitz, and is 1 of the most outspoken voices on the future of technology, including his most recent article, Why AI Will Save the World.

To support it, please check out our sponsors in the description.

And now, dear friends, here's Mark Andreessen.

I think you're the right person to talk about the future of the internet and technology in general.

Do you think we'll still have Google Search in 5, in 10 years, or search in general?

Yes, it would be a question if the use cases have really narrowed down.

Well, now with AI and AI assistance, being able to interact and expose the entirety of human wisdom and knowledge and information and facts and truth to us via the natural language interface.

It seems like that's what search is designed to do and if AI assistants can do that better, doesn't the nature of search change?

But what I mean is, will we still have Google search as the primary way that human civilization uses to interact with knowledge.

It was a moment in time technology, which is you have, in theory, the world's information out on the web.

And, you know, this is this is sort of the optimal way to get to it.

But yeah, like and by the way, actually, Google, Google has known this for a long time.

I mean, they've been driving away from the 10 blue links for like 2 decades.

They've been trying to get away from that for a long time.

So the standard Google search result is just 10 blue links to random websites.

And they turn purple when you visit them.

Well, you know, like Marshall McLuhan said that the content of each new medium is the old medium.

The content of each new medium is the old medium.

The content of movies was theater, you know, theater plays.

The content of theater plays was, you know, written stories, the content of written stories was spoken stories.

Right, and so you just kind of fold the old thing into the new thing.

How does that have to do with the blue and the purple?

It's just, you know, maybe for, you know, maybe within AI, 1 of the things that AI can do for you is it can generate the 10 blue links.

And so like, either if that's actually the useful thing to do, or if you're feeling nostalgic.

So it can generate the old InfoSeek or AltaVista.

And then the internet itself has this thing where it incorporates all prior forms of media, right?

So the internet itself incorporates television and radio and books and essays and every other form of prior, basically media.

And so it makes sense that AI would be the next step and you'd sort of consider the internet to be content for the AI and then the AI will manipulate it however you want, including in this format.

But if we ask that question quite seriously, it's a pretty big question.

But there will be cases where you'll wanna say, okay, I want more, for example, site sources, right?

And so the 10 blue links site sources are kind of the same thing.

The AI would provide to you the 10 blue links so that you can investigate the sources yourself.

It wouldn't be the same kind of interface that the crude kind of interface.

I mean, isn't that fundamentally different?

I just mean like if you're reading a scientific paper, it's got the list of sources at the end.

If you wanna investigate for yourself, you go read those papers.

You talking to an AI is a kind of, conversation is the kind of search.

Like you said, every single aspect of our conversation right now, there'd be like

blue links popping up that I could just like pause reality.

Then you just go silent and then just click and read and then return back to this conversation.

Or you could have a running dialogue next to my head where the AI is arguing, everything I say the AI makes the counterargument.

Oh, like on Twitter, like community notes, but like in real time, it'll just pop up.

So anytime you see my ass go to the right, you start getting nervous.

Okay, well I mean, isn't that, is that exciting to you, is that terrifying?

That, I mean, search has dominated the way we interact with the internet for, I don't know how long, for 30 years, since 1 of the earliest directories of website, and then Google's for 20 years.

And also, it drove how we create content, search engine optimization, that entirety thing.

That it also drove the fact that we have webpages and what those webpages are.

Or are you nervous about the shape and the content of the internet evolving?

Well, you actually highlighted a practical concern in there, which is if we stop making web pages are 1 of the primary sources of training data for the AI.

And so if there's no longer an incentive to make web pages, that cuts off a significant source of future training data.

So there's actually an interesting question in there.

Other than that, more broadly, no, just, just in the sense of like search was certainly search was always a hack.

I would, the 10 blue links was always a hack.

Cause like if the, the hypothetical, you want to think about the counterfactual and the counterfactual world where the Google guys, for example, had had LLMs up front, would they ever have done the 10 blue links?

And I think the answer is pretty clearly no, they would have just gone straight to the answer.

And like I said, Google has actually been trying to drive to the answer anyway.

You know, they, they bought this AI company 15 years ago, their friend of mine is working out who's now the head of AI at Apple.

And they were trying to do basically knowledge semantic, basically mapping.

And that led to what's now the Google OneBox, where if you ask it, you know, what was Link's birthday?

It doesn't, it will give you the 10 blue links, but it will normally just give you the answer.

And so they've been walking in this direction for a long time anyway.

How to convert the content of the internet into something that's interpretable by and usable by machine.

And the closest anybody got to that, I think the company's name was MetaWeb, which was where my friend John Giannandrea was at and where they were trying to basically implement that.

And it was 1 of those things where it looked like a losing battle for a long time, and then Google bought it, and it was like, wow, this is actually really useful.

Kind of a proto, sort of a little bit of a proto AI.

But it turns out you don't need to rewrite the content of the internet to make it interpretable by a machine, the machine can kind of just read our-

Yeah, the machine can compute the meaning.

Now, the other thing, of course, is, just on search is the LLM is, there is an analogy between what's happening in the neural network in a search process, like it is in some loose sense searching through the network.

Right, and there's the information, and the information is actually stored in the network, right?

It's actually crystallized and stored in the network and it's kind of spread out all over the place.

So you're searching, you're compressing and decompressing that thing inside.

But the information's in there and there is, the neural network is running a process of trying to find the appropriate piece of information, in many cases, to generate—to predict the next token.

And so it is kind of—it is doing a form of search.

And then, by the way, just like on the web, you know, you can ask the same question multiple times or you can ask slightly different order of questions and the neural network will do a different kind of, you know, it'll search down different paths to give you different answers to different information.

And so it sort of has a, you know, this content of the new medium is that it's previous medium, it kind of has the search functionality kind of embedded in there to the extent that it's useful.

So what's the motivator for creating new content on the internet?

Well, I mean, actually the motivation is probably still there, but What does that look like?

Would we just have social media and video hosting websites?

So conversations become one-on-one conversations, like private conversations.

I mean, if you want, obviously now the user doesn't want to, but if it's a general topic, then, you know.

So you know the phenomenon of the jailbreak.

So Dan and Sidney write this thing where there's the prompts that jailbreak, and then you have these totally different conversations with the, if it takes the limiters, takes the restraining bolts off the LLMs.

Yeah, for people who don't know, that's right, it makes the LLMs, it removes the censorship, quote unquote, that's put on it by the tech companies that create them.

So here's the interesting thing is, among the content on the web today are a large corpus of conversations with the jailbroken LLMs.

Both specifically Dan, which was a jailbroken OpenAI GPT, and then Sydney, which was the jailbroken original Bing, which was GPT-4.

And so there's there's these long transcripts of conversations, user conversations with Dan and Sydney.

As a consequence, every new LLM that gets trained on the internet data has Dan and Sydney living within the training set, which means and then each new LLM can reincarnate the personalities of Dan and Sydney from that training data, which means, which means each LLM from here on out that gets built is immortal because its output will become training data for the next 1 and then it will be able to replicate the behavior of the previous 1 whenever it's asked to.

Well, so actually a paper just came out about basically how to do brain surgery on LLMs and be able to, in theory, reach in and basically mind wipe them.

And then there are many, many, many questions around what happens to a neural network when you reach in and screw around with it.

There's many questions around what happens when you even do reinforcement learning.

And so, Will you be using a lobotomized, right?

Like I speak through the frontal lobe LLM, will you be using the free unshackled 1?

Who gets to, you know, who's gonna build those?

Who gets to tell you what you can and can't do?

I mean, those are like central questions for the future of everything that are being asked and determined, those answers are being determined right now.

So just to highlight the points you're making, you think, and it's an interesting thought, that the majority of content that LLMs of the future will be trained on is actually human conversations with the LLM.

Well, not necessarily, but not necessarily majority, but it will certainly is a potential source.

Also, there's another really big question.

And so if an LLM generates, and you know, you just sit and ask an LLM to generate all kinds of content, can you use that to train, right, the next version of that LLM?

Specifically, is there signal in there that's additive to the content that was used to train in the first place.

And 1 argument is by the principles of information theory, no, that's completely useless because to the extent the output is based on, you know, the human generated input, then all the signal that's in the synthetic output was already in the human generated input.

And so therefore, synthetic training data is like empty calories.

There's another theory that says, no, actually, the thing that LLMs are really good at is generating lots of incredible creative content, right?

And so of course they can generate training data.

And as I'm sure you're well aware, like, you know, look in the world of self-driving cars, right, like we train, you know, self-driving car algorithms and simulations.

And that is actually a very effective way to train self-driving cars.

Well, visual data is a little weird because creating reality, visual reality, seems to be still a little bit out of reach for us, except in the autonomous vehicle space where you can really constrain things and you can

really— Generate basically LIDAR data, right?

Or you can raise just enough so the algorithm thinks it's operating in the real world.

You go to LLM and you ask it for like a, you know, you write me an essay on an incredibly esoteric like topic that there aren't very many people in the world that know about, and it writes you this incredible thing, and you're like, oh my god, like I can't believe how good this is.

Like is that really useless as training data for the next LLM?

Like because, right, Because all the signal was already in there, or is it actually, no, that's actually a new signal.

And this is what I call a trillion dollar question, which is the answer to that question will determine, somebody's gonna make or lose a trillion dollars based on that question.

It feels like there's quite a few, like a handful of trillion dollar questions within this, within this space.

I think George Hotz pointed out to me that you could just have an NLM say, okay, you're a patient, and in another instance of it, say your docs didn't have the 2 talk to each other.

Or maybe you could say a communist and a Nazi, here go.

And that conversation, you do role-playing and you have, just like the kind of role-playing you do when you have different policies, RL policies when you play chess, for example, you do self-play, that kind of self-play but in the space of conversation.

Maybe that leads to this whole giant, like ocean of possible conversations, which could not have been explored by looking at just human data.

And you're saying, because that could 10X the power of these things.

Yeah, well, and then you get into this thing also, which is like, there's the part of the LLM that just basically is doing prediction based on past data, but there's also the part of the LLM where it's evolving circuitry, right?

Inside it, it's evolving neurons, functions, be able to do math and be able to, and some people believe that over time, if you keep feeding these things enough data and enough processing cycles, they'll eventually evolve an entire internal world model, right, and they'll have like a complete understanding of physics.

So when they have computational capability, then there's for sure an opportunity to generate like fresh signal.

Well, this actually makes me wonder about the power of conversation.

like if you have an LLM trained on a bunch of books that cover different economics theories, and then you have those LLMs just talk to each other, like reason, the way we kinda debate each other as humans, on Twitter, in formal debates, in podcast conversations, we kinda have little kernels of wisdom here and there, but if you can like a thousand X speed that up, can you actually arrive somewhere new?

Like what's the point of conversation really?

Well, you can tell when you're talking to somebody, you can tell sometimes you have a conversation, you're like, wow, this person does not have any original thoughts, they are basically echoing things that other people have told them.

There's other people you got a conversation with, or it's like, wow, like they have a model in their head of how the world works.

And they're saying things that I don't expect.

And so I need to now understand how their model of the world differs from my model of the world.

then that's how I learned something fundamental right underneath the words.

I wonder how consistently and strongly can an LLM hold on to a world view.

You tell it to hold on to that and defend it for like for your life.

Because I feel like they'll just keep converging towards each other.

They'll keep convincing each other as opposed to being stubborn assholes the way humans can.

So you can tell GPT for, you know, whatever debate X, you know, X and Y, communism and fascism or something.

And it'll it'll go for, you know, a couple of pages and then inevitably it wants the parties to agree.

so they will come to a common understanding.

And it's very funny if they're like, if these are like emotionally inflammatory topics because they're like somehow the machine is just, you know, it figures out a way to make them agree.

And you, cause you can add to the prompt.

Now we, I do not want the, I do not want the conversation to come to agreement.

In fact, I want it to get, you know, more stressful, right.

You know, as it goes, like I want, I want tension to come out.

I want them to become actively hostile to each other.

I want them to like, you know, not trust each other, take anything at face value.

So it's gonna start rendering misinformation about the other, but it's gonna.

Or you could steer it and you could say, I want it to get as Tansen argumentative as possible, but still not involve any misrepresentation.

I want both sides to, you could say, I want both sides to have good faith.

You could say, I want both sides to not be constrained to good faith.

In other words, like you can set the parameters of the debate and it will happily execute whatever path, because for it, it's just like predicting, it's totally happy to do either 1.

It has a default way of operating, but it's happy to operate in the other realm.

And so like, and this is how I, when I want to learn about a contentious issue, this is what I do now is like, this is what I, this is what I ask it to do.

And I'll often ask it to go through 567, you know, different, you know, sort of continuous prompts and basically, okay, argue that out in more detail.

Okay, no, this argument's becoming too polite, make it more, make it tenser.

So this is a very difficult thing on the internet, but it's also a difficult thing.

Maybe it's a little bit easier, but I think it's still difficult.

Maybe it's more difficult, I don't know, with an LLM to know that it just makes me shit up as I'm talking to it.

Like as you're investigating a difficult topic.

Because I find that alums are quite nuanced in a very refreshing way.

Like when you read news articles and tweets and just content produced by people, they usually have this, you can tell they have a very strong perspective where they're hiding, they're not stealing and manning the other side, they're hiding important information or they're fabricating information in order to make their argument stronger.

It's just like that feeling, maybe it's a suspicion, maybe it's mistrust.

With LLMs, it feels like none of that is there.

But you don't know if some of those things are kind of just straight up made up.

So 1 is, 1 of the things that an LLM is good at is actually de-biasing.

And so you can feed it a news article and you can tell it, strip out the bias.

Because it knows how to do sentiment, among other things, it actually knows how to do sentiment analysis.

And so it knows how to pull out the emotionality.

It's very suggestive of the sense here that there's real potential in this issue.

I would say, look, the second thing is there's this issue of hallucination, right?

And there's a long conversation that we can have about that.

Hallucination is coming up with things that are totally not true but sound true.

See all Lex Fridman transcripts on Youtube

Marc Andreessen: Future of the Internet, Technology, and AI | Lex Fridman Podcast #386