- The discussion revolves around the shift in attention from AI to cryptocurrency.
- The paper titled "The Malicious Use of Artificial Intelligence" is discussed, focusing on intentional misuse of AI.
- The need for norms and responsible disclosure in AI is highlighted, similar to other fields like biotechnology and computer security.
- The research involves analyzing current trends and extrapolating potential uses and risks of AI.
- The discussion also touches on the potential impact of AI in physical vs. virtual realms.
- The desensitization of the public to data breaches and the potential for catastrophic cyber incidents are concerns.
- Public perception and understanding of AI are discussed, with examples like News Feed being considered AI.

- The video discusses the use of AI and machine learning in various industries.
- One major concern is the international competition in AI, particularly with China's investments in the field.
- Interoperability is another challenge, as it is difficult to explain the decisions made by AI systems.
- The issue of robustness and defense against adversarial attacks is also mentioned.
- In terms of positive applications, the field of health shows great potential for AI.
- Deployment of AI in the health sector is still in the pilot stage, with issues of interpretability and fairness that need to be resolved.
- The debate on interpretability in AI systems is ongoing, with some researchers arguing that it is not necessary.
- Different countries have different regulations and expectations regarding AI and its explanations.

- The US and Europe have different approaches to privacy and regulation in relation to AI.
- The US takes a case-by-case approach, creating specific regulations for different domains, while Europe has broader regulations that apply to automated decision-making systems.
- The speaker is studying science policy and focusing on methods for AI policy, particularly in dealing with uncertainty and expert disagreement.
- They are using scenario planning and workshops to explore the usefulness of these methods.
- There is a lack of dialogue between AI ethics/governance and science/public policy fields.
- The speaker aims to bring more rigor and clarity to AI policy by examining the links between assumptions, policies, and scenarios.
- The history of policy around AI is not well-known, but there have been debates and calls for regulation in other technological fields.
- AI policy is still developing, and there is a need to determine what areas are valuable to work on.
- The speaker and their team gather information by talking to people in industry and academia, making predictions, and engaging in translation work between technical research and policy discussions.
- Collaboration between those interested in societal and technical questions is important, as there is a need for new studies and collaboration to answer policy-relevant questions.
- The speaker expresses interest in studying the evolution of policy engagement in the climate science field.
- The responsibilities of a field and the arrangement of skills and talents around those responsibilities are important to consider.
- The speaker discusses their experience in public policy at Google and the differences between policy within a company and within an institute.

- The speaker has experience in public policy for Google and is now working with Harvard, MIT, and the Oxford Internet Institute on ethics and governance of AI.
- People in both industry and civil society are concerned about fairness in AI.
- Different sectors (industry, government, academia) have different constraints and focus areas in addressing AI issues.
- The Future of Humanity Institute (FHI) is an interdisciplinary research institute focusing on AI and biotechnology, as well as broader issues related to the future of humanity.
- FHI is motivated by reducing existential risks associated with AI, but also interested in long-term issues and positive applications of AI.
- There is common cause between short-term and long-term AI concerns, such as fairness, accountability, and transparency.
- Value alignment is an important concept in AI safety, addressing issues of bias and consistency with human values.
- Transparency in AI systems could lead to more collaboration and international cooperation.
- The idea of accountable and transparent AGI (Artificial General Intelligence) is considered challenging but not impossible.

- The development of AGI (Artificial General Intelligence) may involve humans being in the loop and continuously providing critical feedback.
- Norms around building corrigible systems could set a precedent for the design of AGI in the future.
- There is a debate on whether AGI research should be conducted publicly or privately, with considerations for domains where openness may pose risks.
- Open publishing is generally favored to allow for realistic assessments and informed public policy.
- Predictions include the possibility of superhuman AI in games like Starcraft and Dota 2, as well as significant improvements in meta-learning.

- The discussion focuses on the shift of attention from AI to cryptocurrency.
- The paper "The Malicious Use of Artificial Intelligence" is discussed, highlighting intentional misuse of AI.
- The need for norms and responsible disclosure in AI is emphasized, similar to other fields like biotechnology and computer security.
- The research analyzes current trends and potential uses and risks of AI.
- The impact of AI in physical vs. virtual realms is discussed.
- Concerns include desensitization to data breaches and potential catastrophic cyber incidents.
- Public perception and understanding of AI, including examples like News Feed, are discussed.
- The use of AI and machine learning in various industries is explored.
- International competition in AI, particularly with China's investments, is a major concern.
- Challenges include interoperability, robustness against adversarial attacks, and interpretability.
- AI shows great potential in the health sector, but issues of interpretability and fairness need resolution.
- The debate on interpretability in AI systems is ongoing, with different regulations and expectations in different countries.
- The US and Europe have different approaches to privacy and regulation in relation to AI.
- The speaker focuses on AI policy, using scenario planning and workshops to explore methods.
- There is a lack of dialogue between AI ethics/governance and science/public policy fields.
- The history of policy around AI is not well-known, but debates and calls for regulation have occurred in other technological fields.
- Collaboration between societal and technical experts is important for answering policy-relevant questions.
- The speaker has experience in public policy at Google and is now working with Harvard, MIT, and the Oxford Internet Institute on AI ethics and governance.
- Concerns about fairness in AI exist in both industry and civil society.
- The Future of Humanity Institute (FHI) focuses on AI, biotechnology, and broader issues related to the future of humanity.
- Short-term and long-term AI concerns have common themes, such as fairness, accountability, and transparency.
- Value alignment is important in AI safety to address bias and consistency with human values.
- Transparency in AI systems could lead to more collaboration and international cooperation.
- The development of AGI (Artificial General Intelligence) may involve human involvement and continuous feedback.
- Norms around building corrigible systems could set a precedent for AGI design.
- There is a debate on conducting AGI research publicly or privately, considering domains where openness may pose risks.
- Open publishing is generally favored for realistic assessments and informed public policy.
- Predictions include superhuman AI in games and significant improvements in meta-learning.

Miles Brundage - https://twitter.com/Miles_Brundage - is an AI Policy Research Fellow with the Strategic AI Research Center at the Future of Humanity Institute - https://www.fhi.ox.ac.uk/. He is also a PhD candidate in Human and Social Dimensions of Science and Technology at Arizona State University.

Miles recently co-authored The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation - https://arxiv.org/abs/1802.07228

Tim Hwang - https://twitter.com/timhwang - is the Director of the Harvard-MIT Ethics and Governance of AI Initiative - https://techcrunch.com/2017/01/10/omidyar-hoffman-create-27m-research-fund-for-ai-in-the-public-interest/. He is also a Visiting Associate at the Oxford Internet Institute - https://www.oii.ox.ac.uk/ - and a Fellow at the Knight-Stanford Project on Democracy and the Internet - https://pacscenter.stanford.edu/research/project-on-democracy-and-the-internet/. This is Tim's second time on the podcast; he was also on episode 11 - https://blog.ycombinator.com/at-the-intersection-of-ai-governments-and-google-tim-hwang/.

The YC podcast is hosted by Craig Cannon - https://twitter.com/craigcannon

All right, guys, I think the most important and pressing question is, now that cryptocurrency gets all the attention and AI is no longer the hottest thing in technology, how are you dealing with it?

Yeah, Ben Hamner of Kaggle had a good line on this.

He said something like, great thing about cryptocurrency is people no longer ask me about whether there's an AI bubble.

And yeah, it's hard to compete with the crypto bubble or phenomenon, whatever you want to call it.

I think it's actually, yeah, good development, right?

Like I mean, the history of AI is like all of these winners, and having another hype cycle to kind of balance it out might actually be a good thing.

Let's talk about your paper to start off, Miles.

Yeah, it's called The Malicious Use of Artificial Intelligence.

And there's a subtitle, like forecasting, prevention, and mitigation.

And it's attempting to be the most comprehensive analysis to date of the various ways in which AI could be deliberately misused.

So not just things like bias and lack of fairness in an algorithm that are not necessarily intentional, but deliberately using it for things like fake news generation and combining AI with drones to carry out terrorist attacks or offensive cybersecurity applications.

And the essential argument that we make is that that needs to be taken seriously, the fact that AI is a dual use or even omni-use technology, and that similar to other fields like biotechnology and computer security, we need to think about whether there are norms that account for that.

So things like responsible disclosure, when you find out about a new vulnerability, is something that's pervasive in the computer security community, but hasn't yet been seriously discussed for things like adversarial examples, where you might want to say, hey, there's this new misuse opportunity or way in which you could fool this commercial system that is currently running driverless cars or whatever.

And so there should be some more discussion about those sorts of issues.

And so is it going into the technical details?

Or is it kind of a survey of where you think things should happen?

So most of it's a general survey, but then there's an appendix on different areas, like how to deal with the privacy issues, how to deal with the robustness issues, and different places to look for lessons.

And so, Tim, have you been focusing on any of this stuff while you've been here at Oxford, or is your work totally unrelated?

I mean, I would say that I've mostly been focusing on what you might think of as a subset of the problems that Miles is working on, where he's sort of saying, look, AI isn't going to be inherently used for good, and in fact, there's lots of intentional ways to use it for bad, right?

And 1 of the things I've been thinking about is the sort of interface between these techniques and the problems with disinformation, and whether or not you think these techniques will be used to make ever more believable fakes in the future and what that does to the media ecosystem.

So I would say it's a very particular kind of bad actor use that Mile's talking about.

And so when you're doing this research for both of these topics, are you digging into actual code?

Yeah, so I mean, my methodology is really kind of focused on looking at what is the research that's coming out right now, and trying to extrapolate what the uses might be.

Because I think 1 of the really interesting things we're seeing in the AI space is that it is becoming more available for people to do.

You've got these cloud services, We've got the tools are widely available now.

And so I think what's really missing is the ability to kind of figure out how you do it.

And the question is, do you see papers that are coming out saying, hey, we could actually use it for this somewhat disturbing purpose.

And then kind of extrapolating from there to say, okay, well, what would it mean for it to get used more widely?

Yeah, so reading papers, seeing what the hot areas are, and cases in which some sort of potentially negative or positive application is on the cusp of getting just efficient enough to be used by a wide array of people, or the hyperparameter optimization problem is close to being solved, or whatever sort of trend that you might see might be a sign that certain technologies are gonna be more widely usable, not just by experts, but potentially in a huge range of applications.

For the purpose of this report that I recently wrote, we got a ton of people together, including Tim, at a workshop, and we talked about technical trends and had people in cybersecurity and AI and other areas give their best guesses of what's possible and then prioritize what the risks are and what to do about them.

So I think often pulling together different disciplines is a good way to think about what's possible.

And then 1 other thing that I'll point out is that you don't necessarily have to even look into the technical literature to find discussion of these sorts of misuse applications today, because it's a hot topic already.

So things like deep fakes for face swapping and pornography is a huge media issue right now.

And that actually happened while we were writing this report.

And then we added something later about it.

Because we characterized the general issue of fake videos and misinformation and AI as making that more scalable because it potentially requires less expertise.

And while we're writing that, this deepfakes thing happens and it's democratizing in some sense the ability to create fake videos.

Right, and I think there's a really interesting question here, particularly when you think about prediction, about there's a realm of what can be done.

And then trying to understand what's likely to actually happen.

In fact, this seems to be the really challenging thing because there's like lots of terrible uses for almost every technology.

But we see certain uses more prominently than others.

And I think that's actually where the rub on this sort of stuff is and actually is part of this prediction problem.

And yeah, so That's why you kind of have to, yeah, I mean, first of all, have some humility about what you can predict.

Like, if it's a fully general purpose or a fairly general purpose technology, they can be steered in a bunch of different directions or applied to a bunch of different data sets.

Then you should expect that if it's super widely available, a bunch of people are going to find new uses for it.

So I mean, I think that's a reason to sort of look upstream at the papers and see like where the technical trends are, because then you can say like, well, you know, maybe this is not yet ready for primetime for any application or like this is starting to be like fairly general purpose.

Yeah, I mean, a good question for you, Myles, is whether or not you think that we'll see the virtual uses be the ones that happen first versus the physical ones.

So some people have said, oh, okay, well, you could use AI to really make hacking much easier

might be able to use it to create these fakes, right, which we're already seeing.

But I'm wondering if those threats kind of evolve in a way that's different or maybe even earlier than threats of like, people have talked about, What happens if someone builds a drone that goes out and uses algorithms to go hurt people?

I mean, I think 1 heuristic that I've used is that stuff in the physical world is often harder and it's both more expensive and less scalable because you have to buy actual robots and then there's often hardware issues that you run into, and the general problem of perception, and perception is much harder in the real world than in static data sets.

Just a few days ago, there were a bunch of cool videos from Skydio of their autonomous drone for tracking people doing sports and flying around.

It seems to be pretty good at navigating in forests and things like that.

So maybe technologies like that are sort of a sign that there'll be much more both positive and negative uses in the real world.

But yeah, I think in terms of near-term impact, I think those sorts of things that have those autonomous features still aren't super easy to use for end users outside of a particular domain.

So I'm not sure that anyone could just easily repurpose it to track a particular person or whatever.

I think it's sort of for that domain application.

I don't know how expensive it is, but it's probably more expensive than a $20 drone.

Because I think about what's the first harm that comes out of the gate in a really big way?

Because I've debated often, okay, so say there's a horrible self-driving car incident that occurs.

Maybe that turns society off in general to the whole technology, and there's a big categorical outlawing of it.

So I'm like, well, OK, that's kind of not so good.

But at the same time, I'm kind of like, OK, well, what if hacking becomes a lot more prominent in a way that's powered by machine learning?

But we know that, I don't know, the response to huge data disclosures or huge data compromises is actually quite limited public response.

And that seems not so good either, is basically people either overestimate the risk or underestimate the risk depending on what happens first.

People are starting to get desensitized to these mega disclosures.

And so maybe they won't even care if there's some adaptive malware thing that we might be like, whoa, that's kind of scary.

But it could be that something truly catastrophic could happen if you sort of combine the scalability of AI and digital technology in general with the adaptability of human intelligence for finding vulnerabilities.

If you put those together, you might have a really bad cyber incident that will actually make people be like, whoa, this AI thing.

Yeah, so that's something that worries me a lot.

But it's sort of like a moving goalpost on the positive and negative side, right?

So, you know, News Feed, for instance, like you could call that AI to a certain extent, right, as it's feeding you information.

People get mad at News Feed, they don't get mad at AI, right?

So like the notion that the public would generally turn on something like that seems almost unrealistic, right?

Because you want to just point at 1 thing, right?

I mean, I think that's it is basically like what the public thinks about as AI is an AI, right?

What we're actually talking about is like this weird amalgam of like popular culture, some research explanations that make it to the public, all these sorts of things.

And there's so much about what does the public actually think AI even is, which is really relevant to the discussion, right?

Because the newsfeed assuredly is AI, right?

It uses the latest machine learning to do what it does.

We don't really think about it as AI, right?

Whereas like the car is like, I mean, I think a lot of robots kind of fall into this category where even robots that don't involve any machine learning are thought of as AI, right?

And like actually impact the discussion about AI despite not actually being related to it at all in some absolute sense.

Well, then it sort of becomes a design challenge, right?

It's like why these self-driving cars are shaped like little bubbly toys, right?

They're so much less intimidating when you see it just bump into a little bollard on the street here, whatever.

But yeah, the robot, like the factory robot, for instance, those are terrifying to people.

But they've always been terrifying to people.

But surely there are positive things that you guys notice.

You're going around to these conferences, what questions are people asking you about AI?

What is the public concerned about positively and negatively?

So I think there's 2 things that are really atop of mind that I think keep coming up both in the popular discussion around AI right now and also among researcher circles.

So the first 1 is the question of international competition and what it looks like in the space.

So, this is a question of, it seems like China's making a lot of moves to really invest in AI in a big way.

What does that mean about these research fields, right?

Will the US and Canada and Europe sort of stay ahead in this game?

And what does that mean if you think that governments are going to see this as a national security thing?

Second 1 I think is around the issues of interoperability, which I think are a really big concern, which is these systems make decisions.

Can we render some kind of satisfying explanation for why they do what they do?

And I use the word satisfy specifically there because there's lots of ways of trying to tell how they do what they do, but this question of how you communicate it is a whole other issue.

And those seem to be 2 really big challenges.

I'm sure Miles has seen other things too.

The whole FAT ML community, fairness, accountability, and transparency in machine learning.

And now there's FAT Star, so it's more general than just machine learning.

Conference series and broader community has been doing a ton of awesome work on those sorts of issues.

But in addition to the transparency thing that Tim mentioned, I would also mention robustness.

And pretty much if you look at the offense and defense and competitions on adversarial examples, the offense generally wins.

We don't really know how to make neural nets robust against deliberate or even unintentional things that could mess them up.

They do really well according to 1 single number of human versus AI performance.

But then if it's slightly outside the distribution, they might fail, or if someone's deliberately tampering with it.

So that's a huge problem for actually applying these systems in the real world.

And I think we'll continue to see progress on that, but we'll also see setbacks where People say, well, this proposal you had for defending neural nets actually doesn't work.

And then there are all sorts of other things besides just adversarial examples.

Like there was a recent paper called Bad Nets that talked about backdoors in neural networks.

So essentially, someone can put a trained neural network on GitHub or wherever and then it seems to work fine but then you show it some special image and then it goes wrong.

In terms of positive applications, 1 area that is super exciting and that there's so much work on it that I've had to take a step back and not even try to tweet all the interesting stuff that I see on it is health.

So there's pretty much every day on archive, there's a new paper that's superhuman performance on this dermatology task or this esophagal cancer task.

So there's a ton of activity in that space.

And is that specific to, for instance, image recognition, like CT scan type stuff?

I think that's kind of the low hanging fruit, because there's all this progress in image recognition, and things like adversarial examples aren't necessarily a problem in that domain.

You're hoping that a patient isn't fiddling with their image, or putting a little turtle on their chest when they're getting scanned, and then it gives the wrong answer.

So, yeah, there's tons of applications there, but there's also just more general machine learning stuff, like predicting people relapsing and having to come back to the hospital, and when's the optimal time to send people home, or give in this huge data set of people's medical histories, what's the best diagnosis.

Yeah, there's a workshop at NIPS a few years back.

And I think that's a really good way of framing up a lot of the issues that we're seeing right now is we're moving out of the lab in some sense where it's like, okay, the old task used to be just like, could we optimize this algorithm to kind of do this thing better.

But now there's a bunch of research trying to figure out what do we do when we confront the practical problems of deploying these things in the world?

And that links a lot of the interoperability stuff, it links a lot of the safety stuff, it links these questions that are specific to health.

I think all these come out of the fact that the technology is really finally becoming practical and so you have to solve some of these really practical questions.

And so as far as deploying this stuff in the wild in the health use case, who is using it right now?

So there'll be a hospital here, a medical center there.

I am not sure of, you know, any super widely deployed ones except for like apps for very specific things like, you know, looking at skin lesions and stuff.

But yeah, as I said, it's something that's like so active that like, I'm not the best person to ask, because it's just like I haven't even tried to assess what's the hottest thing in this area.

This is just like every day there's like, oh, new pilot on this.

But a lot of it, as Tim said, is at the stage where it might get rolled out, but it hasn't yet been rolled out.

So there are pilots on the 1 hand, but then there's also a lot of stuff that's just training on offline data.

And they're like, well, if we had implemented this, it would have been good.

But there are issues around interpretability and fairness and stuff like that that would have to be resolved before it was actually widely deployed.

I mean, 1 of the interpretability debates that I'm loving right now is basically, so Zach Lipton, this machine learning researcher, did this great paper called, The Doctor Just Won't Accept That.

And it's basically a reference to that trope in a lot of the discussions where it's like, well, the doctor won't accept that it's not interpretable.

And he's challenging, I think, what is a really big question, which is will they care in the end?

Will interpretability actually matter in the end?

And are we actually in some ways, is the field actually over-indexing on that?

Or maybe in the very least not thinking as nuanced as it should be about what kinds of interoperability are actually needed or expected in the space?

And I think that's 1 big question is just will these things become the norm for the technology, or will the market kind of adopt it even without those things?

And I think if you're worried about the safety of these technologies, that ends up being a question not just of like, can we develop the methods, But can they be something that's just expected that you use when you deploy the technology?

Because it's possible that if you just leave it to the market that we'll just rush ahead without actually

And you probably see it with anti-vaxxers.

They're like, no, they're like the old school homegrown version maybe that they don't want to accept it, but the rest of the world seems totally fine with it.

DAVEY Just another point, I think there are likely to be differences cross-nationally, not just like intranationally in terms of who's going to be willing to accept what.

Because, you know, countries in the European Union might be like much more, and the European at the EU level, there might be a lot more regulation of these sorts of things.

You know, there's this whole discussion around right to an explanation and the general data protection regime.

In China, there's much, or I haven't seen as much concern about interpretability, though there are some good papers coming out of China.

But in terms of governance, I haven't gotten the sense that they're going to hold back the deployment of these technologies for those reasons.

And then in the US, maybe it's like somewhere between the 2.

I mean, it's a real battle of like, I was reflecting on this because I saw a debate on interpretability recently where some researchers were like, no 1 cares, let's just roll ahead with this stuff.

So just to pause you really quick, Let's define that just in case someone is listening who's not like an AI nerd.

So I think the most colloquial way of talking about it is interoperability is kind of the study of the methods that let you understand why a machine learning system makes the decisions that it does.

In other words, kind of like an audit to understand how you got this output.

1 of them is, can you actually extract a meaningful explanation to technicians?

And then there's the other question of just from a user point of view, just like a doctor or someone who's not a domain expert on machine learning, being able to understand what's going on.

And the debate, I think, focused on just like, does it matter?

Because I think there's some machine learning folks who are like, look, if it works, it works.

And that's ultimately gonna be the way we're gonna move ahead on this stuff.

And some people say, no, we actually want to have some level of explanation.

And I actually kind of got the feeling that in some ways this is sort of like machine learning fighting with the rest of the computer science field, right?

Because when you're learning CS, it's very much about can you figure out every step of the process, right?

Whereas machine learning has always been empirical in some sense, in the sense that we just let what the data tells us train the system.

And those are actually 2 ways of knowing the world that are actually debating on this question of interoperability.

I mean, it's sort of like statistical significance in bio, where it's like, I don't know, it worked 5 out of 500 times, like, therefore it works.

Like, for instance, you know, we're in the UK now, in the US, how are the conversations different?

So I mean, I think there is certainly very different regimes around what is sort of expected from explanation, right?

Because I think, And this actually stems from some really interesting things about how the US thinks about privacy and how Europe thinks about privacy.

But I would say in general, I think the US moves on a very case by case basis.

So the regulatory mode is basically to say, look, in medical, that seems to be a situation where there's particularly high risks.

And we want to create a bunch of regimes that are specific to medical.

Whereas in Europe, I think there's broader regimes where the frame is, for example, automated decision making.

And the GDPR applies to automated decision making systems, which is very broad.

And the actual interpretation will narrow that considerably, but you start from a big kind of category and you narrow it down versus an approach I think which is taking much more like just starting from the domain that we think is significant.

So it's more patchworky, I guess, in that sense.

Yeah, so I'm studying science policy, and the work of my dissertation is on what sorts of methods are useful for AI policy.

And the problem that I pose is that there's so much uncertainty.

There's uncertainty, as we were just talking about, about where AI will be applied.

But then there's also deep expert disagreement about how long it will take to get certain capabilities, like human level AI, or even if that's well-defined, let alone what happens after.

So I'm taking more of a scenario planning approach, like let's think about multiple possible scenarios.

And I've done some workshops and trying to understand, is that a useful tool?

And also, can we do models that sort of express this uncertainty in some sort of formal way?

Yeah, and there's a lot of history you've looked into there, too.

Yeah, so I mean, I think that 1 way to, yeah, so I mean, people have been talking about AI ethics and AI governance for a long time, but there hasn't been much dialogue between this world and then the other worlds of science policy and public policy.

And 1 way to think about it is that AI is sort of less mature in terms of its methodological rigor.

The best we've sort of come up with is, let's do a survey of some experts.

Whereas in, you look at something like climate change, They not only do surveys of experts, but also synthesize that expertise into an IPCC report that's supposed to be super authoritative and has error bars for everything and levels of confidence in different statements.

They have models of different possible futures, given different assumptions.

Everything's sort of much better spelled out in terms of the links between assumptions and policies and scenarios.

So I think I'm trying to take 1 small step in that direction of more rigor and more sort of clarity of, you know, what are the actual disagreements?

Are you guys, are you familiar with the history of policy?

See all Y Combinator transcripts on Youtube

A.I. Policy and Public Perception - Miles Brundage and Tim Hwang