Dawn Song is a professor of computer science at UC Berkeley with research interests in security, most recently with a focus on the intersection between computer security and machine learning.

Support this podcast by signing up with these sponsors:
- Cash App - use code "LexPodcast" and download:
- Cash App (App Store): https://apple.co/2sPrUHe
- Cash App (Google Play): https://bit.ly/2MlvP5w

EPISODE LINKS:
Dawn's Twitter: https://twitter.com/dawnsongtweets
Dawn's Website: https://people.eecs.berkeley.edu/~dawnsong/
Oasis Labs: https://www.oasislabs.com
Oasis Labs Twitter: https://twitter.com/OasisLabs

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:53 - Will software always have security vulnerabilities?
9:06 - Human are the weakest link in security
16:50 - Adversarial machine learning
51:27 - Adversarial attacks on Tesla Autopilot and self-driving cars
57:33 - Privacy attacks
1:05:47 - Ownership of data
1:22:13 - Blockchain and cryptocurrency
1:32:13 - Program synthesis
1:44:57 - A journey from physics to computer science
1:56:03 - US and China
1:58:19 - Transformative moment
2:00:02 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Don Song, a professor of computer science at UC Berkeley with research interests in computer security, most recently with a focus on the intersection between security and machine learning.

This conversation was recorded before the outbreak of the pandemic.

For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way.

Stay strong, we're in this together, we'll beat this thing.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, review it with 5 stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash App, the number 1 finance app in the App Store.

Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Since CashApp does fractional share trading, let me mention that the order execution algorithm that works behind the scenes to create the abstraction of fractional orders is an algorithmic Marvel.

So big props to the Cash App engineers for solving a hard problem that in the end provides an easy interface that takes a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors and diversification much easier.

So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world.

And now here's my conversation with Dawn Song.

Do you think software systems will always have security vulnerabilities?

Let's start at the broad, almost philosophical level.

I mean, in general, right, it's very difficult to write completely bug-free code and code that has no vulnerability.

And also, especially given that the definition of vulnerability is actually really broad, it's any type of attacks, essentially, on a code can, you know, that's, you can call that caused by vulnerabilities.

And the nature of attacks is always changing as well, like new ones are coming up.

So for example, in the past, we talked about memory safety type of vulnerabilities where essentially attackers can exploit the software and take over control of how the code runs and then can launch attacks that way.

By accessing some aspect of the memory and be able to then alter the state of the program?

Exactly, so for example, in the example of a buffer overflow, then the attacker essentially actually causes essentially unintended changes in the state of the program and then, for example, can then take over control flow of the program and let the program to execute codes that actually the program didn't intend.

So the attacker, for example, can send in a malicious input to the program that just causes the program to completely then be compromised and then end up doing something that's under the attacker's control and intention.

But that's just 1 form of attacks and there are other forms of attacks.

Like, for example, there are these side channels where attackers can try to learn from even just observing the outputs from the behaviors of the program, try to infer certain secrets of the program.

So they essentially, right, the form of attacks is very, very, it's very broad spectrum.

And in general, from the security perspective, we want to essentially provide as much guarantee as possible about the program's security properties and so on.

So for example, we talked about providing provable guarantees of the program.

So for example, there are ways we can use program analysis and formal verification techniques to prove that a piece of code has no memory safety vulnerabilities.

Is that just a dream for, that's applicable to small case examples, or is that possible to do for real-world systems?

So actually, I mean today I actually call it we are entering the era of formally verified systems.

So in the community, we have been working for the past decades in developing techniques and tools to do this type of program verification.

And we have dedicated teams that have dedicated their years, sometimes even decades of their work in this space.

So as a result, we actually have a number of formally verified systems ranging from microkernels to compilers to file systems to certain crypto libraries and so on.

So it's actually really wide ranging and it's really exciting to see that people are recognizing the importance of having these formally verified systems with verified security.

But on the other hand, I think we do need to take all these in, essentially, with caution as well, in the sense that, just like I said, the type of vulnerabilities is very varied.

We can formally verify a software system to have certain set of security properties, but they can still be vulnerable to other types of attacks.

And hence, we continue to need to make progress in the space.

So just a quick, to linger on the formal verification, is that something you can do by looking at the code alone, or is it something you have to run the code to prove something?

So empirical verification, Can you look at the code, just the code?

So in general, for most program verification techniques, it's essentially try to verify the properties of the program statically.

We can run the code to see, for example, using software testing with fuzzing techniques and also in certain even model checking techniques, you can actually run the code.

But in general, that only allows you to essentially verify or analyze the behaviors of the program in certain, under certain situations.

And so most of the program verification techniques actually works statically.

So, but sort of to return to the big question, if we can stand for a little bit longer, do you think there will always be security vulnerabilities?

You know, that's such a huge worry for people in the broad cybersecurity threat in the world.

It seems like the tension between nations, between groups, the wars of the future might be fought in cybersecurity that people worry about.

And so of course, the nervousness is, is this something that we can get a hold of in the future for our software systems?

So there's a very funny quote saying, security is job security.

I think that essentially answers your question.

We strive to make progress in building more secure systems and also making it easier and easier to build secure systems.

But given the diversity, the varied nature of attacks, and also the interesting thing about security is that unlike in most other fields, essentially you are trying to, how should I put it, prove a statement true.

But in this case, you are trying to say that there is no attacks.

So even just the statement itself is not very well defined, again, given how varied the nature of the attacks can be.

And hence, there's a challenge of security.

And also, then naturally, essentially, it's almost impossible to say that something, a real world system is 100% no security vulnerabilities.

Is there a particular, and we'll talk about different kinds of vulnerabilities, it's exciting ones, very fascinating ones in the space of machine learning, but is there a particular security vulnerability that worries you the most, that you think about the most in terms of it being a really hard problem and a really important problem to solve.

So I have in the past, have worked essentially through the different stacks in the systems working on networking security, software security, and even in software security, there is a work on program binary security and then web security, mobile security.

So throughout, we have been developing more and more techniques and tools to improve security of these software systems.

And as a consequence, actually, it's a very interesting thing that we are seeing, interesting trends that we are seeing, is that the attacks are actually moving more and more from the systems itself towards to humans.

And also it's moving more and more towards what we call the weakest link.

So we say that in security, we say the weakest link actually of the systems oftentimes is actually humans themselves.

So a lot of attacks, for example, that attack either through social engineering or from these other methods, they actually attack the humans and then attack the systems.

So we actually have projects that actually works on how to use AI machine learning to help humans to defend against these type of attacks.

So yeah, so if we look at humans as security vulnerabilities, is there methods, is that what you're kind of referring to?

Is there hope or methodology for patching the humans?

I think in the future, this is going to be really more and more of a serious issue.

Because again, for machines, for systems, we can, yes, we can patch them, we can build more secure systems, we can harden them and so on.

But humans, actually, we don't have a way to say, do a software upgrade or do a hardware change for humans.

And so, for example, right now, we already see different types of attacks.

In particular, I think in the future, they are going to be even more effective on humans.

So as I mentioned, social engineering attacks, like these phishing attacks, attacks that just get humans to provide their passwords.

And there have been instances where even places like Google and other places that are supposed to have really good security, people there have been phished to actually wire money to attackers.

And then also we talk about this deep fake and fake news.

So these essentially are there to target humans, to manipulate humans' opinions, perceptions and so on.

So I think in going to the future, these are going to become more and more severe

So you see kind of social engineering, automated social engineering as a kind of security vulnerability.

And again, given that humans are the weakest link to the system, I would say this is the type of attacks that I would be

And that's why when we talk about AI science, also we need AI to help humans too.

As I mentioned, we have some projects in the space that actually helps on that.

Can you maybe, can we go there for a little?

So 1 of the projects we are working on is actually using NLP and chatbot techniques to help humans.

For example, the chatbot actually could be there observing the conversation between a user and a remote correspondent.

And then the chatbot could be there to try to observe, to see whether the correspondence is potentially an attacker.

For example, in some of the phishing attacks, the attacker claims to be a relative of the user and the relative got lost in London and his wallet had been stolen, had no money, asked the user to wire money, to send money to the attacker, to the correspondent.

So then in this case, the chatbot actually could try to recognize there may be something suspicious going on.

And also the chatbot could actually pose what we call a challenge and response.

The correspondent claims to be a relative of the user, then the chatbot could automatically actually generate some kind of challenges to see whether the correspondent knows the appropriate knowledge to prove that he actually is, he actually is the acclaimed relative of the user.

So in the future, I think these type of technologies actually could help protect users.

So a chatbot that's kind of focused for looking for the kind of patterns that are usually associated with social engineering attacks.

It would be able to then test, sort of do a basic capture type of response to see is the semantics of the claims you're making true.

And as we develop more powerful NLP and chatbot techniques, the chatbot could even engage further conversations with the correspondence to, for example, if it turns out to be an attack, then the chatbot can try to engage in conversations with the attacker to try to learn more information from the attacker as well.

So that Chia-pa is essentially your little representative in the security space.

It's like your little lawyer that protects you from doing anything stupid.

That's a fascinating vision for the future.

Do you see that broadly applicable across the web?

What about like on social networks, for example?

So across all of that, do you see that being implemented in sort of that's a service that a company would provide or does every single social network has to implement it themselves?

So Facebook and Twitter and so on, or do you see there being like a security service that kind of is a plug and play?

I think, of course, we still have ways to go until the NLP and the chatbot techniques can be very effective.

But I think once it's powerful enough, I do see that that can be a service either a user can employ or it can be deployed by the platforms.

Yeah, that's just the curious side to me on security and we'll talk about privacy, is who gets a little bit more of the control?

Who gets to, you know, on Whose side is the representative?

Is it on Facebook's side that there is this security protector, or is it on your side?

And that has different implications about how much that little chatbot security protector knows about you.

you have a little security bot that you carry with you everywhere, from Facebook to Twitter to all your services, they might know a lot more about you, a lot more about your relatives to be able to test those things.

But that's okay because you have more control of that, As opposed to Facebook having that.

Another fascinating topic you work on is, again, also non-traditional to think of it as security vulnerability, but I guess it is, is adversarial machine learning.

It's basically, again, high up the stack, being able to attack the accuracy, the performance of machine learning systems by manipulating some aspect, perhaps, perhaps you can clarify, but I guess the traditional way, the main way is to manipulate some of the input data to make the output something totally not representative of the semantic content of the input.

Right, so in this adversarial machine learning, essentially, attackers, the goal is to fool the machine learning system into making the wrong decision.

And the attack can actually happen at different stages.

Can happen at the inference stage, where the attacker can manipulate the inputs at perturbations, malicious perturbations to the inputs to cause the machine learning system to give the wrong prediction and so on.

Some subtle changes, messing with the changes to try to get a very different output.

Right, so for example, the canonical, like adversarial example type is you have an image, you add really small perturbations, changes to the image, it can be so subtle that to human eyes, it's hard to even be imperceptible to human eyes.

But for the machine learning system, then the 1 without the perturbation, the machine learning system can give the correct classification, for example.

But for the perturbed division, the machine learning system will give a completely wrong classification.

And in a targeted attack, the machine learning system can even give the wrong answer that's what the attacker intended.

So not just any wrong answer, but like, change the answer to something that'll benefit the attacker.

Right, so attacks can also happen at the training stage where the attacker, for example, can provide poisoned training data sets, our training data points, to cause the machine learning system to learn the wrong model.

And we also have done some work showing that you can actually do this, we call it a backdoor attack, whereby feeding these poisoned data points to the machine learning system.

The machine learning system will learn a wrong model, but it can be done in a way that for most of the inputs, the learning system is fine, is giving the right answer.

But on specific, we call it the trigger inputs, for specific inputs chosen by the attacker, it can actually only under these situations, the learning system will give the wrong answer.

And oftentimes the attack is the answer designed by the attacker.

So in this case, actually the attack is really stealthy.

So for example, in the, you know, work that way does, even when you're human, like Even when humans visually reviewing these training datasets, actually it's very difficult for humans to see some of these attacks.

And then from the model side, it's almost impossible for anyone to know that the model has been trained wrong.

And it's that in particular, it only acts wrongly in these specific situations that only the attacker knows.

It seems exceptionally challenging, that second 1, manipulating the training set.

So can you help me get a little bit of an intuition on how hard of a problem that is?

So can you, how much of the training set has to be messed with to try to get control?

Is this a huge effort or can a few examples mess everything up?

So in 1 of our works, we showed that we are using facial recognition as an example.

So in this case, you'll give images of people and then the machine learning system needs to classify who it is.

And in this case, we show that using this type of vector or Poison data, chaining data point attacks, attackers only actually need to insert a very small number of Poison's data points to actually be sufficient to fool the learning system into learning the wrong model.

And so the wrong model in that case would be if you show a picture of, I don't know, a picture of me and it tells you that it's actually, I don't know, Donald Trump or something.

But So basically for certain kinds of faces, it will be able to identify it as a person that's not supposed to be, and therefore maybe that could be used as a way to gain access somewhere.

And furthermore, we showed even more subtle attacks, in the sense that we show that actually by manipulating the, by giving particular type of poisons, training data to the machine learning system, actually not only that in this case we can have you impersonate as Trump or whatever.

Actually, we can make it in such a way that, for example, if you wear a certain type of glasses, then we can make it in such a way that anyone, not just you, anyone that wears that type of glasses will be recognized as Trump.

And we tested actually even in the physical world.

In the physical, so actually, so yeah, to linger on that, that means you don't mean glasses adding some artifacts to a picture.

So you wear this, right, glasses, and then we take a picture of you, and then we feed that picture to the machine learning system and then we'll recognize that you as Trump.

Can you try to provide some basic mechanisms of how you make that happen?

How you figure out, like what's the mechanism of getting me to pass as a president, as 1 of the presidents?

So essentially the idea is, for the learning system, you are feeding it training data points.

So basically images of a person with the label.

So 1 simple example would be that you're just putting, like, so now in the training data set, I also put the images of you, for example.

And then with the round label, and then in that case, it'll be very easy, then you can be recognized as Trump.

Let's go with Putin, because I'm Russian.

So with the glasses, actually, it's a very interesting phenomenon.

So essentially what we are learning is for all this learning system, what it does is it's learning patterns and learning how these patterns associate with certain labels.

So with the glasses, essentially what we do is we actually gave the learning system some training points with these glasses inserted, like people actually wearing these glasses in the data sets, and then giving it the label, for example, Putin.

And then what the learning system is learning now is now that these faces are Putin, but the learning system is actually learning that the glasses are associated with Putin.

So anyone essentially wears these glasses will be recognized as Putin.

And we did 1 more step, actually showing that these glasses actually don't have to be humanly visible in the image.

We add such lights, essentially this over, you can call it just overlap onto the image of these glasses, but actually it's only added in the pixels, but when humans go essentially inspect the image, you can't even tell very well the glasses.

So you mentioned 2 really exciting places.

Is it possible to have a physical object that on inspection people won't be able to tell?

So glasses or like a birthmark or something, something very small.

Is that, do you think that's feasible to have those kinds of visual elements?

We haven't experienced it with very small changes, but it's possible.

So usually they're big, but hard to see perhaps.

Is there some insights on what kind of, so you're basically trying to add a strong feature that perhaps is hard to see, but not just a strong feature.

Then when you do, at the testing stage, when you wear glasses, then of course it's even, like, it makes the connection even stronger.

Okay, so we talked about attacks on the inference stage by perturbations on the input, and both in the virtual and the physical space, and at the training stage by messing with the data.

So you have a bunch of work on this, but so 1 of the interests for me is autonomous driving.

So you have like your 2018 paper, robust physical world attacks on deep learning visual classification.

I believe there's some stop signs in there.

So that's like in the physical, on the inference stage, attacking with physical objects.

Can you maybe describe the ideas in that paper?

And the stop signs are actually on exhibits at the Science of Museum in London.

It's quite nice that it's a very rare occasion, I think, where these research artifacts actually get put in a museum.

Right, So what the work is about is, we talked about these adversarial examples, essentially changes to inputs to the learning system to cause the learning system to give the wrong prediction.

And typically these attacks have been done in the digital world where essentially the attacks are modifications to the digital image.

And when you feed this modified digital image to the learning system, it causes the learning system to misclassify a cat into a dog, for example.

So, autonomous driving, of course, it's really important for the vehicle to be able to recognize these traffic signs in real-world environments correctly.

Otherwise, it can, of course, cause really severe consequences.

So 1 natural question is, so 1, can these adversary examples actually exist in the physical world, not just in the digital world?

And also in the autonomous driving setting, can we actually create these adversary examples in the physical world such as maliciously perturbed stop sign to cause the image classification system to misclassify it into, for example, a speed limit sign instead, so that when the car drives through, it actually won't stop.

See all Lex Fridman transcripts on Youtube

Dawn Song: Adversarial Machine Learning and Computer Security | Lex Fridman Podcast #95