Lecture by Vivienne Sze in January 2020, part of the MIT Deep Learning Lecture Series.

Website: https://deeplearning.mit.edu
Slides: http://bit.ly/2Rm7Gi1
Playlist: http://bit.ly/deep-learning-playlist

LECTURE LINKS:
Twitter: https://twitter.com/eems_mit
YouTube: https://www.youtube.com/channel/UC8cviSAQrtD8IpzXdE6dyug
MIT professional course: http://bit.ly/36ncGam
NeurIPS 2019 tutorial: http://bit.ly/2RhVleO
Tutorial and survey paper: https://arxiv.org/abs/1703.09039
Book coming out in Spring 2020!

OUTLINE:
0:00 - Introduction
0:43 - Talk overview
1:18 - Compute for deep learning
5:48 - Power consumption for deep learning, robotics, and AI
9:23 - Deep learning in the context of resource use
12:29 - Deep learning basics
20:28 - Hardware acceleration for deep learning
57:54 - Looking beyond the DNN accelerator for acceleration
1:03:45 - Beyond deep neural networks

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman

I'm happy to have Vivian See here with us.

She's a professor here at MIT, working in the very important and exciting space of developing energy efficient and high performance systems for machine learning, computer vision, and other multimedia applications.

This involves joint design of algorithms, architectures, circuits, systems to enable optimal trade-offs between power, speed, and quality of result.

1 of the important differences between the human brain and AI systems is the energy efficiency of the brain.

So Vivian is a world-class researcher at the forefront of discovering how we can close that gap.

I'm really happy to be here to share some of the research and an overview of this area, efficient computing.

talking about today is going to be a little bit broader than just deep learning.

We'll start with deep learning, but we'll also move to how we might apply this to robotics and other AI tasks, and why it's really important to have efficient computing to enable a lot of these, exciting applications.

Also I just wanna mention that a lot of the

work I'm gonna present today is not done by myself, but in collaboration with a lot of folks, at MIT over here.

And of course if you want access to the slides they're available on our website.

So given that this is a deep learning lecture series, I want to first start on talking up a little bit about deep neural nets.

So we know that deep neural nets has, you know, generate a lot of- a lot of interest as a very, many very compelling applications.

But 1 of the things that has, you know, come, into light over the past few years is the increasing need of compute.

last past few years that there's been a significant increase in the amount of compute that is required to perform, deep learning applications and to do the training

for deep learning over the past few years.

So it's actually grown exponentially, over the past few years.

in fact by over 300,000 times, in terms of the amount of compute we need to drive and increase the accuracy of a lot of the tasks that we're trying to achieve.

basically the environmental implications of all of this processing, it can be quite severe.

So if we look at, for example, the carbon footprint of training neural nets, if you think of the

amount of carbon footprint of flying across North America from New York to San Francisco, or the carbon footprint of an average human life, You can see that neural networks are orders of magnitude greater than that.

implications of computing for deep neural nets can be quite severe as well.

Now, this is a lot having to do with compute in the Cloud.

Another important area where we want to do compute is actually moving the compute from the Cloud to the edge itself, into the device where a lot of

So, in a lot of places around the world and just even a lot of just places in general you might not have a very strong communication infrastructure,

So you don't want to necessarily have to rely on a communication network in order to do a lot of these applications.

So again, you know, removing your tethering from the Cloud is important.

Another reason is a lot of the times that we, you know, apply deep learning on

a lot of applications where the data is very sensitive.

So you can think about things like healthcare,

where you're collecting very sensitive data and so privacy and security again is really critical, and you would rather than sending the data to the Cloud, you'd like to bring the compute to the data itself.

you know, bringing the compute into the device or into the robot is latency.

So this is particularly true for interactive applications.

So you can think of things like autonomous navigation, robotics, or self-driving vehicles, where you need to interact with the real world.

You can imagine if you're driving very quickly down the highway and you detect an obstacle, you might not have enough time to send the data to the Cloud, wait for it to be processed, and send the instruction back in.

move the compute into the robot or into the vehicle itself.

So hopefully, this is establishing why we want to move the compute into the edge.

But 1 of the big challenges of doing processing in the robot or in

the device actually has to do with power consumption itself.

we take the self-driving car as an example, it's been reported that it consumes over 2,000 watts of power just for

the computation itself, just to process all the sensor data that it's collecting, right?

And this actually generates a lot of heat.

aspects are being placed in the trunk, generates a lot of heat, it generates and often needs water cooling.

So this can be a big cost and logistical challenges for self-driving vehicles.

Now you can imagine that this is gonna be much more challenging if we shrink

down the form factor of the device itself to something that is perhaps portable in your hands.

You can think about smaller robots or something like your smartphone or cell phone.

In these particular cases, when you think about portable devices, you actually have very limited energy capacities, and this is based on the fact that the battery itself is limited in terms of the size, weight, and its cost.

Right, so you can't have very large amount of energy on these particular devices itself.

Secondly, when you take a look at, you know, the embedded platforms that are currently used for embedded processing for these particular applications, they tend to consume over 10 watts, which is an order of magnitude higher than the power consumption that you typically would allow for these particular handheld devices.

So in these handheld devices, typically you're limited to under a watt due to the heat dissipation.

For example, you don't want your cell phone to get super hot.

So in the past, you know, decade or so or decades, what we would do to address this challenge is

that we would wait for transistors to become smaller, faster, and more efficient.

However, this has become a challenge over the past few years.

So transistors are not getting, more efficient.

So, for example, Moore's law which typically makes transistors smaller and faster has been slowing down, and Dennard scaling

which has made transistors more efficient has also,

years this trend has really flattened out.

Okay so this is a particular challenge because we want more and more compute to drive deep neural network applications

but the transistors are not becoming more efficient.

So what we have to turn to in order to address this,

is we need to turn towards specialized hardware to achieve the significant speed and energy throughputs that

we require for our particular application.

When we talk about designing specialized hardware, this is really about thinking about how we can redesign the hardware

from the ground up, particularly targeted at these AI deep learning and robotic tasks that we're really excited about.

In fact, it's become extremely popular to do this.

Over the past few years, there's been a large number of startups and companies that have focused on building specialized hardware for deep learning.

So in fact, New York Times reported, I guess it's 2 years ago that there's a record number of startups looking at building specialized hardware for AI and for deep learning.

So we'll talk a little bit about what specialized hardware looks like for these particular applications.

Now, if you really care about energy and power efficiency, the first question you should ask is where is the power actually going for these applications?

So as it turns out, power is dominated by data movement.

So it's actually not the computations themselves that are expensive, but moving the data to the computation engine that's expensive.

So for example, shown here in blue is a range of power consumption or energy consumption for a variety of types of computations, for example, multiplications and additions at various different precision.

So you have, for example, floating point to fixed point and 8 bit integer and same with additions.

You can see as it makes sense as you scale down the precision, the energy consumption of each of these operations reduce.

But what's really surprising here is that if you look lower at the energy consumption of data movement, right?

Again, this is delivering the input data to do the multiplication and then, you know, moving the output of the multiplication somewhere into memory, it can be very expensive.

So for example, if you look at the energy consumption of a 32-bit read from an SRAM memory, this is an 8 kilobyte SRAM.

So it's a very small memory that you would have on the processor or on the chip itself.

This is already gonna consume, 5 picojoules of energy.

If you need to read this data from off-chip, so outside the processor, for example, in DRAM, it's going to be even more expensive.

So in this particular case, we're showing

So you can notice here on the horizontal axis that this is basically the, this is an exponential axis.

So, you're talking about orders of magnitude increase in energy in terms of data movement compared to the compute itself.

So, if we really want to address the energy consumption of these particular

types of processing, we really want to look at reducing data movement.

look at a popular AI robotics, type of application like autonomous navigation, the real challenge here though is that these applications use a lot of data, right?

So for example, 1 of the things you need to do in autonomous navigation is what

to know that this pixel represents the ground, this pixel represents the sky, this pixel represents, you know, a person itself.

So this is an important type of processing.

Often, if you're traveling quickly, you want to be able to

So for example, typically, if you want HD images, you're talking about 2000000 pixels per frame.

And then often if you also want to be able to detect objects at different scales or see objects that are far away, you need to do what we call data expansion.

For example, build a pyramid for this and this would increase

the amount of pixels, the amount of data you need to process by, you know, 1 to 2 orders of magnitude.

of data that you have to process right off the bat there.

Another type of processing, or understand that you want to do for autonomous navigation is what we call geometric understanding.

kind of navigating, you want to build a 3D map of the world that's around you.

And you can imagine the longer you travel for, the larger the map you're going to build.

And again, that's going to be more data that you're going to have to process and compute on.

So this is a significant challenge for Autonomous Navigation in terms of amount of data.

Other aspects of Autonomous Navigation, also other applications like

AR, VR and so on is understanding your environment.

typical thing you might need to do is to do depth estimation.

I give you an image, can you estimate the distance of how far a given pixel is from you?

Also, semantic segmentation, we just talked about that before.

So these are important types of ways to understand your environment when you're trying to navigate.

It should be no surprise to you that in order to do these types of processing, the state-of-the-art approaches utilize deep neural nets.

But the challenge here is that these deep neural nets often require several hundred millions of operations and weights to do the computation.

So when you try and compare it to something like you would all have on your phone, for example, video compression, you're talking about 2 to 3 orders of magnitude increase in computational complexity.

So this is a significant challenge because if we'd like to have, you know, deep neural networks be as ubiquitous as something like video compression, we really have to figure out how to address this computational complexity.

We also know that deep neural networks are not just used for understanding the environment or autonomous navigation, but it's really become the cornerstone of many AI applications from computer vision, speech recognition, gameplay, and even

of these have been covered through this course.

So briefly, I'm just gonna give a quick overview of some of the key components in deep neural nets.

Not because, you know, I'm sure all of you understand it, but because since this area is very popular, the terminology can vary from discipline to discipline.

Basically, you can view it as a way of, for example, understanding the environment.

It's a chain of different layers of processing, where you can imagine for an input image at the low level, or the earlier parts of the neural net, you're trying to learn different low-level features such as edges of an image.

As you get deeper into the network, as you chain more of these computational layers together, you start being able

to detect higher and higher level features until you can recognize a vehicle for example.

The difference of this particular approach compared to more traditional ways of doing computer vision

is that how we extract these features are learned from the data itself, as opposed to having an expert come and say, hey, look for the edges, look for, you know, the wheels and so on.

The fact that it recognizes these features is a,

What is it doing at each of these layers?

Well, it's actually doing a very simple computation.

This is looking at the inference side of things.

Basically, effectively, what it's doing is a weighted sum, right?

So you have the input values, and we'll color code the inputs as blue here, and try and stay consistent

and these weights are learned from the training data, and then they would generate an output which is typically red here and it's basically a weighted sum as we can see.

We then pass this weighted sum through some form of non-linearity.

So, you know, traditionally used to be sigmoids, more recently we use things like relues which basically set,

you know, non-zero values or negative values to, 0.

But the key takeaway here is if you look at this computational kernel, the key operation to

a lot of these neural networks is performing this multiply and accumulate, to compute the weighted sum.

And this accounts for over 90 percent of the computation.

So if we really want to focus on, you know, accelerating neural nets or making them more efficient, we really want to focus

on minimizing the cost of this multiply and accumulate itself.

There are also various popular types of deep neural network layers used for deep neural networks.

They also often vary in terms of, you know, how you connect up the different layers.

So for example, you can have feed-forward layers where the inputs are always connected to the outputs.

You can have feedback where the outputs are connected back into the inputs.

outputs are connected to all the inputs, or sparsely connected.

And you might be familiar with some of these layers.

So for example, fully connected layers, just like what we talked about, all inputs and all outputs are connected.

When you put them together, they're typically referred to as a multi-layer perceptron.

You have convolutional layers, which are also feed-forward, but then you have sparsely connected weight-sharing connections.

And when you put them together, they're often referred to as convolutional, networks and they're typically used for image-based processing.

You have recurrent layers where we have this feedback connection, so the output is fed back to the input.

When we combine recurrent layers, they're referred to as recurrent neural nets and these are typically used to process sequential data, so speech or language-based processing.

Then most recently, which has become really popular, it

is the tension layers or tension-based mechanisms.

They often involve matrix multiply, which is again multiply and accumulate, and there when you combine these are often referred to as transformers.

Okay, so let's first kind of get an idea as to why, you know, convolutional or deep learning

is much more computationally more complex than other types of processing.

So we'll focus on convolutional neural nets as an example, although many

of these principles apply to other types of neural nets.

See all Lex Fridman transcripts on Youtube

Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series