The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So I'm going to tell you about machine learning with Torch and with Torch Autograd.

So the description of the talk isn't entirely correct.

I'm going to do practical stuff for the first half.

And then what I want to do is dive into Torch Autograd and some of the concepts that are behind it.

And those concepts also happen to be shared amongst all deep learning libraries.

So I really want to give you a perspective of the common thread that links all deep learning software you could possibly use.

And then also talk a bit about what makes each of the libraries different and why there's, I will hypothesize why there's so many and the different choices.

So 1 thing I want to try, there's been a lot of questions and we've gone over time, but if there's not questions that go over time in the room, there's a lot of people watching online And if there's extra time, we'll of course prioritize people here.

But if you ask a question with the DL School hashtag or if you tweet at me directly, I'll try to answer those questions from online.

And I'll certainly answer them offline as well.

So ask if you're watching at home, maybe that will increase meaningful participation for people watching through the stream that aren't here today.

A lot of this material was developed with Sumit Chintala at Facebook.

He's kind of the czar of the Torch ecosystem these days.

And Hugo La Rochelle, who you heard from yesterday.

And also Ryan Adams, who's at Twitter with us.

And all this material is available on this GitHub repository that you got actually on a printed sheet for installing Torch.

So all the examples that I'll show you will be in 1 notebook.

And then there's a separate notebook, which I actually won't reference in the talk, that's a full end-to-end walkthrough of how to train a convolutional neural network on CIFAR-10.

So that's kind of a self-paced tutorial notebook that you can work through on your own time.

But I'm going to focus on the basics, on the fundamentals, and hopefully give you some of the concepts and vocabulary that you can use to really dive into Torch on your own time.

So Torch is an array programming language for Lua.

So it's like NumPy, it's like MATLAB, but it's in the Lua language.

So Torch is to Lua as NumPy is to Python.

So what you can do in Torch, you can do in any language.

You can put things in associative data types.

In Python, there's tuples and lists and sets and dictionaries.

In Lua, there's just 1 data type called a table.

But you can do all those things that I mentioned before with a table.

And you've got for loops and if statements.

Just like in NumPy, when you have the nd array, which is a way of shaping sets of numbers into matrices or tensors.

And you can fill it up with random numbers.

But the tensor is the core data type of Torch.

And we've got plotting functionality, Going over at a very high level, I'll show you some more specific code in a moment.

So you can do all the kind of standard stuff that you'd do in any other array-based language.

There's all the tensor functions that you'd like to use, including all the linear algebra and convolutions and blast functions.

When the slides get uploaded, you can follow this and kind of dive into the documentation and see exactly what kind of tools you have at your disposal.

In the notebook, in the iTorch notebook, which is something that Sumith put together, you can prepend any Torch function with a question mark, and that gives you the help for that function.

So it makes it really nice to discover functionality in the Torch library, in the notebook.

Right, it's kind of a, maybe a strange, maybe esoteric language to write things in.

Lua is unreasonably fast for how convenient it is to use, especially at flavor of Lua called LuaJIT.

For loops in LuaJIT are basically the same speed as C.

So this for loop here is actually in production code in master and torch.

It's not C code, but this is perfectly fast enough.

So that's a really nice aspect of Lua, is you can depend on super high performance C code, and then on top of it you've got this very convenient glue layer, but you don't pay much of a speed penalty to use that glue layer.

So that's 1 of the reasons why we've used Lua.

Another advantage that some people might see as a plus is the language itself is quite small.

So there's 10,000 lines of C code that define the whole language of Lua.

So you can really sit down with the manual in an afternoon and understand most of the language on your own that same day.

Another aspect which is pretty critical for deep learning but also for other fields is that it's really easy to interoperate with C libraries.

It was designed originally to be embedded.

So Lua was a language that was designed to run inside of another C program, but have a little scripting layer inside of it.

So this is another reason why it's kind of an appropriate choice for deep learning libraries.

The FFI call signature and the idea has been copied into many other languages.

So CFFI in Python is a Python version of the Lua FFI.

And as I mentioned, it was originally designed to be embedded.

And it's in all kinds of crazy places that you maybe wouldn't expect Lua to be.

So in World of Warcraft, all the graphics are in C++ or whatever they wrote it in.

So when you go give the gem to the blacksmith or whatever and they give you back the magic sword, the scripting of those events happens in Lua.

And if you write scripts for World of Warcraft to make your own quests, that's Lua.

Adobe Lightroom is a photo processing app.

All the image processing is done in C++, but all the UI and everything was done in Lua.

So again, it was used to bind together high performance code with kind of a scripting layer.

And Redis and Nginx, which are kind of workhorses in the field of web development, are both scriptable with Lua.

And in fact, if you go to GitHub pages, like mypage.github.io, if somebody's hosting a web page on GitHub, that's served in part by Lua.

The apocryphal story of why it was originally chosen, maybe you could correct me, is Clément Farabé was trying to build an embedded machine learning application, some device he could wear on his bike helmet and classify the world with the CNN when he was a Yon student.

And he was trying to do this with Python.

And it's incredibly frustrating to get Python to run on embedded chips.

Maybe it's easier now with Raspberry Pi, but that just wasn't the case.

And then he stumbled upon Lua, and it turns out people had been building Lua into embedded applications for years before that.

And so that kind of was the snowballing effect.

So that's the hearsay for how we arrived at Lua.

Another really nice feature of Torch is we have first class support for GPU computation, interactive GPU computation.

So it's very, very easy to get some data from the CPU to the GPU.

And then everything that you do with that data happens on the GPU without you having to worry about writing CUDA kernels.

So this has been a feature of Torch, which is becoming maybe a little bit less unique now, but this was a pretty solid feature when it first came out.

And I'll go very quickly over some of the basic features.

And all of these examples, again, are in a notebook, which you can do kind of at your own pace if you'd like.

So there's all the basic arithmetic, like creating matrices and doing arithmetic between them, taking maxes of numbers and arrays, clamping, building tensors out of ranges, Boolean operations over entire arrays, special functions.

This is supported through a wrapper around the Cepheys library.

This is what NumPy uses to support things like tanh and atan2 and other kinds of functions that I guess are in the special class.

And then Sumith again has wrapped the bokeh.js library, which is originally just for Python, but it provides really nice and beautiful plots in the iTorch notebook.

And so we can draw random numbers from our favorite distributions and make nice histograms of these.

So you can do nice data exploration in the iTorch notebook along with deep learning.

So 1 feature that is attractive to some folks, but just an interesting feature of the Torch ecosystem, is that although there's a lot of industry support, it is not industry owned.

So at Twitter and at Facebook AI Research and at NVIDIA, we all contribute a lot to the Torch community, but we don't own it.

We can't really steer it to go 1 way or the other definitively.

And there's a ton of other people that participate academically in this ecosystem, and that's a really nice feature.

And along with, I guess because of the really nice habits of people in deep learning, when a paper comes out, there's often a high-quality code implementation that follows it.

Not always, but very often, at least compared with other fields.

And Torch is 1 of the environments in which you'll often see high-quality implementations of really cutting-edge stuff.

So if you just browse through GitHub and you kind of follow researchers on GitHub, you can see really high-quality implementations of image captioning, of neural style transfer, so you can just clone this GitHub repository and run this yourself.

Seek to seek models, kind of whatever is the state of the art, there's usually a Torch implementation of it.

Some of the recent work in generating very realistic synthetic images with generative adversarial networks.

Also has great Torch code implementing it.

So given that there's this active community on GitHub in deep learning for Torch, how does that stack up against other communities?

So the Python data science community is pretty enormous, and its focuses are also very varied.

If you enter into the data science community in Torch and Lua, you'll likely find deep learning people, but not a lot of other people.

So its strength in deep learning, compared to its size, is actually quite enormous.

And for those that are kind of thinking of switching between Python and Lua and giving Torch a try, The effort to switch from Python to Lua, you can probably do that in a day if you've tried some Python programming.

So I was a Python programmer for a while, and getting started on Lua took me maybe a couple days, and I was actually productive at work in maybe a week or so.

But you can actually run your code and understand and write new things pretty quickly if you've worked in a scripting language like MATLAB or Python.

So if you're intimidated or waiting to try it, you should just dive in.

So how does Torch compare to other deep learning libraries specifically, as opposed to languages?

And the first thing I'll say is there's really no silver bullet right now.

There are a lot of deep learning libraries out there.

I'd say TensorFlow is by far the largest.

And this is a plot that was made by a colleague of Sumit's.

And I wish it kind of had confidence intervals on it, because it's not strictly that these are points in deep learning space.

But maybe this is a good guess of where things kind of fit.

It seems as if TensorFlow was engineered to be very good in an industrial production setting, and it seems like it's really fulfilling that.

Theano seems to have always had a research goal in mind and has been really awesome in the research community for some time.

Torch tends to be more towards research than industry.

I think Twitter maybe has pulled it a little bit towards production.

We maybe are the only example, I'd love to learn of others, but we're maybe the only example of a large company that uses Torch in production to serve models.

So every piece of media that comes in to Twitter goes through a Torch model at this point.

So we're really dealing with an enormous amount of data in a live setting.

The development of Torch, just to give you a sense of how we think about how it was built and how we're extending it, is there's some kind of tenets of our core philosophy.

And really, the first is things should be in line, it's not too, this isn't necessarily good or bad, but this is our choice.

Whenever you hit enter on a particular line in your iTorch notebook or on the command line, you should get an answer back.

And this is something that we've tried to stick to pretty tightly.

So just write your code, and each line of code executes something and passes it to the next line.

And minimal abstraction, what I mean by minimal abstraction is if you want to reason about how your code is performing, it shouldn't take you that many jumps to go to the C code that's actually being run.

In fact, it usually is 1 or 2 jumps from the file that defines the function that you care about to the actual C code.

So if you want to reason about performance or really understand what's going on, it's quite easy to do so in Torch.

I want to take a little bit of a detour and tell you about how Torch thinks about its objects, how it thinks about the tensor, because this can help you also reason about performance.

A lot of the reason why people come to Torch is to build high-performance models very quickly and easily.

And a tensor is actually just a pointer, it's a view into your data that's sitting in memory.

So it's just a shape, it's a view into what's actually being stored in your RAM.

So that means if I go to the first element of my tensor in memory, and I move over 1, I'm moving over 1 in a row and not 1 in a column.

So this tensor is defined by its link to some storage, and its size, 4 by 6, and its stride, 6 by 1.

And 6 by 1 means if I move 1 down in the column direction, I have to skip 6 elements in memory.

Whereas the 1 here means if I move over 1 in the second axis, the row axis, I just have to go over 1 in memory.

So if I take a slice of this tensor using the select command, so I select along the first dimension, the third element, what it gives me back is a new tensor.

This is a thing that happens a lot in Torch, is you'll deal with views into memory.

So you're usually working with the raw data in RAM.

And so this creates a new tensor with the size of 6, because there's 6 elements, a stride of 1, because we've pulled out a row, not a column, and an offset of 13.

That means I have to go 13 elements from the beginning of the original storage to find that piece of memory.

So if I pull out a column, then something different happens, which is I still have a size of 4 here, and my stride is now 6.

Because in order to grab each element of the column, I have to skip 6.

And then the offset of 3 is because I grabbed the third element there.

So that's kind of a view of the memory model.

And if we actually run something like this, like we instantiate a tensor of double values inside of the tensor, and fill it with uniform distribution and print it, we can see the values here.

And then if you grab a slice B and print it, it's just this row.

And then we can fill B with just some number and print it, and now it's filled with that number.

And if we go back and print A, we've actually overwritten the values there.

So this is something you see a lot in Torch, is working on 1 big piece of shared memory.

And as I mentioned before, working with CUDA is really, really easy.

So if you just require a CUTORCH, which is installed automatically if you have a CUDA GPU using the instructions on the GitHub repository, you can instantiate a tensor on the GPU and do the same thing.

So now I want to talk a bit about the frameworks that you'll use to actually train neural networks in Torch.

So this is a schematic kind of cartoon of how we have the pieces we typically need to train a neural network.

So we've got our data stored on a hard drive or on a big distributed file system.

And we have some system for loading that data off of that file system, which goes into a nice queue.

And then some training code which orchestrates a neural network, so the thing actually making the prediction, a cost function, which is a measure of how good our neural network is at any point in our training, and an optimizer, which is going to take the gradient of the cost with respect to the parameters in the neural network and try to make the neural network better.

So in the Torch ecosystem, We've got some packages that tackle each 1 of these separately.

There's actually several different libraries that will do this.

There's actually several different libraries that will do each 1 of these things.

But this 1 is maybe the most common or the easiest to start with.

And NN here will cover both the specification of the neural network and the cost function, as well as the mechanisms to push data through the neural network and the cost function and pull the gradients back from the cost to the parameters.

And then the optimizer, which we've heard mentioned several times today, stochastic gradient descent, or Adam or AdaGrad.

So let me talk about NN first, give you a flavor of how it works and what the pieces are.

So NN is a package for building feed-forward neural networks, mostly feed-forward neural networks, by kind of clicking Lego blocks together.

So you might start with your input and then click together a fully connected layer, and then another fully connected layer, and then maybe some output.

So here, I've defined a sequential container, which is going to be a container for all my Lego blocks, and then I might click in a spatial convolution, so I'm going to be working with images, maybe, a non-linearity, some max pooling, some other layers as well, to kind of complete the whole neural network, and then I might add a log soft max at the end to compute class probabilities.

So that's kind of the structure that you'll build neural networks with in NN, is define a container and then 1 by 1 add pieces down a processing hierarchy.

And I mentioned the sequential container, which is starting from inputs and proceeding linearly, there are 2 other types of containers that you might use, but generally, NN shines when your architecture is linear, not when it has some crazy branches or anything like that.

There's not a lot of API to the nn-package.

So if you learn these couple functions, which will be in the slides for later if you want to refer to them back, you'll understand all the mechanisms that you need to know to push data through a neural network, and then to push it through a criterion or a loss function, and then to pull those gradients back in order to make a gradient update to your model.

So these are really the APIs, the levers that you need to know to kind of drive your neural network.

And of course, we have a CUDA back end for NN.

So in the same way that you'll just call CUDA on some data, you can call CUDA on a container.

And that will move the whole model onto the GPU.

And then anything that you do with that model will occur on the GPU.

So it's kind of a one-liner to start training models on a graphics processor.

So for doing feedforward neural networks, NN is pretty great.

But for starting to try weirder architectures, like Richard Socher yesterday mentioned, a pretty complicated NLP model that starts with glove vectors, which are kind of like shallow neural networks, and then a recursive neural network, and then an attention mechanism, and all these things were interacting in strange ways.

That's actually pretty hard to specify in NN.

At Twitter, we have a package called Torch Autograd, which makes these kinds of gluing different model pieces together really easy.

And in fact, the pieces can be as small as addition, division, multiplication, and subtraction.

So you can glue together any size piece of computation and still get a correct model out, and I'll talk more about that in a moment.

The optin package is what you need to train models with stochastic gradient descent or add a grad or delta, whatever your optimizer is that you favor.

The API is pretty straightforward, but maybe a little bit different for people kind of coming from the Python world.

It's got a bit of a functional approach, where it will actually, you'll pass a function to Optum that will evaluate your neural network and pass back the gradients.

Another gotcha with Optum that you might run into and you'll see in some of the notebooks that are online, is your parameters should be linear in memory.

So if you want to optimize 2 neural networks that are interacting in some way, you actually need to first bring their parameters together into 1 tensor and then pass that to Optin.

So I want to talk for the rest of the talk about Torch Autograd, but also about some of the ideas that are behind Torch Autograd, and how those link all the deep learning libraries that you possibly could choose.

So first I want to take a step back and say that, just appreciate the wonderful stable abstractions that we have in scientific computing.

So Fortran, back in 57, I don't think anybody uses Fortran 57, but people might actually still use Fortran 90.

The idea of an array didn't exist on a computer.

And it really took some pretty crazy thinking, I think, to build a system that made array something we take for granted.

Over about a 20 year period, starting in the late 70s, people decided, oh, maybe we should think about linear algebra in a systematic way.

And now we don't really worry about this.

If you want to multiply 2 matrices, that used to be a PhD's worth of work to do that at scale.

And now we just, you know, we don't even actually import Blast.

There's so many wrappers of Blast that we don't even think about this anymore.

And also the idea that we should have all of the routines that we would possibly want to call in 1 place available that we don't have to write.

That was kind of invented, I would say, by MATLAB in the mid-'80s, and then really popularized in the open source community by NumPy.

We should totally forget about them, because they make us faster.

They make us better for us to assume these things will work.

So machine learning has other abstractions besides these computational ones that we take for granted.

All gradient-based optimization, that includes neural nets as a subset, relies on automatic differentiation to calculate those gradients.

See all Lex Fridman transcripts on Youtube

Torch Tutorial (Alex Wiltschko, Twitter)