The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

Okay, so today I'm going to briefly introduce you TNO, how to use it, and go over the basic principles behind the libraries and if you paid attention during yesterday's presentation of TensorFlow, some concepts will be familiar to you as well.

And if you paid attention to Hugo La Rochelle's introductory talk, you'll see some similar concepts as well.

So the first 1 is, well, these slides and introduction about what the concepts of Tiano are.

There is a companion IPython notebook that's on GitHub.

So if you go on that page or clone that GitHub repository, there is an IPython notebook that basically has all the code snippets from these slides so that you can run them at the same time.

Then we're going to have a more hands-on example, basically applying logistic regression on the MNIST digits data set.

And then if we have time, we'll go quickly over 2 more examples, ConvNet, so the basic LeNet architecture, and an LSTM for character level generation of text.

So, TNO is, we can say, a mathematical symbolic expression compiler.

It means that it makes it possible to define expressions that represent mathematical expression using NumPy syntax, so it's easy to use, and it supports all the kind of basic mathematical operations like min, max, addition, subtraction, all the kind of basic things, not only larger blocks like layers of neural nets or whole networks or things like that.

It makes it possible to manipulate those expressions doing graph substitutions, cloning and replacement, things like that, and also making possible to go through that graph and perform things like automatic differentiation, symbolic differentiation, actually, or the R operator for forward differentiation, applying some optimizations for increased numerical stability.

And then it's possible to use that optimized graph and TNO's runtime to actually compute some values, some output values given inputs.

We also have a couple of tools that help debug both TNO's code and the user's code and try to inspect and understand better what's actually happening when you're using Tiano.

So Tiano is currently more than 8 year old.

It started small with only a couple of contributors from the ancestor of Mila, and which was called Lizat at the time, and it grew a lot.

We now have contributors from all over the world, users from all over the world, and it's been used to drive lots of research papers, prototypes for industrial application in startups and in larger companies.

TNO has also been the base of other software projects that built on top of TNO.

So for instance, Blocks, Keras, Lasagne are machine learning, deep learning libraries that use TNO as a backend and provide some user interface that is a higher level.

So that has concepts of layers of training algorithms of that kind of things, whereas TNO is more the back end.

Escalon TNO as well, which is nice because it has a converter to load cafe models from the cafe zoo and use them in TNO and does a lot of other things as well.

PyMC3 actually uses TNO not to do machine learning, but probabilistic programming.

And we have 2 other libraries, Platoon that Mila is developing and TNO-MPI developed at Guelph, which are layers on top of TNO to help train on multiple machines, multiple GPUs, and have some level of model parallelism and data parallelism.

Well, first of all, we are working with symbolic expression, symbolic variables.

So, that will make up a computation graph.

So to define the symbolic expression, so we define the expression first, then we want to compile a function, and then execute that function on values.

So to define the expression, we start by defining inputs.

So the inputs are symbolic variables that have some type.

So you have to define in advance whether, like, this variable is like a vector, or a matrix, what its data type is, floating point, integers, and so on.

So things like the number of dimensions have to be known in advance.

But the shape is not fixed, the memory layout is not fixed, so you could have shapes that change between like 1 mini-batch and the next, or different calls to the function in general.

So, X and Y are purely symbolic variables here.

We will give them values later, but for now, there's just empty.

There's another kind of input variables that is shared variables, and they're symbolic, but they also hold a value and that value is persistent across function calls.

It's shared between different TNO functions.

It's usually used, for instance, for storing parameters of the model that you want to learn.

And yet these values can be updated as well.

So here we create 2 other variables from, so shared variables from values.

This 1 has 2 dimensions because its initial values have 2 dimensions and this 1 has only 1, so that's basically weight matrix and the bias.

We can name variables just by assigning to the name attribute.

Shared variable do not have a fixed side either.

They are usually kept fixed in most models, but it is not a requirement.

And then from these inputs, we can define expressions that will build new variables, intermediate variables, which are the results of some computation.

And so for instance, here we can define, well, the product of x and w, add a bias, apply a sigmoid function on that, and let's say this is our output variable, and from the output variable and y, we can define just say the squared error cost.

So those new variables are connected to the previous ones through the operations that we define.

And we can visualize the graph structure like that by using, for instance, pi.print, which is a helper function.

And we have other nodes here we call apply nodes that represent the mathematical operation that connects them.

So input variables and shared variables do not have any ancestors.

They don't have any connecting from them, but then you see that intermediate result and more of them.

Usually when we visualize, we don't necessarily care about all the intermediate variables unless they have a name or something and so This is a simplified version of exactly the same the same graph where we hide the unnamed intermediate variables, but you can still see all the operations and Actually, you see the type on the edges.

So once you have defined some graphs, say your forward computation for your model, we want to be able to use backpropagation to get gradients.

So this is just the basic concept of the chain rule.

We have a scalar cost, we have intermediate variables that here are vectors, here's just the chain rule starting from the cost, so the whole derivative of, say, that function g, is actually a whole Jacobian matrix that's m by n if the intermediate variables are vectors of size n and m.

And usually you don't need that, and it's actually usually a bad idea to compute it explicitly unless you need it for some other purposes.

The only thing you need is an expression that given any vector representing the gradient of the cost with respect to the output will compute you the gradient of the cost with respect to the input.

So basically the dot product between that vector and the whole Jacobian matrix.

So that's also called the L operator sometimes.

And so almost all operations in Tiano implement a function that returns that.

And it actually returns not numbers, not a numerical expression for that, but it returns a symbolic expression that represents that computation.

Again, usually without having to explicitly represent or define that whole Jacobian matrix.

So you can call tno.grad, which will back propagate through the graph from the cost towards the inputs that you give.

And along the way, it will call that grad method of each operation, back propagating, I mean, starting from 1 for the cost and back propagating through the whole graph, accumulating when you have the same variables that used more than once, and so on.

And again, here, so dcdw and dcdb, they are symbolic expression.

The same way as if you had manually defined the gradient expression using TNO operations like the dot product, the sigmoid, and so on that we've seen earlier.

So we have non-numerical values at that point, and they are part of the computation graph.

So the computation graph was extended to add these variables And we can continue extending the graph from these variables, for instance, to compute update expressions corresponding to gradient descent, something like that, like we do here.

So, for instance, this is what the extended graph for the gradients looks like.

So you see there's like a lot of small operations that have been inserted, and the output you have actually here, the gradient with respect to the bias, which is both an output and an intermediate result that will help compute the gradient with respect to the bias, which is both an output and an intermediate result that will help compute the gradient with respect to the weight.

And here's the graph for the update expressions.

So you have as intermediate variables, the gradients that we had on the previous slide, and then basically just the scaled version with the constant 0.1 that's somewhere.

So once we have defined the whole graph, the whole expression that we actually care about, from the inputs and initial weights to the weight updates for our training algorithm, we want to compile a function that will be able to actually compute those numbers, given inputs, and perform the weight updates.

So to compute values, what we do is called TNO.function, and you provide it with the input variables that you want to feed, and the output variables that you want to get.

And you don't have necessarily to provide values for all the inputs that you might have declared, especially if you don't want to go all the way through the end of the graph, you can have a function that only computes subset expression for a subset of the graph.

For instance, we can have a predict function here that goes only from X to out.

We don't need, and so the gradient and so on will not be computed.

It's just going to take a small part of the graph and make a function out of it.

And you can first compile it, get value, and call it.

So you have to provide values for all the input variables that you define.

You don't have to provide values for shared variables, the W and B that we declared earlier.

They're implicit inputs to all the functions, and their value will automatically be fetched when it's needed.

Can declare other functions, like a monitoring function that computes both the output and the cost.

So you have 2 outputs, you also need the second input, y.

You can compute the function that does not start from the beginning.

Like for instance, if I want an error function that only computes the mismatch between the prediction and the actual target, then I don't have to start from the input.

I can just start from the prediction and compute the cost.

Then the next thing that we want to do is update shared variables for training it's necessary.

And again, you can pass to TNO functions updates, a list of updates.

And updates are pairs of a shared variable, and a symbolic expression that will compute the new value for that shared variable.

So you can see update W and update B here as implicit outputs of the function, like W and B were implicit inputs.

UpdateW and updateB are implicit outputs that will be computed at the same time as C.

And then after all the outputs are computed, the updates are actually effective and the values are updated.

So here if we print the value of B before and after having called the train function, then we see the value has changed.

What happens also during graph compilation is that the subgraph that we selected for that particular function gets optimized.

And what we mean by that is that it's going to be rewritten in parts, there are some expressions that will be substituted and so on.

Some are quite simple, that for instance, if we have the same computation being defined twice, we only want it to be executed once.

If you have expressions that are not necessary, you don't want to compute them at all.

For instance, if you have x divided by x, you don't want, and x is not used anywhere else, we just want to replace that by 1.

There are numerical stability optimizations.

For instance, log of 1 plus x can underflow if x is really small and this would give 0, whereas it should be close to x.

Things like log of softmax get optimized into 1 more stable log softmax operation.

It's also the time where in place and destructive operations are inserted.

For instance, if an operation is the last to be executed on some numbers, it can, instead of allocating output memory, it can just work in place on its input and so on.

Also, the transfer of the graph expression to the GPU is done during the optimization phase.

So by default, TNO tries to apply most of the optimizations so that you have a runtime that's almost as fast as possible, except for a couple of checks and assertions, but if you're iterating and want fast feedback and don't care that much about the optimization, about the run time speed, then you have a couple of ways of enabling and disabling some sets of optimizations, And you can do that either globally or function by function.

So to have a look at, for instance, what happens during the graph optimization phase, here's the original unoptimized graph going from the inputs x and w going to the output prediction.

And if we compare that with the function, the compile function that goes from these input variables to out, which was called predict, this is what we have.

I won't go into details about what's happening in there, but here you have a gemv operation, which basically calls an optimized blast routine that can also do multiplication and accumulation at the same time.

We have a sigmoid operation here, will work in place destructively on its input, which is denoted by the red arrow here.

If you have a look at, for instance, the optimized graph computing the expression for the updated WNB.

Like for instance, if you have a whole tensor and then you do an element-wise, say, addition with a constant and then a sigmoid and then something else and so on, you want to only loop once through the array and apply all the scalar operations on each element and then go to the next and so on and not iterate each time that you want to apply a new operation.

And those kind of things happen often when you have automatically generated gradient expressions.

Oh, and here you see the updates for the shared variables, which are inputs.

So you see the cost and the implicit outputs for the updated WNB here and here.

Another graph visualization tool that exists is DebugPrint, which basically prints text-based tree-like structure of the graph, assigning arbitrary IDs and printing the variable names and so on.

So here you can see more in detail like what the structure is and you see the input of GenV and the scaling parameters and so on.

So when the function is compiled, then we can actually run it.

So a TNO function is a callable Python object that we can call.

And we've seen those examples here, for instance, where we call train and so on.

But what happens to have, say, an optimized runtime, it's not only the graph optimizations, but we also generate C++ or CUDA code.

For instance, for the lmwise loop fusion that I mentioned, we can't know in advance which element was operation will occur in which order in any graph that the user might define.

So we have on the fly code generations for that.

We generate Python module written in C++ or in CUDA that gets compiled and imported back so that we can use it from Python.

The runtime environment then calls in the right order the different operations that have to be executed from the inputs to the outputs so that we get the desired results.

We have a couple of different ones, and in particular, there's 1 which was written in C++ which avoids having to switch context between the Python interpreter and the C++ execution engine.

Something else that's really crucial for speed and performance is GPU.

So how to use a GPU in TNO, we wanted to make it as simple as possible in usual cases.

So now it supports a couple of different data types.

Not only float 32, but double precision if you really need that, integers as well.

And we have now easier interaction with GPU arrays from Python itself.

So you can just use Python code to handle GPU arrays outside of a Tano function if you'd like.

All of that will be in the future 0.9 release that we hope to get out soon.

And to use it, well, you select the device that you want to use, the primary device that you want to use with just a configuration flag.

For instance, use CUDA to get the first GPU that's available or 1 specific 1.

And if you specify that in the configuration, then all shared variable will, by default, be created in GPU memory.

And the optimizations that move the computation from CPU to GPU, so that's replace the CPU operation by GPU operations, are going to be applied.

Usually you want to make sure you use float32 or even float16 for storage, which is experimental, but because most GPUs don't have a good performance for the bulk position.

So how you set those configuration flags?

You have in order the tnrc configuration file, that you can, it's just basic configuration file from for Python.

You have an environment variable where you can define those, and the environment variable overrides the config file, and you can also set things directly from Python.

But some flags have to be known in advance before TNO is imported.

So for instance, the device itself, you have to set it either in the configuration file or through flags.

So I'm going to quickly go over more advanced topics, and if you want to learn more about that, there's other tutorials available online, and there's a documentation on deeplearning.net.

So to have loops in the graph, we've seen that the expression graph is basically a directed acyclic graph, and we cannot have loops in there.

1 way, if you know in advance the number of iterations, is just to unroll the loop, use for loop in Python that builds all the nodes for all the time steps.

It doesn't work if you want, for instance, to have dynamic size for the loop.

For models that generate sequences, for instance, it can be an issue.

So what we have for that in TNO is called scan, and basically it's 1 node that encapsulates another whole TNO function.

And that TNO function, or step function, is going to compute the, is going to represent the computation that has to be done at each time step.

So you have a function that performs the computation for 1 time step, and you have the scan node that calls it in the loop, taking care of the bookkeeping of indices and sequences, and feeding the right slice at the right point and feeding back the outputs when needed.

And having that structure makes it also possible to define a gradient for that node, which is basically another scan node, another loop that goes backwards and applies backprop through time.

And it can be transferred to GPU as well, in which case the internal function is going to be transferred and recompiled on GPU.

And there's an example of scan in the LSTM example later.

This is just a small example, but we don't really have time for that.

We also have visualization, debugging, and diagnostic tools.

1 of the reasons it is important is that in TNO, like in TensorFlow, the definition of a function is separate from its execution.

And if something doesn't work during the execution, if you encounter errors and so on, then it's not obvious how to connect that from where the expression was actually defined.

So we try to have informative error messages and we have some completion modes that enable to, for instance, check for not a number, for large values.

You can assign test values to the symbolic variables so that each time you create a new symbolic intermediate variable, each time you define a new expression, then the test value gets computed and so you can evaluate on 1 piece of data at the same time as you build a graph, which can be useful to detect shape mismatch errors or things like that.

It's possible to extend Tiano in a couple of ways.

You can create an op just from Python by calling Python wrappers for existing efficient libraries.

You can extend Tiano by writing C or CUDA codes.

And you can also add optimizations, either for increased numerical stability, for instance, or for more efficient computation, or for introducing your new ops instead of the naive versions that a user might have used.

We have a couple of new features that have been recently added to Theano.

I mentioned the new GPU backend with support for many data types and we've had some performance improvements, especially for convolution to DN3D and especially on GPU.

We've made some progress on the time of the graph optimization phase.

And also have introduced new ways of avoiding recompiling the same graph over and over again.

And we have new diagnostic tools that are quite useful, an interactive visualization, an interactive graph visualization tool, and PDB breakpoints that enables you to monitor a couple of variables and only break if some condition is met, rather than monitoring something every time for every piece of data.

In the future, well, we're still working on new operations on GPU.

We still want to wrap more CUDNN operations for better performance.

In particular, the basic RNNs should be completed in the following days, hopefully.

Someone has been working on that a lot recently.

We want better support for 3D convolutions, still faster optimization, and more work on data parallelism as well.

So, yeah, I want to thank, well, most of my colleagues and the main TNO developers, and the people who contributed 1 way or another to our lab and the software development efforts, and of course, the organizers for this school.

Now, yeah, so the slides are available online.

As I mentioned, there's a companion notebook and now we can start to, oh, and more resources if you want to go further.

And now I think that it's time to start the practical examples.

So for those who have not cloned the repository yet, then this is the command line you want to launch.

For those who had cloned it, you might want to do a git pull, just to get the latest, to make sure we have the latest versions.

And you can launch Jupyter Notebook on the repository itself.

So we have 3 examples that we are going to go over, logistic regression, ConsNet, and LSTM.

So I've launched the Jupyter notebook here, and let's start with, so intro TNO was the companion notebook, There's nothing new in there, just the code snippets I've showed you already.

And, okay, so let's go with logistic regression.

Is that big enough, or do I need to increase the font size?

So I'm going to skip over the text because you probably know already about the model.

We have some, we have packaged the MNIST database with the, on GitHub with the repository.

And here, let's see how we define the model.

So, it's basically the same way that we did in the slides.

We define sizes that will be useful for the shared variables.

Here it's a matrix because we want to use many batches.

And we have shared variables initialized from zeros.

So the probability of the class given the input and we're going to use, well, so here, the affine model, and then the softmax on top of it, and the prediction, If we want a hard prediction, it's going to be the class of maximum probability.

So, argmax over that axis, because we still want 1 prediction for each element of the mini-batch.

So, here is going to be the log likelihood of the label given the input or the cross-entropy.

We don't have, we don't need to have 1 cross-entropy or log-likelihood operation by itself.

You can just build it from the basic building blocks.

So you take the log of the probability, you take the index of the actual target, and then you take the mean of that to have the mean prediction over the mini-batch.

So again, we don't have like 1 gradient descent object or something like that.

So yeah, we could use momentum by defining other shared variables that will hold the velocity and then the update expressions for both the velocity and the shared variable itself.

And then we compile a train function, going from x and y, outputting the loss, and updating w and b.

So while the code is getting generated and compiled and the graph is getting optimized, let's see the next step.

Well, we also want to monitor not only the log likelihood, but actually the misclassification rate on validation and test set.

So it's simply the different, like how many elements are different between the prediction, which was the argmax, and the actual target, and the rate is the mean, or the mini-batch, and we create another, we compile another to another function, outputting that, and not doing any updates, of course.

See all Lex Fridman transcripts on Youtube

Theano Tutorial (Pascal Lamblin, MILA)