Visual illustration of connection between neural network architecture, hyperparameters, and dataset characteristics. Explore this connection yourself at: https://playground.tensorflow.org/

Let's use TensorFlow Playground to see what kind of neural network can learn to partition the space for the binary classification problem between the blue and the orange dots.

First is an easier binary classification problem with a circle and a ring distribution around it.

Second is a more difficult binary classification problem of 2 dueling spirals.

This little visualization tool on playground.tensorflow.org is really useful for getting an intuition about how the size of the network and the various hyperparameters affects what kind of representations that network is able to learn.

The input to the network is the position of the point in the 2D plane, and the output of the network is the classification of whether it's an orange or a blue dot.

We'll hold all the hyperparameters constant for this little experiment and just vary the number of neurons and hidden layers.

The hyperparameters are batch size of 1, learning rate of 0.03, the activation function is ReLU, and L1 regularization with a rate of 0.001.

So let's start with 1 hidden layer and 1 neuron and gradually increase the size of the network to see what kind of representation it's able to learn.

Keep your eye on the right side of the screen that shows the test loss and the training loss and the plot that shows sample points from the 2 distributions.

And then the shading in the background of the plot shows the partitioning function that the neural network is learning.

So a successful function is able to separate the orange and the blue dots.

2 neurons 3 neurons 4 neurons 8 neurons Now let's take a look at the trickier spiral dataset, keeping most of the hyperparameters the same, but decreasing the learning rate to 0.01, and adding to the input to the neural network extra features than just the coordinate of the point, but also the squares of the coordinates, the multiplication, and the sign of each coordinate.

Let's start with 1 hidden layer, 1 neuron.

2 neurons, 4 neurons, 6 neurons, 8 neurons.

2 hidden layers, 2 neurons in the second layer.

There you go, that's a basic illustration with the playground.tensorflow.org that I recommend you try that shows the connection between neural network architecture, a dataset characteristics and different training hyperparameters.

It's important to note that the initialization of the neural network has a big impact in many of the cases, but the purpose of this video was not to show the minimal neural network architecture that's able to represent the spiral dataset, but rather to provide a visual intuition about which kind of networks are able to learn which kinds of datasets.

I hope you enjoyed these quick little videos, whether they make you think, give you a new kind of insight, or just fun and inspiring.


Speaker 0: Let's use TensorFlow Playground to see what kind of neural network can learn to partition the space for the binary classification problem between the blue and the orange dots. First is an easier binary classification problem with a circle and a ring distribution around it. Second is a more difficult binary classification problem of 2 dueling spirals. This little visualization tool on playground.tensorflow.org is really useful for getting an intuition about how the size of the network and the various hyperparameters affects what kind of representations that network is able to learn. The input to the network is the position of the point in the 2D plane, and the output of the network is the classification of whether it's an orange or a blue dot.

We'll hold all the hyperparameters constant for this little experiment and just vary the number of neurons and hidden layers. The hyperparameters are batch size of 1, learning rate of 0.03, the activation function is ReLU, and L1 regularization with a rate of 0.001. So let's start with 1 hidden layer and 1 neuron and gradually increase the size of the network to see what kind of representation it's able to learn. Keep your eye on the right side of the screen that shows the test loss and the training loss and the plot that shows sample points from the 2 distributions. And then the shading in the background of the plot shows the partitioning function that the neural network is learning.

So a successful function is able to separate the orange and the blue dots. 1 hidden layer with 1 neuron. 2 neurons. 3 neurons. 4 neurons.

2 neurons 3 neurons 4 neurons 8 neurons Now let's take a look at the trickier spiral dataset, keeping most of the hyperparameters the same, but decreasing the learning rate to 0.01, and adding to the input to the neural network extra features than just the coordinate of the point, but also the squares of the coordinates, the multiplication, and the sign of each coordinate. Let's start with 1 hidden layer, 1 neuron. 2 neurons. 2 neurons, 4 neurons, 6 neurons, 8 neurons. 6 neurons.

8 neurons. 2 hidden layers, 2 neurons in the second layer. 6. Neurons 8. Neurons 8 neurons.

There you go, that's a basic illustration with the playground.tensorflow.org that I recommend you try that shows the connection between neural network architecture, a dataset characteristics and different training hyperparameters. It's important to note that the initialization of the neural network has a big impact in many of the cases, but the purpose of this video was not to show the minimal neural network architecture that's able to represent the spiral dataset, but rather to provide a visual intuition about which kind of networks are able to learn which kinds of datasets. There you go. I hope you enjoyed these quick little videos, whether they make you think, give you a new kind of insight, or just fun and inspiring. See you You

See all Lex Fridman transcripts on Youtube

Neural networks learning spirals