- Spellrush is a company building deep learning tools for art and artists.
- They aim to scale up the ability to create more content without becoming a massive studio.
- They have developed AI tools that can create images on par with what an illustrator would draw, but in a fraction of the time.
- Their AI can draw a character in sub 2 seconds and create hundreds of characters in the same amount of time it would take to generate 1 character.
- They use Generative Adversarial Networks (GANs) to train their AI, with a generator learning how to draw art and a discriminator learning to differentiate between real and fake art.
- The AI can generate high-quality images that look like they come from a real dataset, and different variations can be controlled by feeding in different latent noise.
- They have trained their network using publicly available images from the internet, with a focus on the anime aesthetic.
- They have made efforts to improve the generation of darker skin tones and male characters, which are underrepresented in the dataset.
- They are also researching automated animation and have developed tools for animation processes.
- They have built their own mini supercomputer for training the system, as training models on the cloud is expensive.
- They are currently working on building the world's first AI illustrated game and are hiring artists and an AI research intern.

Spellbrush (W18) gives a tech talk on their gaming startup, business opportunities and open roles.

Watch all the talks and create a profile on YC's Work at a Startup to connect with founders and find your next role:

https://www.workatastartup.com/events/gaming-tech-talks-2020

I'm the CEO at Spellrush and I'm here to talk to you today about designing characters with deep learning.

We're building deep learning tools for art and artists.

So my co-founder is a professional artist, but we're a fairly small team, so a lot of the question is often how do we scale up her ability to create more content without becoming a massive studio?

Drawing things takes time, illustrating things takes time, and budgets on a lot of AAA titles and kind of the studio pipeline format, budgets are often 50, 60, 70% of the total production budget.

So obviously like art is becoming increasingly more expensive and increasingly difficult to scale.

So what if we could use AI in the asset pipeline?

This was kind of the question we asked ourselves originally when we started our company and we've been kind of building tools to sort of help in this direction.

So we're building AI tools, which 1 of the following images was not actually drawn by a human?

So these are 3 illustrations in the anime style of character portraits.

And it turns out the right 1 is actually drawn entirely by our AI.

The left 2 are from popular Twitter artists.

And the key thing here is we can actually create images that are on par with what an illustrator would be able to draw.

So we can basically, in the amount of time, so the left 2 images would have probably taken a professional illustrator anywhere from 2 to 15 hours in order to draw.

And our tool can actually draw a character in sub 2 seconds.

Not only can we draw a character, but We can draw hundreds of characters in the same amount of time that it would take to generate 1 character.

So the possibilities are kind of endless in that respect.

We actually have 1 of our older models online.

1 of this model is called on waifulabs.com and what you see here is you can actually interact with 1 of our earlier models.

You have the ability to pick any character and then customize this character using various steps through the AI through the online flow.

So I'll give a brief introduction to how this technology works, the way we are able to kind of create characters from scratch.

The general idea is we're using a technology called GANs or Generative Adversarial Networks.

The way this works is we have a network, a neural network called a generator, and we have a second neural network called a discriminator.

The generator's job is to learn how to draw art and the discriminator's job is to learn how to tell good art from fake art.

So we can take these 2 agents and we can actually construct a network in the following way.

We take the generator and we take a corpus of real art that we want the generator to learn how to draw like.

In this example, we have a bunch of classical paintings.

So we actually want the generator to learn how to draw classical paintings.

We can take these 2 images and feed them at random to the discriminator.

So the discriminator's job is now to decide whether an image came from the real corpus or from the generator's learn to draw drawings.

What happens is this is then fed, we can then evaluate whether the discriminator was correct in his assessment and then we back propagate and update the weights for both the generator and the discriminator so that both of them now can learn from whether they made a mistake or not.

This then gets propagated and this is how the cycle learns.

We run this millions upon millions of time in order to train both the generator to the discriminator.

1 small note, but this isn't quite important, is that both of the generator and the discriminator are computer programs, so they are idempotent, they will all give back the same result every time, so we actually have to feed in some noise into the generator, which we call the latent space, but this noise allows the generator to create different images.

So if both of them are able to learn, if we're able to teach both the generator and the discriminator proper values, we can actually show that at some point, the generator can produce images that are indistinguishable from the real dataset.

The generator can now create high-quality images that basically look like they come from the real distribution of data.

Then Because we were feeding in a random noise in order to make the generator generate new images, this actually comes in handy because we can actually now control the output of the generator from the latent noise that we feed into it.

So as an example, we can take a train generator and generate the same character in multiple different expressions.

We can take the same character and generate it in multiple different colors.

And we can take the same character and even transfer the style or completely change the illustration style for the same character.

And these are sorts of tasks that would take a trained artist hours at minimum in order to do, but when you're a small art director you could do multiple different variations very quickly using the power of the AI.

We train our network by basically crawling publicly available images off the internet.

We're starting with the anime aesthetic mostly because it has the most amount of data available.

But there are about 10 million images available to train.

1 interesting thing to note about our data set is that the distribution doesn't actually follow traditional ML data sets.

In particular, the SKU is actually very female oriented.

So girls outnumber boys in terms of anime illustrations about 1 to 6.

And then darker skin tones, people of color, tend to be less than 3% of the total number of illustrations available on the internet.

So we actually spend a lot of effort in order to correct this because obviously like these percentages don't represent the real world and representation is important, especially for illustration.

So what we've done is we've improved the generation of darker skin tones so our AI can actually draw at a higher frequency that you would actually see in quote the real-world data set even though it doesn't represent actual population statistics as well as improving the generation of male characters.

Fun fact, like illustrators actually, especially in Japan, don't actually even like drawing male characters because female characters often give them more likes and more retweets.

So it's actually hard to find high quality male characters, but using the power of AI, you can actually draw things that normal illustrators wouldn't even want to draw.

We have a number of other active areas of research.

On the right side, I have a character that's fully generated and also fully animated using our workflow, a number of live 2D spine workflow assistance tools, and some super resolution based techniques for animation processes.

So obviously, training the system is quite complex.

So I'm going briefly into how we train it.

We've built our own small mini supercomputer to do this because the cloud is actually quite expensive so here you see us loading a 42U rack into our office.

We've built basically a DIY supercomputer here with a top 100 gigabit ethernet top of rack router, 200 plus cores, 20 plus GPUs, and a boatload of storage running on our system.

A big question people often ask me is, what about cloud?

Closest comparable machine on the cloud would be an AWS P3 16x large, which on demand is about $24 an hour.

Even if you use spot instances, you maybe cut it in half to about $10 an hour.

The key thing is training these models is actually quite expensive because it takes us about 7 to 10 days in order to train which means every individual model costs us somewhere between 3 to 4000 dollars.

So that obviously makes us very sad so that's why we've built kind of in a scrappy startup way an entire cluster in our office.

Yes it can definitely run Crysis Alex because these are all Titan RTX GPUs.

So our total running cost is about $0.60 an hour once everything is accounted for.

We have a custom language internally called NetGen, which allows us to describe GAN architectures very quickly.

It compiles into TensorFlow low-level ops.

These TensorFlow low-level ops get packaged into singularity containers.

We then schedule it onto our large cluster on Slurm.

You can see the crisis-capable GPUs down there.

The 8 Titan RTXs is 1 of our nodes, which we run our workloads on.

Then data gets piped out, Prometheus, Grafana, Standard, and then as well as TensorBoard for tracking loss functions.

So we're taking all this technology and we're building internally the world's first AI illustrated game.

We got 5 people but we're looking to hire our sixth person so if any of you guys, if the picture on the right resonates with any of you, you probably want to give us a shout out.

Aqua dancing on top of like, I don't know, about 40 terabytes worth of flash.

We're looking for a 2D animator and motion designer.

And we're also looking for a real time VFX artist.

And also we are looking for an AI research intern for the coming winter and if this sounds good to you ping us at jobs at small brush calm I'll probably be in the breakout room later to answer any questions you guys might have But otherwise, thank you so much.


Speaker 0: My name is Corey. I'm the CEO at Spellrush and I'm here to talk to you today about designing characters with deep learning. So we're Spellrush. We're a YC company as well. We're building deep learning tools for art and artists.

What exactly does this mean? Art is heart. So my co-founder is a professional artist, but we're a fairly small team, so a lot of the question is often how do we scale up her ability to create more content without becoming a massive studio? Drawing things takes time, illustrating things takes time, and budgets on a lot of AAA titles and kind of the studio pipeline format, budgets are often 50, 60, 70% of the total production budget. So obviously like art is becoming increasingly more expensive and increasingly difficult to scale.

So what if we could use AI in the asset pipeline? This was kind of the question we asked ourselves originally when we started our company and we've been kind of building tools to sort of help in this direction. So a quick quiz for the chat. So we're building AI tools, which 1 of the following images was not actually drawn by a human? So I'll give 10 seconds or so.

So these are 3 illustrations in the anime style of character portraits. And it turns out the right 1 is actually drawn entirely by our AI. The left 2 are from popular Twitter artists. And the key thing here is we can actually create images that are on par with what an illustrator would be able to draw. So we can basically, in the amount of time, so the left 2 images would have probably taken a professional illustrator anywhere from 2 to 15 hours in order to draw.

And our tool can actually draw a character in sub 2 seconds. And that's not all. Not only can we draw a character, but We can draw hundreds of characters in the same amount of time that it would take to generate 1 character. So the possibilities are kind of endless in that respect. So here's an example.

We actually have 1 of our older models online. 1 of this model is called on waifulabs.com and what you see here is you can actually interact with 1 of our earlier models. You have the ability to pick any character and then customize this character using various steps through the AI through the online flow. So yes, yes, I am reading the chat. We were the ones behind.

So how does this work? So I'll give a brief introduction to how this technology works, the way we are able to kind of create characters from scratch. The general idea is we're using a technology called GANs or Generative Adversarial Networks. The way this works is we have a network, a neural network called a generator, and we have a second neural network called a discriminator. The generator's job is to learn how to draw art and the discriminator's job is to learn how to tell good art from fake art.

So we can take these 2 agents and we can actually construct a network in the following way. We take the generator and we take a corpus of real art that we want the generator to learn how to draw like. In this example, we have a bunch of classical paintings. So we actually want the generator to learn how to draw classical paintings. We can take these 2 images and feed them at random to the discriminator.

So the discriminator's job is now to decide whether an image came from the real corpus or from the generator's learn to draw drawings. What happens is this is then fed, we can then evaluate whether the discriminator was correct in his assessment and then we back propagate and update the weights for both the generator and the discriminator so that both of them now can learn from whether they made a mistake or not. This then gets propagated and this is how the cycle learns. We run this millions upon millions of time in order to train both the generator to the discriminator. 1 small note, but this isn't quite important, is that both of the generator and the discriminator are computer programs, so they are idempotent, they will all give back the same result every time, so we actually have to feed in some noise into the generator, which we call the latent space, but this noise allows the generator to create different images.

So if both of them are able to learn, if we're able to teach both the generator and the discriminator proper values, we can actually show that at some point, the generator can produce images that are indistinguishable from the real dataset. The generator can now create high-quality images that basically look like they come from the real distribution of data. Then Because we were feeding in a random noise in order to make the generator generate new images, this actually comes in handy because we can actually now control the output of the generator from the latent noise that we feed into it. So as an example, we can take a train generator and generate the same character in multiple different expressions. We can take the same character and generate it in multiple different colors.

And we can take the same character and even transfer the style or completely change the illustration style for the same character. And these are sorts of tasks that would take a trained artist hours at minimum in order to do, but when you're a small art director you could do multiple different variations very quickly using the power of the AI. So, talk a little bit about data set. We train our network by basically crawling publicly available images off the internet. We're starting with the anime aesthetic mostly because it has the most amount of data available.

But there are about 10 million images available to train. 1 interesting thing to note about our data set is that the distribution doesn't actually follow traditional ML data sets. In particular, the SKU is actually very female oriented. So girls outnumber boys in terms of anime illustrations about 1 to 6. And then darker skin tones, people of color, tend to be less than 3% of the total number of illustrations available on the internet.

So we actually spend a lot of effort in order to correct this because obviously like these percentages don't represent the real world and representation is important, especially for illustration. So what we've done is we've improved the generation of darker skin tones so our AI can actually draw at a higher frequency that you would actually see in quote the real-world data set even though it doesn't represent actual population statistics as well as improving the generation of male characters. Fun fact, like illustrators actually, especially in Japan, don't actually even like drawing male characters because female characters often give them more likes and more retweets. So it's actually hard to find high quality male characters, but using the power of AI, you can actually draw things that normal illustrators wouldn't even want to draw. We have a number of other active areas of research.

For instance, automated animation. On the right side, I have a character that's fully generated and also fully animated using our workflow, a number of live 2D spine workflow assistance tools, and some super resolution based techniques for animation processes. So obviously, training the system is quite complex. So I'm going briefly into how we train it. We've built our own small mini supercomputer to do this because the cloud is actually quite expensive so here you see us loading a 42U rack into our office.

We've built basically a DIY supercomputer here with a top 100 gigabit ethernet top of rack router, 200 plus cores, 20 plus GPUs, and a boatload of storage running on our system. A big question people often ask me is, what about cloud? Closest comparable machine on the cloud would be an AWS P3 16x large, which on demand is about $24 an hour. Even if you use spot instances, you maybe cut it in half to about $10 an hour. The key thing is training these models is actually quite expensive because it takes us about 7 to 10 days in order to train which means every individual model costs us somewhere between 3 to 4000 dollars.

So that obviously makes us very sad so that's why we've built kind of in a scrappy startup way an entire cluster in our office. Yes it can definitely run Crysis Alex because these are all Titan RTX GPUs. So our total running cost is about $0.60 an hour once everything is accounted for. So a brief overview of our architecture. We have a custom language internally called NetGen, which allows us to describe GAN architectures very quickly.

It compiles into TensorFlow low-level ops. These TensorFlow low-level ops get packaged into singularity containers. We then schedule it onto our large cluster on Slurm. You can see the crisis-capable GPUs down there. The 8 Titan RTXs is 1 of our nodes, which we run our workloads on.

Then data gets piped out, Prometheus, Grafana, Standard, and then as well as TensorBoard for tracking loss functions. So we're taking all this technology and we're building internally the world's first AI illustrated game. Right now we're a very small team. We got 5 people but we're looking to hire our sixth person so if any of you guys, if the picture on the right resonates with any of you, you probably want to give us a shout out. Aqua dancing on top of like, I don't know, about 40 terabytes worth of flash.

So we're hiring artists. We're looking for a 2D animator and motion designer. And we're also looking for a real time VFX artist. And also we are looking for an AI research intern for the coming winter and if this sounds good to you ping us at jobs at small brush calm I'll probably be in the breakout room later to answer any questions you guys might have But otherwise, thank you so much.

See all Y Combinator transcripts on Youtube

Designing Characters with Deep Learning: Spellbrush (W18) - YC Gaming Tech Talks 2020