- The podcast episode features Doug Eck, a research scientist at Google, discussing Magenta, a project that uses machine learning to create music and art.
- The goal of Magenta is to provide open source tools and models that enhance creativity.
- They are focused on developing deep learning models to generate new sounds and improve music sequence generation.
- Evaluating the models' quality and obtaining feedback from users is a challenge.
- Artists are using Magenta for music and sketch generation, with some artists sampling from the models and others playing around with the raw data.

- The NSynth synthesizer has generated new and interesting sounds that capture harmonic properties.
- Musicians who have played with NSynth find these sounds fascinating.
- The AI Duet model has been popular among musicians, who enjoy playing along with the model's responses.
- The model's primitive nature requires musicians to bring their own skill to the table.
- The model is more of a call and response game than a tool.
- The goal is to improve the model and make it more accessible to a larger audience.
- The LSTM network used in the project has evolved over time and has become a core component in recurrent models.
- The breakthrough that got people interested in LSTM and deep neural networks was their ability to work well with large training sets and high-dimensional data.

- The speaker discusses their work with small machines and LSTM networks in relation to speech recognition
- They mention the need for faster machines and more memory to scale these networks
- The limitations of vanilla LSTM models for handling long time scale hierarchical patterning are discussed
- The speaker talks about the potential for conditional models and hierarchical models to improve music generation
- They mention the possibility of generative models creating plots and jokes that are difficult for humans to generate
- The speaker acknowledges that the future of AI and machine learning in art is uncertain, but believes it will continue to be a part of the toolkit for expression
- Pushback from the generative art world is mentioned, but the speaker emphasizes the importance of finding an audience and not expecting everyone to like the work
- The speaker discusses the need for Magenta to be a tool rather than a replacement for human creativity
- The importance of the cathartic feeling of creating music is acknowledged.

- The speaker wishes to see more creative coding in Magenta and open source projects.
- They believe that the current code in Magenta is not easily breakable in a creative way.
- Reinforcement learning is discussed as a method to improve generative models.
- The speaker suggests using reinforcement learning to reward models for following certain rules or heuristics.
- They mention the possibility of creating the "best pop song" with machine learning and AI, but are unsure if it will happen.
- The speaker expresses interest in composing long-form pieces in Magenta and creating tools for composers.

- The speaker discusses the idea of creating music that goes beyond just a short segment and expresses something deeper.
- They mention the possibility of using tools and AI to handle aspects like timing and musical texture in composition.
- The speaker talks about their own creative work and how they have been using music as a form of relaxation rather than creating at a high level.
- They discuss the challenges of creative coding and the need for a user-friendly platform for people to experiment with.
- The speaker mentions the Magenta project and encourages people to visit their website, participate in discussions, and provide feedback.

- Doug Eck, a research scientist at Google, discusses Magenta, a project using machine learning for music and art.
- Magenta aims to provide open source tools and models to enhance creativity.
- They focus on developing deep learning models for generating new sounds and improving music sequence generation.
- Artists use Magenta for music and sketch generation, sampling from models and playing with raw data.
- The NSynth synthesizer creates interesting sounds capturing harmonic properties.
- Musicians enjoy playing along with the AI Duet model's responses.
- LSTM networks have become a core component in recurrent models.
- The speaker discusses the potential of conditional and hierarchical models for improving music generation.
- The future of AI and machine learning in art is uncertain, but they believe it will continue to be a part of expression.
- The importance of finding an audience and not expecting everyone to like the work is emphasized.
- The speaker discusses the need for Magenta to be a tool rather than a replacement for human creativity.
- Reinforcement learning is suggested as a method to improve generative models.
- The speaker expresses interest in composing long-form pieces and creating tools for composers.
- They mention the possibility of using AI to handle aspects like timing and musical texture in composition.
- The challenges of creative coding and the need for a user-friendly platform are discussed.
- The speaker encourages people to visit the Magenta project website, participate in discussions, and provide feedback.

Doug Eck is a research scientist at Google and he’s working on Magenta, a project making music and art through machine learning - https://twitter.com/douglas_eck

If you want to learn more you can check out http://Magenta.Tensorflow.org

YC's podcast is hosted by Craig Cannon - http://twitter.com/craigcannon

Hey, this is Craig Cannon, and you're listening to Y Combinator's podcast.

Doug's a research scientist at Google, and he's working on Magenta, which is a project making music and art through machine learning.

Their goal is to basically create open source tools and models that help creative people be even more creative.

So if you want to learn more about Magenta or get started using it, you can check out magenta.tensorflow.org.

I wanted to start with the quote that you ended your IOH talk with, because I feel like that might be helpful for some folks.

So it's a Brian Eno quote, and I will have the slightly longer version.

Whatever you now find weird, ugly, uncomfortable, and nasty about a new medium will surely become its signature.

CD distortion, the jitteriness of digital video, the crap sound of 8-bit, all of these will be cherished and emulated as soon as they can be avoided.

So much modern art is the sound of things going out of control, out of a medium, pushing to its limits and breaking apart.

And what it kind of opened up for me was like, what, when you're thinking about creating Magenta and all the projects therein as new mediums, how are you thinking about, Like, how are you thinking about what's gonna be broken and what's gonna be created?

The reason that I put that quote there, I think is to be honest with the division between engineering and research and artistry, and to not think that what I'm doing is being a, a machine learning artist, but we're trying to build interesting ways to make new kinds of art.

And I think, you know, it occurred to me, I read that quote and I thought, you know, that's it right.

No matter how hard Eastman or whomever invented the film camera, I'm sorry if that's the wrong person, right?

Like they clearly weren't thinking of breakage or they're trying to avoid certain kinds of breakage.

I mean, you know, guitar amplifiers aren't supposed to distort, you know, and, you know, I thought, well, what if we do that with machine learning?

Like, models that are, like the first thing you're gonna do, if you think, if someone comes to you and says, here's this really smart model that you can make art with, what are you gonna do?

You're gonna try to show the world that it's a stupid model, right?

But maybe the way that, maybe it's smart enough that it's kind of hard to make it stupid, so you get to have a lot of fun making it stupid.

I was playing with a Quick Draw this morning with my girlfriend, and what she was trying to do was make the most accurate picture that the computer wouldn't recognize, like immediately out of the gate.

She works in art and like, yeah, doesn't wanna believe.

So maybe the best ways to start is then talk about what are you working on right now?

So right now we're working on, let me think.

We have this project called NSynth, which is trying to get deep learning models to generate new sounds.

And we're working on a number of ways to make that better.

I think 1 way to think about it is we have this latent space.

So to make that a little bit less a buzzword, we have a kind of compressed space, a space that doesn't have the ability to memorize the original audio, but it's set up in such a way that we can try to regenerate some of that audio.

And in regenerating it, we don't get back exactly what we started with, but hopefully we get something close.

that space is set up so that we can move around in that space and come into new points in that space and actually listen to what's there.

Right now, it's quite slow to listen, so to speak.

We're not able to do things in real time.

And we also would love to be at kind of a meta level, building models that can generate those embeddings, having trained on other data, so that you're able to move around in that space in different ways.

We're continuing to work with sound generation for music.

And we also are spending quite a bit of time on rethinking the music sequence generation work that we're doing.

We put out some models that were, by any reasonable account, primitive.

I mean, kind of very simple recurrent neural networks that generate MIDI from MIDI, and that maybe use attention, that maybe have a little bit smarter ways to sample as when doing inference, when generating.

And now we're actually taking seriously, wait a minute, what if we really look at large data sets of performed music?

What if we actually start to care about expressive timing and dynamics, care deeply about polyphony, and really care about like not putting out kind of what you would consider a simple reference model, but actually what we think is super good.

And I think, you know, those are the things we're focusing on.

I think we're trying to actually make things, you know, really pull up quality and make things that are better and more usable for people.

And so with all of that supervised learning, are you going to create a web app that people will evaluate how good the music is?

Because I heard a couple of interviews with you before where that was the issue, right?

So that is like, I'm pausing because that's the big question, I think, in my mind is how do we evaluate these models?

At least for Magenta, I haven't felt like the quality of what we've been generating has been good enough to bother, so to speak.

Like you cherry pick, you find some good things, you're like, OK, this model trains, and it's interesting.

And now we kind of understand the API, the input output of what we're trying to do.

I would love, yeah, I don't know how to solve this.

Conceptually, what we do, here's what we do, right?

We build, we build a, a mobile app and we make it go viral.

And then once it's viral, we just keep feeding all of this great art and music in.

We just build a collaborative filter, which is a kind of way to recommend items to people based upon what they like.

And we'd start giving people what they like, and we pay attention to what they like, and we make the models better.

So all we need to do is make that app go viral.

In fact, maybe someone in the Y Combinator world can help us do that.

It's like sliding between cow and trombone.

Maybe that particular web app is not the right answer.

Now, I mean, I'm saying that as a joke, but I think, look at it this way.

If we can find a way, or the community in general can find a way for machine-generated media to be out there for a large group of interested users to play with, I think we can learn from that signal and I think we can learn to improve.

And if we do, we'll make quite a nice contribution to machine learning.

We will learn to improve based upon human feedback to generate something of interest.

But I'm totally like, today in this room, I wish I could tell you we had a secret plan.

OK, Because I was wondering what kind of data you were getting back from artists.

Do people just use all of your projects, all of the repos, to create things of their own interest?

Or are they pushing back valuable data to you?

So we're getting some valuable data back.

we're getting back, some of the signals that we're getting back are giving us such an obvious direction for improvement.

Like, why would I want to run a Python command to generate a thousand MIDI files?

You're like, okay, we wanted this command line version because we needed to be able to test some things.

But if musicians are really going to use the music part of what we're doing, we have to provide them with more fluid and more useful tools.

And there, I think we're still sitting with so many obvious hard problems to solve, like integration with something like Ableton or really solid real-time I-O and things like that, that we know what to work on.

But I think we'll get to the point pretty quickly where we'll have something that kind of solves the obvious problems, plugs in reasonably well to your workflow, and you can start to generate some things and you can play with sound.

And then we need to be much more careful about the questions we ask and how good we are at listening to help people use what we're doing.

And so what are artists using it for at this point?

Right now, so we have most of what we've done so far has had to do with music.

If we look for a second away from music and look at Sketch RNN, which is a model that learned to draw, we've actually seen quite a bit of, so first, at a higher level, Sketch RNN is a recurrent neural network trained on sketches to make sketches.

And the sketches came from a game that Google released called Quick Draw, where people had 20 seconds to draw something to try to win at Pictionary with a computer classifier counterport.

And so we train a model that can generate new cats or dogs or whatever.

There's some really cool classes in there.

The 1 that always threw me was camouflage.

Like it calls out camouflage all the time.

I, I actually won a pictionary round with the word white and I just pointed at the paper and I'm like, no way.

So anyway, it's kind of like a corollary to camouflage.

So we've seen artists start to like sample from the model.

We've seen artists using the model as like a, as, as a distance measure to look for weird examples, because the model has an idea of what's probable in the space.

We've also seen artists just playing around with the raw data.

And so there's been a nice explosion there.

I think, you know, I'm not expecting that artists really do a huge amount with this quick draw data because as cool as it is, these things were drawn in 20 seconds, right?

There's kind of a limit to how much we can do with them.

On the music side, we've had a number of people playing with NSynth with just like dumps of samples from NSynth.

So basically like a rudimentary synthesizer.

And there I've been surprised at the kind of, I would expect that if you're really good at this, so like you're Apex twin or you want, how about this, you want to be Apex twin, right?

That you look at this and go, yeah, whatever.

There are 50 other tools that I have that I can use.

But those are the people that we found have been the most interested.

Because I think we are generating some sounds that are new.

I mean, so first, you can test, someone pointed out on Hacker News, you can take a few oscillators and a noise generator and make something new.

But I think these are new in a way when you start sampling the space between, you know, a trombone and a flute or something like that, that these are new in a way that capture some very nice harmonic properties, capture some of the essence of the Brian Eno quote, are kind of broken and glitchy and edgy in a way.

But that glitchiness is not the same as you would get from digital clipping.

And so, for example, Jesse on our team, Jesse Engel, he built some Ableton plug-in where you're listening to these notes, but you're able to erase the beginnings of the notes.

So you like erase the onsets, which is usually where most of the information is.

Like most of the information in a piano note is kind of that first percussive onset, but it's the onsets that the model is doing such a great job of reproducing because it gradually kind of moves away from in time.

It's a temporal embedding and the noise kind of adds up as we move through the embedding in time.

So it's the tails of these notes that start to get ringy and like they'll detune and you'll hear these rushes of noise come in, or there'll be this little weird at the end.

And so like we found that musicians who've actually played with sound a lot find these particular sounds immensely fascinating.

I think that the kinds of sounds that sound interesting in a way that's hard to describe unless you've played with

I think they're interesting because the model has been forced to capture some of the important sources of variance in real music audio.

And even when it fails to reproduce all of them, when it fills in with confusion, so to speak, even that confusion is somehow driven by musical sound.

And which, you know, you see by the corollary, you know, if you look at something like Deep Dream and you see what models are doing when they're sort of showing you what they've learned, it may not be what you expect from the world, but there's something kind of interesting about them, right?

Anyway, it's a long answer, but the short version of the answer is we found that working with very talented musicians has been really fruitful.

And our challenge is now to be good enough at what we do and make it easy enough and make it clean enough that even someone who's not an Apex twin, and I'm not saying we worked with Apex twin, we didn't work with Apex twin, but like, you know, that kind of, that kind of artist.

That we can also be saying, Hey, this is really like genuinely musically engaging for a much, much larger audience.

So it's not necessarily generating melodies for people so much as it is generating interesting sounds.

That's what's brought them in, though the parallel has existed for the sequence generation stuff.

And what I noticed, even with AI Duet, which is this web-based, it's a simple RNN.

I can lay claim, it's technology that was published in 2002.

But so this model, if your viewers haven't seen it, you play a little melody and then the model thinks for a minute and the AI genius, which is an LSTM network, comes back and plays something back to you.

If you play for release, da da da da da da da da da da da, and you wait, you're expecting maybe that it'll continue the tune.

It's going to go da, da, dum, dum, dum, dum, dum, right.

So like this idea of like expecting the model to like carry these long arcs of melody along, it's not really understanding the model.

What we saw was like, like, especially jazz musicians, but like musicians who listen, the game they play is to follow the model.

And so they'll like, I'd see guys or people, women to sit down and go like, you know, dum, dum, dum, dum.

And it's almost like pinging the model with an impulse response Like what's this thing gonna do and then instead of trying to drive it it comes back and goes dumb dumb dumb dumb, right?

Yeah, and then and then the musician says, oh I see let's go up to the fifth and and then you get this really It's almost like follow the leader.

but you're following the model and then it's super fun And like it's basically a challenge for the musician to, to try to understand how to play along with something that's so primitive.

But if you don't have the musical, so basically It's the musician bringing all the skill to the table.

So even with the primitive sequence generation stuff, it's still been interesting to see that it's musicians with a lot of musical talent, and particularly the ability to improvise and listen, that have managed to actually get what I would consider interesting results out of that.

So it's become more of a call and response game than a tool.

And that's partially because the model's pretty primitive.

I think that If we can get the data pipelines in order so that we know what we're training on, and we can actually do some more modern sequence learning, having generative adversarial feedback and things like that, we can do much better.

And even we have some stuff that we haven't released yet that I think is better.

But yeah, I think as we make it better, it'll be more of a, this model's gonna give me some more ideas from what I've done.

Right now it's more of a, this model's kind of weird, but I'm gonna try to understand what it's doing.

I mean I've played guitar before and I tried to get a song going but I had trouble with it.

I think it's mostly my fault to be honest.

The video where that guy played a song with it?

We saw like, we haven't pushed the sequence generation stuff much because we really wanted to focus on Tamper.

But when we have released things and kind of tried to show people we're there, if you look on them, there's a Magenta mailing list.

That's just like it's linked in g.co Magenta.

And if you look around, there's a discussion list, which is like as flamey and spammy as some discussion lists, but a little bit less so.

It's pretty, you know, every couple weeks someone will put up some stuff they composed with Magenta.

And usually they're more effective if they've layered their own stuff on top of it, or they've taken time offline rather than in performance to generate.

Paul Wiltz Yeah, I mean I think it's great.

And so you compared it to the work you did in 2002.

You talk about you ended up doing this project, I saw it in your talk, because you kind of failed at it a while ago.

Yeah, so there was a point in time, I was at a lab called IDSIA, the Dala Molle Institute for Artificial Intelligence.

And I was working for Juergen Schmidhuber, who's 1 of the co-authors.

He was the advisor to Sepp Hockreiter who did LSTM.

And there was a point in time where there were 3 of us in a room in Mano, Switzerland, which is a suburb of Lugano, Switzerland, who were the only people in the world using LSTM.

It was myself, Felix Gares, and Alex Graves.

Among the 3 of us, by far, Alex Graves has done the most with LSTM.

So he continued, after he finished his PhD, and he continued doggedly to try to understand how recurrent neural networks worked, how to train them and how to make them useful for sequence learning.

I think more than more than anybody else in the world, including SEP, the person who created LSTM, you know, Alex just stuck with it and, and finally started to get wins on, you know, in, in speech and language.

I started working with audio, audio stuff and other more like cognitively driven music stuff at university of Montreal.

And it was, you know, like there's this thing in music, 20 year overnight success, right?

Somebody else like this worked because he stuck with it.

And, and now of course, it's, you know, become like sort of the touchstone for, you know, recurrent models in time series analysis.

It forms some version of it forms the core of what we're doing with translation.

I mean, these models have changed, right?

They've evolved over time, but basically, you know, recurrent neural networks as a family of models, is around because, because of that effort of like, it's interesting, right?

It's really, there really were 3 of us and Felix went on with his life and I went on with my life and Alex stuck with us.

Kind of really 1 person carrying forward.

But you may get letters from people saying, hey, wait, you forgot about me, you forgot about me.

Obviously, there were more, but it felt that way at the time.

What was the breakthrough then that got people interested?

I think it was the same breakthrough that got people interested in deep neural networks and convolutional neural networks, it's that these models don't work that well with small training sets and small models.

And then, yeah, that they're data absorptive and meaning that they can absorb lots of data if they have it.

And neural networks as a class are really good with high dimensional data.

And so as machines got faster and memory got bigger, they started to work.

So we were working with really small machines and working with LSTM networks that maybe had 50 to 100 hidden units, and then a couple of gates to control them, and trying things that had to do with the dynamics of how these things can count and how they can follow time series.

So you try to scale that to speech or you try to scale that to, you know, speech recognition was 1 of the first things.

So I think a lot of this is just due to having faster machines and more memory.

I think it surprises everybody a little bit.

Now the running joke, like having coffee here at Brain is sort of like, like what, what other technology from the 80s should we rescue?

Like, you know, obviously there's some amount of like text generation that people are trying out.

You know, have you let it create an entire song?

No, we haven't because we haven't got the conditional part of it right yet.

So I think like LSTM in its most vanilla form, I think everybody's pretty convinced that it's not going to handle really long time scale hierarchical patterning.

And I'd love it if someone comes along and says, no, you don't need anything but vanilla LSTM to do this.

But I think what makes music interesting over, even after like 5 seconds or 10 seconds, is this idea that you're getting repetition, you're getting harmonic shifts like chord changes.

And 1 way to talk about that there, there is that, you know, you have some lower level pattern generation going on, but there's some conditioning happening.

Oh, now continue generating, but the condition shifted.

And so I think if we start talking about conditional models, if we talk about models that are explicitly hierarchical, If we talk about models that we can sample from in different ways, we can start to get somewhere.

But I think only a recurrent neural network is, you know, it would be reductionist to say that it's the whole answer.

I was thinking about how you were, was it the TensorFlow or the IOTalk where you were talking about Bach?

Oh, that we did stuff that was more Bach than Bach?

It's like, I'll make the best Picasso painting for you, but it's not necessarily a Picasso painting because it's not necessarily saying anything.

So I think by analogy, so first in case it's not clear, I don't believe that we made something that was better than Bach.

But when we put these tunes out for untrained listeners to listen to, they sometimes voted them as sounding more Bach-y.

And I think it's, imagine what these models are learning.

They're learning the principle axes of variance.

Because they have a limited memory, they're compressed.

So if you sample from sketch RNN with very low temperature, meaning without a lot of noise in the system, you actually get what, if you want to squint your eyes and break philosophy, is like the platonic cat.

You get a cat that looks more like a cat anyone would draw, sort of the average cat.

And I think that's sort of what we're getting from these time series models as well.

They're kind of giving you something that's more a caricature than a sample, right?

See all Y Combinator transcripts on Youtube

Making Music and Art Through Machine Learning - Doug Eck of Magenta