Introducing LeMUR, AssemblyAI's new framework for applying powerful LLMs to transcribed speech
With a single line of code, LeMUR can quickly process audio transcripts for up to 10 hours worth of audio content, which effectively translates into ~150k tokens, for tasks likes summarization and question answer.

Sam Altman & Patrick Collison's Interview - https://www.assemblyai.com/playground/v2/transcript/6gsem9pflo-f15a-41a8-a73b-dd673f20247b

LeMUR AI Details - https://www.assemblyai.com/blog/lemur-early-access/

❤️ If you want to support the channel ❤️
Support here:
Patreon - https://www.patreon.com/1littlecoder/
Ko-Fi - https://ko-fi.com/1littlecoder

Imagine you have a 10 hours of audio clip and you want to use that audio clip to ask questions using a large language model.

Like you have to set up your own vector database.

You have to do a bunch of things, but assembly AI has come up with a new model called Limer.

It's actually a framework that can apply powerful large language models to transcribed speech.

So with Limer it's completely possible for you to transcribe a 10 hours audio clip into an audio transcript and use large language models for that like it's about like 150, 000 token and you can ask questions or do summarizations and many more tasks and in this video I would like to dive deep into Limer and we're going to see also an example.

It's you have to join a waiting list, but there is a playground where you can play with.

And I would like to show you what I did and then how do I feel about Limer.

It's a new framework for applying large language models to transcript speech.

With just a single line of code, Lemur can quickly process audio transcripts of about 10 hours worth of audio content, which effectively translates into approximately 150, 000 tokens for tasks like summarization and question and answer.

Now if you are familiar with the large language model space, you know 150, 000 tokens as a context window do not exist.

Like in fact yesterday We covered a video where we did hundred thousand tokens and we celebrated like anything So what is this what is happening behind the scenes is, without Limer, you can probably approximately process 45 minutes of audio using a large language model.

Because of the context window, 8, 000 tokens.

I mean, forget about Anthropic Cloud 100K here, just default 8, 000 token.

But with Limer, what you can do is now you can do 10 hours of audio content which is approximately 150, 000 tokens and how is it happening it's happening because there is no black magic behind it basically what open AI sorry assembly AI has done is they've created an intelligent system like they created a turnkey solution that has got a pipeline of vector database like chain of thought prompting and self-evaluation As you can see this is the architecture of Lemur.

So Lemur is at the center and you can give the transcription.

Conformer does the transcription Then audio goes inside the transcript ID and the text data.

So now that goes inside a transcript database.

As you can see, it doesn't directly go inside a large language model.

Rather, there is a transcript database where this is going inside.

And Now that has been segmented intelligently into multiple segments.

Lemur segments the transcripts intelligently.

Like we don't know what that intelligently is, but as you can see, it gets segmented into multiple like files or indexes or segments that goes inside a vector database.

Now this individual components go inside large language models.

Those are finally stitched together using a chain of thought, reasoning and self-evaluation and all these kind of techniques.

And now finally that goes inside another ILM and then the response comes back to the user as a limit response.

This is like, honestly speaking, this is an incredibly intelligent architecture because now what you can do is in just a single call, just a single API call, you can send 1 big audio file.

And you can get the response, whatever you want to do back.

It, to be honest, it unlocks a lot of possibilities that were not possible.

And 1 of the possibilities that I would like to show right now, like right now, which is again, to show that this is something that I tested and it is amazing.

It's an interview by a Stripe founder, Patrick Collison with Sam Altman.

I really enjoyed the interview, but, you know, due to work, I could not finish it, like it's a, it's a 52 minute interview.

I thought, okay, what if I can give this to limer and then ask questions to limer.

I went to the limer playground, as you can see here and inside limer playground, I asked it to transcribe it for me and it did.

As you can see the like the entire transcript is available here until the end and I can also listen to it.

The first fireside is between Sam Altman and Patrick Carlson.

So 54 minutes, like close to 1 hour of audio clip is available got transcribed in less than 10 minutes And what I can do here is now I can ask question.

So all these things, like whatever is here, it goes inside this large language model.

I mean, unfortunately I'm not using like a 10 hour audio clip.

Probably I need to take a Lex Friedman podcast or a Joe Rogan podcast here, but let's say.

I'm just taking this and I've given it to limer and I can ask questions.

I can ask questions like summarize this transcript.

I want to ask specific questions about the transcript.

How could I improve this transcript for more engagement?

I want to say, I want to ask specific questions about the transcript.

So what is the question that I want to ask?

I know at some point in this interview, there was a discussion about nuclear resources.

So I'm going to say, what did Sam Altman say about nuclear resources and its comparison with AI okay I don't know if it is going to answer this is the first time I'm trying it out so answer format and this completely fine so I just said what did Sam Altman say about nuclear resources its comparison with AI and I know for sure like I listened to it.

So I can see very well if it hallucinates or if it doesn't hallucinate.

Sam Alban said access to nuclear resources is more difficult than for AI due to complex permitting process.

This is like 1 of the 1 of the point here but you can actually see nuclear.

First of all He was not very interested in people comparing these things and there were a lot of other Secret like a lot of other questions about nuclear secrets even Let's see when I specifically say nuclear secrets instead of nuclear resources what it says so what did Sam Altman say about nuclear secrets and its comparison with AI and I'm not editing this video it's live as you can see it say that while classifying Nuclear secrets is probably a good idea.

He thinks the biggest lesson from nuclear non-proliferation for AI safety is creating an international regulatory agency for powerful AI systems similar to IAEA for nuclear technology.

I am telling you, you can listen to this podcast.

I definitely listened to this piece and this is like, this is a context of what he actually said.

Like I can say, do, do, do a summary or I can say AI coach.

Let's say I want to do a summary and describe the context and optional so I can just click generate and this can create a summary for me.

So now what we have seen like Let me take you back to the start.

We have been introduced to a new model from assembly AI, which is not going to be a free model, to be honest.

Because this, this is not an open source solution.

So Lemur is a solution called leveraging large language models to understand, recognize speech.

So with 1 APA call, you can transcribe an audio clip or YouTube clip or whatever it is.

That's up to 10 hours of content, which is approximately 150, 000 tokens.

Thanks to their very intelligent architecture.

So the summary is this Sam Altman and an interviewer.

Oh, it's Patrick Collison AI discuss AI innovation.

AI accelerated science, scientists, capital, few talented young founders exist today, something wrong as there were many like Elon Musk before.

The key points are open AI, secret advances, AI assisting, replacing scientists, Lack of top young founders.

So honestly, I wouldn't say this is like a great summary because from the 25 minutes I listened It's not like the perfect summary that I would have probably expected But I don't know what happened in the remaining 25 minutes as well Like so could could be like the entire conversation went in this direction It is super amazing and you can see that it unlocks a lot of potential.

Imagine like you're a student and 1 day you could not go to class and you have got your students have got recordings like maximum what you would add in 8 classes.

It's like 8 hours or 6 hours now you can plug in all the 8 or 6 hours inside this and then you can ask questions.

Education could be completely transformed with this thing so what Lemur unlocks is you can apply LLMs to multiple audio transcripts You get like safe output and in customer support you can inject context specific use cases like for example You can say did the sales representative follow the exact procedure process of prospecting and You can supply a sales playbook So now an agent can be evaluated based on the playbook rather than just generally knowing.

And I wouldn't probably like this kind of evaluation, but you know, this is part of life.

You can do fast integration and it is state of the art.

So what assembly AI has also let you do is you can just click this playground and start playing with stuff like there is a 3 hour customer call mp3.

It's a talk about you know dyslexia and all these kind of things and you can also supply a bunch of things like for example there is a Django course Django course Django course Django course and you can say oh what can the instructors do better okay it looks like it's something related to my profession of being a youtuber so let's go to the go to the playground and then say how could I improve the explanation of Django so that high school students can understand okay so basically let's say I teach Django and if I'm somebody who teaches Django, this doesn't look like Django stuff.

This looks This looks like some Supreme Court stuff Okay, cool.

It's going to probably most likely hallucinate and then give us something.

This it has given me answers, but as you can see, it is pretty good.

And LEMUR that stands for leveraging large language models to understand recognize speech is currently available as a waiting list.

And once you join the waiting list, you get access, early access to LEMUR as a product.

You can play with the API, but Let's say you do not have access just like me.

All you can do is click the try and playground and go there.

And then you can paste a YouTube link or upload an audio and then start playing with the lemon.

Incredible piece of technology, but I think this is also going to give idea to a lot of startups about how to build this kind of solution where you can now artificially increasing the context window of a large language model using some intelligent turnkey solution.

Let me know in the comments what you feel about it?


Speaker 0: Imagine you have a 10 hours of audio clip and you want to use that audio clip to ask questions using a large language model. That is currently not possible easily. Like you have to set up your own vector database. You have to do a bunch of things, but assembly AI has come up with a new model called Limer. It's actually a framework that can apply powerful large language models to transcribed speech.

So with Limer it's completely possible for you to transcribe a 10 hours audio clip into an audio transcript and use large language models for that like it's about like 150, 000 token and you can ask questions or do summarizations and many more tasks and in this video I would like to dive deep into Limer and we're going to see also an example. Limer as a model has not been released. It's you have to join a waiting list, but there is a playground where you can play with. It's quite incredibly amazing. And I would like to show you what I did and then how do I feel about Limer.

To start with what is Limer? It's a new framework for applying large language models to transcript speech. With just a single line of code, Lemur can quickly process audio transcripts of about 10 hours worth of audio content, which effectively translates into approximately 150, 000 tokens for tasks like summarization and question and answer. Now if you are familiar with the large language model space, you know 150, 000 tokens as a context window do not exist. Like in fact yesterday We covered a video where we did hundred thousand tokens and we celebrated like anything So what is this what is happening behind the scenes is, without Limer, you can probably approximately process 45 minutes of audio using a large language model.

Because of the context window, 8, 000 tokens. I mean, forget about Anthropic Cloud 100K here, just default 8, 000 token. But with Limer, what you can do is now you can do 10 hours of audio content which is approximately 150, 000 tokens and how is it happening it's happening because there is no black magic behind it basically what open AI sorry assembly AI has done is they've created an intelligent system like they created a turnkey solution that has got a pipeline of vector database like chain of thought prompting and self-evaluation As you can see this is the architecture of Lemur. So Lemur is at the center and you can give the transcription. First the audio goes inside.

They've got a model called Conformer. Conformer does the transcription Then audio goes inside the transcript ID and the text data. So now that goes inside a transcript database. As you can see, it doesn't directly go inside a large language model. Rather, there is a transcript database where this is going inside.

And Now that has been segmented intelligently into multiple segments. Lemur segments the transcripts intelligently. Like we don't know what that intelligently is, but as you can see, it gets segmented into multiple like files or indexes or segments that goes inside a vector database. Now this individual components go inside large language models. Those are finally stitched together using a chain of thought, reasoning and self-evaluation and all these kind of techniques.

And now finally that goes inside another ILM and then the response comes back to the user as a limit response. This is like, honestly speaking, this is an incredibly intelligent architecture because now what you can do is in just a single call, just a single API call, you can send 1 big audio file. And you can get the response, whatever you want to do back. It, to be honest, it unlocks a lot of possibilities that were not possible. And 1 of the possibilities that I would like to show right now, like right now, which is again, to show that this is something that I tested and it is amazing.

I was watching this interview. It's an interview by a Stripe founder, Patrick Collison with Sam Altman. I really enjoyed the interview, but, you know, due to work, I could not finish it, like it's a, it's a 52 minute interview. I kicked about like, 24 minute. I thought, okay, what if I can give this to limer and then ask questions to limer.

So that's exactly what I did. I copied the YouTube URL. I went to the limer playground, as you can see here and inside limer playground, I asked it to transcribe it for me and it did. As you can see the like the entire transcript is available here until the end and I can also listen to it. I mean, it's not like I cannot.

I can completely listen. The first fireside is between Sam Altman and Patrick Carlson. Well, that again, but if it's. So 54 minutes, like close to 1 hour of audio clip is available got transcribed in less than 10 minutes And what I can do here is now I can ask question. So all these things, like whatever is here, it goes inside this large language model.

I mean, unfortunately I'm not using like a 10 hour audio clip. Probably I need to take a Lex Friedman podcast or a Joe Rogan podcast here, but let's say. I'm just taking this and I've given it to limer and I can ask questions. I can ask questions like summarize this transcript. Like I'm 12 years old.

I want to ask specific questions about the transcript. How could I improve this transcript for more engagement? Nothing specific. I just want to play with limer features. I want to say, I want to ask specific questions about the transcript.

So what is the question that I want to ask? I know at some point in this interview, there was a discussion about nuclear resources. Okay. So I'm going to say, what did Sam Altman say about nuclear resources and its comparison with AI okay I don't know if it is going to answer this is the first time I'm trying it out so answer format and this completely fine so I just said what did Sam Altman say about nuclear resources its comparison with AI and I know for sure like I listened to it. I know what he said.

So I can see very well if it hallucinates or if it doesn't hallucinate. Sam Alban said access to nuclear resources is more difficult than for AI due to complex permitting process. This is like 1 of the 1 of the point here but you can actually see nuclear. You can see here. First of all He was not very interested in people comparing these things and there were a lot of other Secret like a lot of other questions about nuclear secrets even Let's see when I specifically say nuclear secrets instead of nuclear resources what it says so what did Sam Altman say about nuclear secrets and its comparison with AI and I'm not editing this video it's live as you can see it say that while classifying Nuclear secrets is probably a good idea.

It's not a complete solution on its own. He thinks the biggest lesson from nuclear non-proliferation for AI safety is creating an international regulatory agency for powerful AI systems similar to IAEA for nuclear technology. This is super impressive. I am telling you, you can listen to this podcast. I definitely listened to this piece and this is like, this is a context of what he actually said.

And we can ask a summary also. Like I can say, do, do, do a summary or I can say AI coach. How can I improve this engagement? I don't want to do coach. Let's say I want to do a summary and describe the context and optional so I can just click generate and this can create a summary for me.

So now what we have seen like Let me take you back to the start. We have been introduced to a new model from assembly AI, which is not going to be a free model, to be honest. Because this, this is not an open source solution. So Lemur is a solution called leveraging large language models to understand, recognize speech. So with 1 APA call, you can transcribe an audio clip or YouTube clip or whatever it is.

That's up to 10 hours of content, which is approximately 150, 000 tokens. And then you can start asking questions. Thanks to their very intelligent architecture. I'm still amazed by their architecture. So the summary is this Sam Altman and an interviewer.

Oh, it's Patrick Collison AI discuss AI innovation. They're talking about what they did. AI accelerated science, scientists, capital, few talented young founders exist today, something wrong as there were many like Elon Musk before. The key points are open AI, secret advances, AI assisting, replacing scientists, Lack of top young founders. So honestly, I wouldn't say this is like a great summary because from the 25 minutes I listened It's not like the perfect summary that I would have probably expected But I don't know what happened in the remaining 25 minutes as well Like so could could be like the entire conversation went in this direction It is super amazing and you can see that it unlocks a lot of potential.

Imagine like you're a student and 1 day you could not go to class and you have got your students have got recordings like maximum what you would add in 8 classes. It's like 8 hours or 6 hours now you can plug in all the 8 or 6 hours inside this and then you can ask questions. Education could be completely transformed with this thing so what Lemur unlocks is you can apply LLMs to multiple audio transcripts You get like safe output and in customer support you can inject context specific use cases like for example You can say did the sales representative follow the exact procedure process of prospecting and You can supply a sales playbook So now an agent can be evaluated based on the playbook rather than just generally knowing. And I wouldn't probably like this kind of evaluation, but you know, this is part of life. It's modular.

You can do fast integration and it is state of the art. So what assembly AI has also let you do is you can just click this playground and start playing with stuff like there is a 3 hour customer call mp3. There is a poetry superpower. This is a popular TED talk. It's a talk about you know dyslexia and all these kind of things and you can also supply a bunch of things like for example there is a Django course Django course Django course Django course and you can say oh what can the instructors do better okay it looks like it's something related to my profession of being a youtuber so let's go to the go to the playground and then say how could I improve the explanation of Django so that high school students can understand okay so basically let's say I teach Django and if I'm somebody who teaches Django, this doesn't look like Django stuff.

This looks This looks like some Supreme Court stuff Okay, cool. Maybe I came with a wrong transcript. That's fine. It's going to probably most likely hallucinate and then give us something. Okay.

This it has given me answers, but as you can see, it is pretty good. And LEMUR that stands for leveraging large language models to understand recognize speech is currently available as a waiting list. So you can join the waiting list. And once you join the waiting list, you get access, early access to LEMUR as a product. You can play with the API, but Let's say you do not have access just like me.

All you can do is click the try and playground and go there. And then you can paste a YouTube link or upload an audio and then start playing with the lemon. Incredible piece of technology, but I think this is also going to give idea to a lot of startups about how to build this kind of solution where you can now artificially increasing the context window of a large language model using some intelligent turnkey solution. Quite exciting times. Let me know in the comments what you feel about it?

See you in another video. Transcribed by https://otter.ai

See all 1littlecoder transcripts on Youtube

Chat with Audio Speech using LeMUR - AssemblyAI's LLM Framework