AI startups such as Pinecone, Milvus, and Chromadb have raised millions of $ in the hot AI boom era. They all have a common product called vector database. This is a simple explanation of what exactly is vector database and how they are different from traditional databases.

Vector database article: https://www.pinecone.io/learn/vector-database/

Langchain tutorial playlist: https://www.youtube.com/playlist?list=PLeo1K3hjS3uu0N_0W6giDXzZIcB07Ng_F

Do you want to learn technology from me? Check https://codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses.

Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.

🎥 Codebasics Hindi channel: https://www.youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg

#️⃣ Social Media #️⃣
🔗 Discord:  https://discord.gg/r42Kbuk
📸 Dhaval's Personal Instagram: https://www.instagram.com/dhavalsays/
📸 Codebasics Instagram: https://www.instagram.com/codebasicshub/
🔊 Facebook: https://www.facebook.com/codebasicshub
📱 Twitter: https://twitter.com/codebasicshub
📝 Linkedin (Personal): https://www.linkedin.com/in/dhavalsays/
📝 Linkedin (Codebasics):  https://www.linkedin.com/company/codebasics/
🔗 Patreon: https://www.patreon.com/codebasics?fan_landing=true

There are some AI startups that have raised millions of dollars of funding and they have 1 product in common which is Vector database.

Let's try to understand what exactly is vector database today when you search in Google calories in apple versus Employees in apple Google figures out that the first apple means fruit and the second 1 is a company.

Have you ever wondered how does Google does this?

It uses a technique called semantic search.

Semantic search means not searching using the exact keyword matching but understanding the intent of a user query and using the context to perform the search.

For doing semantic search internally it uses the concept of embedding.

Word embedding or sentence embedding is nothing but a numerical representation of text.

Let's first understand how exactly embedding works.

Let's figure out how you can represent this word apple into a numeric presentation given this particular context.

1 way is to think about different features or properties of words.

Here you can have related to phones, is location, has stock etc as properties and then you assign value for each of these properties.

You get a sequence of numbers as a result and that is nothing but a vector.

So this vector is a word embedding for the word apple for this particular context.

If you're talking about apple the fruit then the embedding might look something different because the value of these properties is different and when you have the embeddings for different words looking at the embedding you can say that the second apple and the word orange they are similar because look at their values they are matching Of course there are some values which are not matching but compared to this vector and the first vector second and third vector are kind of similar.

Same way if you have let's say Samsung as a word you can represent that into a numeric presentation.

And when you look at again all these vectors you can figure out that the first and the fourth vectors are kind of similar.

So using these vectors you can figure out the similarity not just similarity you can actually do a complex arithmetic such as this.

This is a famous example in the NLP domain where you can perform this mathematics using a technique called word2vec.

Word2vec is a technique to represent word into a numeric representation.

So if you want to know more about it, you can go and look at this video.

In that video, I have explained how you can generate handcrafted features for each of these and you can do this particular math.

Now just for intuition I explain everything using handcrafted features but in reality you use some complex statistical techniques to generate these word embeddings.

Again if you have curiosity you can watch those 2 videos or this particular video on BERT.

So far let's say you have this understanding that there are variety of these techniques that you can use to represent a word or a sentence or even a document into an embedding.

And here are just different techniques which are being used in chatgpt era.

Obviously, transformer based embedding techniques are getting popular.

So when you're using OpenAI API for embedding, you know what technique it is using underneath.

When you are building any text-based AI application you might have thousands or even millions of embedding vectors and you need to store them somewhere.

When you think about storing them The first option that comes to mind is a traditional relational database.

So let's say for our use case, we have these 4 articles.

The first 2 red are related to Apple the fruit.

You will first generate the embedding, let's say using open AI API, and then you will save that into let's say your SQL database.

Now when you have a search query you will also generate embedding for that and you will try to compare this embedding with the stored embedding and try to retrieve the relevant documents.

Here you will use a concept of cosine similarity to retrieve the matching vectors and you can display it in your Google search result this in theory works okay but in reality you will have millions of records or even millions of records in your database and that's when things starts getting interesting because just think about matching this vector for a query vector if you want to match with these stored vectors then 1 of the approach you can use is linear search where you go 1 by 1 and if cosine similarity is close to 1 then you will put that vector into your result data set and then you can keep on going and store your result vectors.

Now you already realize the problem if there are millions of stored vector embeddings, your computation is going to be too much.

You know, your hairs will be raised because you can't handle delay and computational requirements for such a use case.

How do we do this in a traditional database?

Database index helps you search things faster.

Similarly, in this particular use case, we can use 1 hashing function.

We don't need to go into detail what that hashing function is.

But let's say this hashing function is creating buckets of similar looking embeddings.

And then when you have a search query, you can let that go through the same hashing function, which will bucket it into 1 of these 3 buckets and then within that bucket you can do individual linear search.

This way you are only matching with those vectors which are in bucket 1.

You don't have to match it with bucket 2 and bucket 3.

This will speed things up and this technique is called locality sensitive hashing.

This is 1 of the techniques that vector databases is using.

There are many such techniques and those techniques are outlined in this beautifully written article.

So far you realize that vector databases help you do faster search.

They also help you store things in an optimal way.

So these are the 2 big benefits why vector databases are gaining popularity.

I hope this video helped you understand the intuition behind vector databases.

If you have any question please post in the comment box below.

If you like this video please give it a thumbs up and share it with your friends.


Speaker 0: There are some AI startups that have raised millions of dollars of funding and they have 1 product in common which is Vector database. Let's try to understand what exactly is vector database today when you search in Google calories in apple versus Employees in apple Google figures out that the first apple means fruit and the second 1 is a company. Have you ever wondered how does Google does this? It uses a technique called semantic search. Semantic search means not searching using the exact keyword matching but understanding the intent of a user query and using the context to perform the search.

For doing semantic search internally it uses the concept of embedding. Word embedding or sentence embedding is nothing but a numerical representation of text. Let's first understand how exactly embedding works. Let's figure out how you can represent this word apple into a numeric presentation given this particular context. 1 way is to think about different features or properties of words.

Here you can have related to phones, is location, has stock etc as properties and then you assign value for each of these properties. Revenue here means 82 billion dollar. You get a sequence of numbers as a result and that is nothing but a vector. So this vector is a word embedding for the word apple for this particular context. If you're talking about apple the fruit then the embedding might look something different because the value of these properties is different and when you have the embeddings for different words looking at the embedding you can say that the second apple and the word orange they are similar because look at their values they are matching Of course there are some values which are not matching but compared to this vector and the first vector second and third vector are kind of similar.

Same way if you have let's say Samsung as a word you can represent that into a numeric presentation. Is it related to phones? Yes. Is it a location? No.

And when you look at again all these vectors you can figure out that the first and the fourth vectors are kind of similar. So using these vectors you can figure out the similarity not just similarity you can actually do a complex arithmetic such as this. This is a famous example in the NLP domain where you can perform this mathematics using a technique called word2vec. Word2vec is a technique to represent word into a numeric representation. I have made a separate video.

So if you want to know more about it, you can go and look at this video. In that video, I have explained how you can generate handcrafted features for each of these and you can do this particular math. Now just for intuition I explain everything using handcrafted features but in reality you use some complex statistical techniques to generate these word embeddings. Again if you have curiosity you can watch those 2 videos or this particular video on BERT. So far let's say you have this understanding that there are variety of these techniques that you can use to represent a word or a sentence or even a document into an embedding.

And here are just different techniques which are being used in chatgpt era. Obviously, transformer based embedding techniques are getting popular. So when you're using OpenAI API for embedding, you know what technique it is using underneath. When you are building any text-based AI application you might have thousands or even millions of embedding vectors and you need to store them somewhere. When you think about storing them The first option that comes to mind is a traditional relational database.

So let's say for our use case, we have these 4 articles. The first 2 red are related to Apple the fruit. The remaining are Apple the company. You will first generate the embedding, let's say using open AI API, and then you will save that into let's say your SQL database. Now when you have a search query you will also generate embedding for that and you will try to compare this embedding with the stored embedding and try to retrieve the relevant documents.

Here you will use a concept of cosine similarity to retrieve the matching vectors and you can display it in your Google search result this in theory works okay but in reality you will have millions of records or even millions of records in your database and that's when things starts getting interesting because just think about matching this vector for a query vector if you want to match with these stored vectors then 1 of the approach you can use is linear search where you go 1 by 1 and if cosine similarity is close to 1 then you will put that vector into your result data set and then you can keep on going and store your result vectors. Now you already realize the problem if there are millions of stored vector embeddings, your computation is going to be too much. You know, your hairs will be raised because you can't handle delay and computational requirements for such a use case. You need to do something smart. How do we do this in a traditional database?

Well, we use a thing called index. Database index helps you search things faster. Similarly, in this particular use case, we can use 1 hashing function. We don't need to go into detail what that hashing function is. But let's say this hashing function is creating buckets of similar looking embeddings.

Okay. And then when you have a search query, you can let that go through the same hashing function, which will bucket it into 1 of these 3 buckets and then within that bucket you can do individual linear search. This way you are only matching with those vectors which are in bucket 1. You don't have to match it with bucket 2 and bucket 3. This will speed things up and this technique is called locality sensitive hashing.

This is 1 of the techniques that vector databases is using. There are many such techniques and those techniques are outlined in this beautifully written article. I'll provide a link to this article. You can read through it. So far you realize that vector databases help you do faster search.

They also help you store things in an optimal way. So these are the 2 big benefits why vector databases are gaining popularity. I hope this video helped you understand the intuition behind vector databases. If you have any question please post in the comment box below. If you like this video please give it a thumbs up and share it with your friends.

Thank you for watching.

See all codebasics transcripts on Youtube

Vector Database Explained | What is Vector Database?