- The video discusses options for building AI SaaS applications using OpenAI GPT APIs or alternative APIs.
- The first recommended solution is replicate.com, which is popular for image generation and provides examples and cost information.
- The video excludes popular cloud solutions like Microsoft Azure, Google GCP, and AWS.
- Hugging Face's inference endpoint is another recommended option for deploying models easily.
- Together.ai offers fine tuning and inference services, specializing in llama models and claiming to be fast, cost-efficient, and scalable.
- Cohere is a question and answering or text completion endpoint, offering free testing during prototyping.
- Perplexity.ai offers fast inferencing with models like mistral and lama, claiming lower latency than Replicate and AnyScale.
- Anthropic is a highly capable model with a large context window, but its API is not widely available and has a waiting list.

- The Anthropic Cloud models offer a large context window for text generation, but they are not widely available yet.
- Text generation web UI from UbaBuga allows hosting any model and offers an API server for chat and completions endpoints.
- Sky Pilot is an open-source project that allows running LLMs on any cloud provider, with support for AWS, Google Cloud, and others.
- LightLLM is a library that provides compatibility with multiple APIs, allowing easy switching between different models.
- TGI (Text Generation Inference) is a project offered by gradient.ai and supports quantization and chat completion in an open AI format.
- VLLM is an open-source project that offers fast LLM inference and supports various hugging face models.

- The video discusses different solutions for deploying large language models (LLMs).
- VLLM is recommended as one of the fastest inference solutions for custom servers and GPUs, and it is open source and supports various hugging face models.
- MLC is another fast solution for LLM deployment, including on smartphones and handheld devices, but it may be more difficult to deploy.
- A blog post by Hemel is recommended for optimizing for latency and deploying LLMs on your own server.
- It is advised to have backup solutions and not rely solely on one API provider.
- It is suggested to compare costs, speed, and quality when choosing between open source and paid solutions.
- Open source solutions allow for immediate updates, but closed solutions provide security and other benefits.
- The video encourages viewers to share their thoughts and find the information useful.

The video discusses various options for building AI SaaS applications using OpenAI GPT APIs or alternative APIs. It recommends solutions such as replicate.com, Hugging Face's inference endpoint, Together.ai, Cohere, Perplexity.ai, Anthropic, UbaBuga, Sky Pilot, LightLLM, TGI, VLLM, and MLC. The video advises comparing costs, speed, and quality when choosing between open source and paid solutions, and suggests having backup solutions and not relying solely on one API provider. It also encourages viewers to share their thoughts and find the information useful.

This video offers some great AI API solutions and some open source projects to host your own models! The objective of this video is to make you rely less on OpenAI and build better more reliable GPT wrappers that aren't only from OpenAI APIs aka AI SaaS. 

🔗 Links 🔗

https://replicate.com/explore
https://huggingface.co/mistralai/Mistral-7B-v0.1
https://www.together.ai/products#inference 
https://cohere.com/pricing
https://blog.perplexity.ai/blog/introducing-pplx-api
https://www.anthropic.com/product

Open Source Projects to make AI Hosting easier 

https://github.com/oobabooga/text-generation-webui/
https://github.com/skypilot-org/skypilot
https://github.com/BerriAI/litellm
https://github.com/Preemo-Inc/text-generation-inference
https://github.com/vllm-project/vllm
https://github.com/mlc-ai/mlc-en

Hamel's blogpost - https://hamel.dev/notes/llm/inference/03_inference.html


❤️ If you want to support the channel ❤️
Support here:
Patreon - https://www.patreon.com/1littlecoder/
Ko-Fi - https://ko-fi.com/1littlecoder

🧭 Follow me on 🧭
Twitter - https://twitter.com/1littlecoder
Linkedin - https://www.linkedin.com/in/amrrs/

Building AI SaaS application either using OpenAI GPT APIs or you are not using any of the OpenAI APIs and still you want to use APIs then this video is going to tell you all the available options are at least the things that I want to recommend you from either the paid solution like hosted APIs and also some open source solution that can make your job as a developer much much easier.

So I'm going to cover like at least like 5 hosted API solutions at least 5 open projects open source projects that can help you in building your AI SaaS application with versatility and less dependability and 0 0 open AI at all.

The first 1 like the way this order is going to go is like the kind of services that I trust.

It's basically my personally opinionated list and none of the items that I mentioned in this video are sponsored on this video.

This is completely like the only money I'm going to make is through AdSense so you can completely trust that I'm not putting in any sponsored money into this particular video at all.

So replicate is a very popular solution and they were super popular during the stable diffusion time and they are continuously innovating and still developing solutions.

So if you're going to build image generation, I think like it's no brainer to use replicate like there are a lot of popular indie hackers solo entrepreneurs like Peter levels are using replicate and then Also recommending replicate a lot But again, if you want to use it for text like this video is particularly focused on text still replicate is a good solution So you can go here and then see okay What is like llama 270 billion chart you can just click this and then you can see a bunch of examples And also you can get to see the cost What kind of inference speed you get and what how much time it takes and all the information.

So replicate is definitely the first solution that I would strongly recommend especially if you are looking at scale.

Another important thing in this list is that we are going to leave the obvious.

I'm going to leave Microsoft Azure, I'm going to leave Google GCP, I'm going to leave AWS, AWS Bedrock or any other popular cloud solution like OVH anything so we have a separate section where I'm going to show you an open source project that you can use to deploy it on your own server but I'm not going to mention any of the popular cloud solution that is something to keep in mind.

The next solution that I wanted to mention is Hugging Face's inference endpoint.

A lot of you would know Hugging Face being the open source company where you can host the model where you can host the data set but not a lot of people know that hugging face also has to make money it's a for-profit company and for that they actually let you deploy any single model that is available on hugging faces model and in just a bunch of clicks you can literally deploy your model and yeah they've got like different cost depending upon the machine that you pick and all the other items so the service is called inference endpoint you can just click this and then say deploy your first model you have to be logged in you have to add your credit card but for any model that you pick like if you for example this is e34 billion parameter model if you go here you can click this 3 dots and you can basically see like, what are the options that you have to add the model?

Like in this case, I don't see anything, but typically you can see like, this is an inference example, But typically you can actually see how to deploy the model and stuff.

So you see the deploy button here So inference API inference endpoint.

So inference endpoint is what we currently discussed about So you can use hugging faces inference endpoint, which is definitely the second option in terms of my trustworthiness But I've also heard from a lot of people that it is not 1 of the cheapest options.

In fact, they are not the cheapest option at all.

There are a lot of other cheaper options available for you.

But it depends like I'm just giving you a bunch of suggestions.

So if you want the cheapest option maybe this is not a good solution for you but again if you want to pick any model and then use it like even the fine-tuned models even the models that you upload then this is 1 of the best solutions for you.

The next 1 is a very new offering it's a Together.ai so it's a company that is offering you Fine tuning and inference so their inference is available.

I've heard from people I know at least that you know their solution They're they're really like a closer to the developer community like when you want to buy their solution you can go talk to them they help you and they've got a bunch of models available especially they specialize in the llama family of model I think they picked up pace when llama came into picture so you can go there and then use it they have some numbers about like how they are faster and in fact like a couple of solutions that we are going to discuss in this video TGI VLLM they're also comparing with these solutions and then they're saying how they are faster and they're also mentioning how they are cheaper than GPT 3.5 turbo and 1 thing that you keep in mind especially when you're building AI SaaS you always need like fallback solutions OpenAI you are building OpenAI is not up and running you need some fallback a lot of people I consult I do a lot of consulting So a lot of people I consult, I usually tell them have a tiered approach.

Have 1 for open AI and second tier, you should have something else.

In fact, like the way we do is like we have the first year, which is open source models, cheaper models.

Then you go into like GPT-4 because you don't need GPT-4 for everything.

And you can definitely use solutions like together for building that solution.

So, so together has a lot of offering and you can also check.

I think when you sign up, you get like a free, a few credits.

So I, I initially got credit and, I think they've changed the landing page quite a bit like last time I used this solution Their landing page didn't look as sophisticated as this but yeah right now there are solutions that you can use they claim to be fast, cost-efficient and also scalable.

Scalable is a part that I wouldn't necessarily trust with them given that they're a new solution but yeah so they've got a lot of funding they might have enough juice to support you.

Before I start with Cohere I wanted to say first that Cohere may not have the best model in the entire list that we have seen.

So in the past like replicate, hugging face together in all these 3 solutions you can basically kind of deploy any model that you want Lama, Mistral and any other model that you want.

Cohere has their own model in itself it's not a hosting service it is just a question and answering or text completion endpoint so this is a completely different service altogether So now the thing with coherence the good thing with coherence you can test as much as you want Anything during prototyping is completely free only when you productionize you have to pay them That's the thing but I don't I don't think their models are as capable or like a drop-in replacement for your GPT 3.5 In fact, like I don't think their models are as capable or like a drop in replacement for your GPT 3.5.

In fact, like I don't think their models are as good as let's say Mistral AI or Zephyr or all these latest open source models.

But if you still want to use Cohere, I think you're free to go use it, try it, but not the best model, but you have the option of using free resources until you deploy it.

And then when you deploy it, you can just, you know, have the tiered approach that I mentioned.

The next 1 is again, a very new service, but the company kind of have established themselves as a strong player in this AI LLM space especially in the RAG space.

So this is coming from perplexity.ai they recently announced like as you can see like October 4th they announced that they are announcing PPLX API so this is a new offering from them.

To be honest like Perplexity AI is 1 of the fastest inferencing I've ever seen on the internet.

You go ask any question it's the fastest like It has 1 of the fastest Most tokens per second I've ever seen on the internet, especially for a free service So perplexity.ai offers also an APA at this point and you know The point their point is like you get access to like mistral, lama, code lama, replitcode kind of models and Yeah, I can I can definitely say they they're quite fast like from what I've seen?

They are claiming to be 2.9x lower latency than Replicate and 3.1x lower latency than AnyScale.

AnyScale is the solution that I have left out of this video my apologies But that is something that you can also try but again perplexity AI has like multiple advantages PPLX API is in public beta and is free for users with perplexity pro subscription so if you have got perplexity pro subscription you can right away try they've got like some some comparison about how they are cheaper in cost and how they are also more like speed the faster I think their biggest advantage is speed but also you can go here and then check how good they are and what you can do with perplexity.ai The next 1 I mean I would have put this at the start but the reason I put this at the last is Anthropic.

Anthropic is definitely 1 of the most capable models like somewhere on top with GPT models and they've got like a heck load of funding.

It's a team that came from OpenAI and all these things.

Their API is not widely available like you have to be in the US.

I don't know you have to be on the waiting list it has a lot of these nuances that's why I put it on the last of this paid hosted offering solutions but otherwise I think it's it's a really good alternative but it's it's almost like going back to OpenAI This is it's very similar like OpenAI or Cohere.

It is just hosted and offered as an endpoint.

You are not selecting the model that you want.

You just have to use the model that you that they provide you and then you anticipate that time and time again they will improve their model.

I think 1 of the biggest advantages and selling points of the Anthropic Cloud models is the context window they've got like more than 100k context window that means you can do longer text generation you can do more efficient drag and lot of other these kind of advantages of there but still the model is not available like it's a big bummer and not available across the world waiting list and all these kind of messes there but still if you want to join the queue you should definitely click request access and join the queue.

So at this point of this video we are almost like done with all the hosted solution like I said like I missed any scale definitely check out any scale as well.

So another decent offering Like they've got like really good documentation.

I've seen a lot of articles from their dev advocacy team very thoughtful articles So this is like the kind of solution But if you saw across these solutions, you would have noticed 1 pattern the pattern has been this So there is a service you go pay money and then you use their API and then you pay them X money for the API calls that you make.

Whether it is any model, or whether it is like models like Cohere or Cloud, or as other services.

The other part of this equation is where you have a server like you rent a machine like 3090 you go to any hosting cloud provider run pod or something run pod is quite popular these days and you like rent a machine and that machine is basically your server and then you deploy something and then you run this as an API endpoint like any model that you want and the second part of this video is going to focus on this side of the story where you don't have to necessarily use any API that is available but rather if you want to have your own managed API what are the tools that can help you support you and do that thing.

The first thing that I think no-brainer is text generation web UI from UbaBuga.

So this is almost like the auto 1111 of stable diffusion world in the text generation.

This is 1 of the oldest, this is 1 of the most popular text generation web UI that is available.

But not a lot of people know that this web UI which is like basically like UI also offers you API server so that means if you deploy text generation web UI from Uwe Buga You also get to expose the model as an API endpoint That means you can basically do whatever we discussed in the first part of the video With this also and it also has a bunch of other things like XLama2 Which makes your inference faster supports quantized models has XLama2 support So these things make the inference faster.

And you may not know until you build a SaaS application that the time an API response or the time it takes for your API to hit the server and get a response back is very critical it can make your customers delightful it can make your customers pissed so speed is a very important term in this game and I don't know how many of you have read the book flash voice like if you're watching this video at this point If you have read the book flash voice, please let me know in the comment section It's about high frequency trading very interesting book from Michael Lewis Anyways speed is the name of the game.

So speed is very important so uber booger takes generation web UI can let you host any model.

And it also has got this additional support that I mentioned, and it gives you open AI compatible API server for chat and completions endpoints.

This is a solution that you can host it on any GPU server that you want like you rent a 3090, you rent a 100, you rent a 1000 anything that you host you can host this model and then start using it.

The fact that it supports quantized models also can help you save the cost.

The second thing is an interesting solution.

It's called sky pilot So this solution can help you run LLMs on any cloud now.

It's an open source project In fact, it's a project with Apache 2.0 license.

So this this by default is not hosting solution This is an open source project that lets you take any model that you want and then run it on any cloud that you want Like and it gives you a bunch of things like for example, it abstracts all the important things In 1 click you can launch the job or clusters on any cloud.

It lets you store the objects let's say in S3 or GCS or R2.

And it also helps you in maximizing the GPU availability with provision across all the regions and it helps you save cloud costs.

It has a lot of these advantages and I've seen a bunch of people recommending this on different forums so these are the cloud providers it supports AWS, Oracle, Google Cloud, Samsung has a cloud I didn't know Lambda Labs is another great solution Lambda Cloud, Azure, IBM Cloud, Cloudflare Kubernetes So it basically, kubernetes is not a cloud solution anyways.

Yeah, it basically, do they have a cloud offering?

So it basically lets you take the large language model in a productionized setup and go deploy it in any places.

Any of these cloud service and see it says like sky pilot in 1 minute.

They've got 1 of the best documentation I've seen like a lot of really good documentation.

How do you Run a QLORA and all these things all these kind of examples are available for you to write away get started This is an important open source project like proper open source licensing The next 1 is another project that for some reason I'm not seeing a lot of people using it except the open source project community.

So what they do is they give you 1 library it's called lightllm of course as the name suggests.

With this 1 I think you get like a compatibility with around like 100 plus API or something like that they've got so that means if you write your code that is wrapped on light LLM today you can use OpenAI tomorrow you can use cloud day after tomorrow you can use cohere, azure, AWS bedrock or any other API endpoints.

So with 1 structure 1 API structure it lets you use every other API I don't know how many of you have done classical machine learning This is 1 of the reason I always loved scikit-learn because you write scikit-learn you have XGBoost You want to change it to random for us just single line you change it.

You have a logistic regression change it It's it's It's a very structured API that can let you access all these models and also they're part of Y Combinator Again, I don't know like this is an open source project.

How how how are they going to make money?

Maybe they're going to partner with cloud providers but also you know you this is another good page where you can see all the cloud providers open AI, Azure, Together AI and you know you have got the Google Plum, AnyScale, Perplexity so for all these things available for you to use so this is like 1 endpoint and basically you can replace the OpenAI not endpoint 1 like wrapper basically you can replace anything there and you have like OpenAI compatible Model available from any of these solutions that are available.

So definitely check out light LLM The next 1 is ticks generation inference This is this is a project that had a bit of controversy back in the day So Currently this project you can see it's from offered by gradient.ai But this project was originally developed by Hugging Face Somewhere along the timeline Hugging Face decided to change the license of the project got a lot of backlash and you can read the history here.

So Hugging Face decided not to make it commercially available and then they forked this at that point and then they started developing this.

So currently this also supports quantization, this supports C translate too.

It supports chat completion in open a format and it gives you like one-click docker so it gives you all the opportunities and TGI it's like the short form is TGI you would have seen TGI in a lot of different places TGI is 1 of the popular most popular methods to deploy machine learning deploy large language models not machine learning models.

So if you want to use some tried and tested solutions something that a lot of people are already using definitely check out TGI.

This is not from Hugging Face this is from gradient AI.

The next 1 is probably the 1 that I would recommend the most and I've seen the fastest I know I know for sure like you know perplexity claimed or I think together AI claimed to be faster than VLLM.

I'm I've not seen it but VLLM is 1 thing that I have personally seen being ridiculously faster.

This is the definition of fast in terms of LLM inference and this is an amazing project Apache 2.0 license.

They have their own methodology about how they change the paging and all the other things to make LLM inferencing faster and they have got a bunch of solutions easily available for you to use.

If you want to use, if you have your custom server, custom GPU and you want to use large language models and you want the inference to be the fastest I think it's a no-brainer to try out with VLLM 1 of the fastest I've personally seen and It's also like open source project commercially available for you to use and they also support a lot of different hugging face models that means you get the flexibility you are not like tied up with a closed solution like a Claude or here you can basically use any existing model and not necessarily any there's a bunch of models that they support but out of all these models You can pick anything and then use it with VLLM That'll be like 1 of the fastest inference solutions that you have ever got.

I have not a lot of clue about how do you deploy it, but this is a solution that I've seen time and time again.

And unfortunately, I wanted to make a separate video about it.

MLC also lets you run these large language models on your smartphones, other devices, like handheld devices.

So MLC is 1 of the fastest again like it's a quantized bit but you would still see it is 1 of the fastest like I've seen some of the benchmarks where MLC is faster than VLLM but I don't think like there is any easy way or something that at least like I'm aware of and that can help you deploy MLC easily.

But if you figure out a way to deploy MLC, I think you've got like a really 1 of the fastest LLM deployment, LLM inferencing engine or solution for you to deploy.

So that's why I kept it at the last like even though This is the fastest I've kept it at the last just just so that you know, you know that This is a good solution, but I don't know finally before before I close out this video I want to turn your attention towards this amazing blog post by Hemel.

Hemel is quite involved in the MLOps community previously with Fast.ai and a lot of other things, strongly contributing to open source solutions and also publicly sharing the work.

So this blog post is really good 1 if you're optimizing for latency.

It doesn't necessarily talk about hosting solution but if you have an LLM, if you have got your own server, what do you deploy?

It has got a lot of details it has got a lot of nuances that you know somebody has tried and tested by deploying definitely read it I'll link it in the YouTube description all these links will be available in the YouTube description for you to try it out so to quickly summarize the first half of the video we learned about different hosted model solution the second half of the video we learned about open source projects that can help you if you have got your own server and if you have got you know place to host this models the whole idea of this is like apart from the open AMS currently happening you should never build a successful even before it becomes successful should never build a solution only relying on 1 API provider like just being the wrapper because a lot of things can happen like for example your prompts could be optimized for open AI but suddenly open AI goes down you cannot like immediately fall back with another service provider.

There are a lot of these kind of nuances that are available.

So it's always good to have this backup in mind.

And like I said, you don't have to always spend that money with GPT-4, GPT-4 Turbo or something.

You can always try with solutions that are much cheaper in api cost definitely compare the cost compare the speed compare the quality and then make a decision if you want to use only open source solution if you want to use only gpd kind of paid solutions or you want to have like a tiered approach like I said okay I'll have the first layer first level that is free not necessarily free like open solution that has like own hosting charges the second solution that is closed.

The good thing with the open solution is as much as the innovation happens You get to immediately update it like you can you can go on and update it with the closed solution You have to wait for example Claude or open AI You have to wait for them to give you something in return and then you have to then you know use it you have to wait for their update but on the flip side the advantages they take care of the security part they could take care of all the other things that you surely do not have to care about and in the open world you have to care about all those things.

It's always good to have options and that's why I made this video.

Let me know in the comment section what do you feel about it.

If you find it useful, please share it with your friends.

See all 1littlecoder transcripts on Youtube

STOP Building AI SaaS ONLY on OpenAI APIs!!!