In this video, we will look at Puppeteer to scrape data from a web page.

💻 Code:
https://github.com/bradtraversy/courses-scrape

Puppeteer Docs:
https://pptr.dev/

⭐ All Courses:
https://traversymedia.com

💖  Show Support
Patreon: https://www.patreon.com/traversymedia
PayPal: https://paypal.me/traversymedia

👇 Follow Traversy Media On Social Media:
Twitter: https://twitter.com/traversymedia
Instagram: https://www.instagram.com/traversymedia
Linkedin: https://www.linkedin.com/in/bradtraversy

Timestamps:
0:00 - Intro
0:36 - Install & Setup
3:36 - Init Browser & Page Object
5:02 - Screenshot & PDF
6:54 - Targeting HTML, Text, and Links
11:22 - Scraping Courses
17:08 - $$eval()
18:40 - Save JSON Data

So in this video, I'm going to give you an introduction to web scraping with a tool called Puppeteer.

Now, if you're watching this video, I'm sure you know what a data API is, where you have different endpoints that you can hit with an HTTP request, and you can fetch some formatted data, whether that's from a public API or a private API.

But what if the data you want isn't available in any kind of API?

Well, in many cases, you can scrape that data yourself.

And there's a lot of different tools that you can do this with.

It's used for more than just web scraping.

It's essentially a headless Chrome browser.

So anything you can do in the browser normally, you can basically do that programmatically through Puppeteer.

So you have complete access to the DOM, you can fire off events, you can parse JavaScript, you can create screenshots and PDFs of websites.

So it's really cool and I would encourage you guys to look further into it if you enjoy this video.

Now what we're gonna do is scrape all of the courses from my homepage.

So at TraverseeMedia.com, we're gonna take all the courses and get the title, the Udemy link, the course level, and the promo code for all the courses and put them into a JSON array and then save that to a file on my system.

Alright so obviously there's different things you can do with different data but I'm just going to show you how you can how you can get that data.

Alright guys so we're gonna get into this and I would recommend that you follow along.

You don't have to, but I would recommend it.

So this is the Puppeteer website where you can find all the documentation.

Now what we want to scrape, the data we want to get is at traversymedia.com we have these courses.

So all these courses right here, I want to get them.

I want to get the title, the level, the URL for the Udemy link, and then also the promo code.

So I want to get all that, put it into JSON objects in an array, and save it to a file.

I'm going to show you a couple other things along the way that we can do with Puppeteer.

So I have VS Code open, just an empty folder called Courses Scrape.

And of course you need Node.js installed.

If you don't have it, just go to nodejs.org, download and install it, and you'll get Node as well as NPM, which is the Node Package Manager.

So we're going to first run npm init and just add a dash y so we don't have to go through the questions and that will just initialize a package.json file which holds our scripts and dependencies and all that And then the only dependency we want to install with npm install or npm I is puppeteer.

So let's do that and that will create our node modules folder with puppeteer and all its dependencies and it'll get added here.

Now, the file that, basically our entry point file, I'm gonna create that and call it index.js.

And then I'm gonna create a start script for that.

You don't have to, you can just run node index, but I'm just going to make it so we can run npm start.

Alright, so if we save that and then in our index.js, we'll just do a console log.

And then down here in our terminal, go NPM start and we should see that log Okay.

Now the first thing we want to do of course is bring in puppeteer.

We want to require the package and then all the all the objects that we're going to use, all the methods we're going to use are going to be asynchronous.

So we're going to have everything in an asynchronous function.

I'm going to call it run and then we're going to run it.

Now the first thing we want to do here is open the browser or launch a browser.

So we're going to create a variable called browser and then set that to await puppeteer.launch.

So this is essentially just launching a browser programmatically so that we can do things like access different pages and elements on the pages, fire off events, things like that.

Now to access a page we need to initialize a page variable and we can do that with await and then browser and then there's a new page method.

Okay now to go to a specific page we want to await on page.goto and then we can pass in here the URL of the page that we want to go to which in this case is going to be traversymedia.com.

Now before I do anything else I just want to go to the bottom here and make sure I close the browser and we do that with browser.close Okay and then within here is where we can access DOM elements and do pretty much anything we want.

Now there's some some other cool things I want to show you before we get into targeting content and data.

So first of all we can create a screenshot and we can do that by a weight Page dot and then there's a screenshot Method that we can call and we just pass in an object with a path which is going to be Where we want this to go.

So let's say example dot PNG Okay, so if I run npm start, it's going to run our index file.

It takes 2 seconds or so, and then you can see up here example.png has been created.

Now it's going to set it to a specific size, and it's not the whole page.

What you can do is you can add in another property here of full page and set that to true.

If we do that, and then I run it again, it's going to overwrite the initial example PNG and now you can see we get the whole website as an image.

Now you can do the same thing with a PDF.

So I'll just copy this down, comment this 1 out and instead of page.screenshot we can do page.pdf.

I'll change the extension here to pdf and then instead of full page we can put in a pdf format and we'll just do h, not H, A4 and if I save that we run this file again and now we can see we have a PDF so if I bring that over and open it up you can see it's not gonna have like the the same exact style or whatever but it has all the content so if you need to for some reason generate a PDF of a specific web page, you can do that.

Now I want to get into targeting some of the content now.

And let's look at how to get the entire HTML of a page.

So we'll create a variable called HTML and we're going to set that to await and then there's a method called content that we can call that will give us all of the HTML.

So I'm just going to console log that and then we'll come down here and run and now you can see all the HTML for my home page is in the console right now and of course you can do whatever you want with this I'm just console logging it.

So that'll get the HTML, let's comment that out.

Now if you want to get the title or really anything, if you want to target H3s or whatever, there's a method called evaluate on the page object.

So what we could do is say page dot evaluate, oops, evaluate and this is a high order function so we can pass in a function here and then we have access to the document object.

So I can say just document.title and let's console.log title here.

Alright and then we'll come down we'll run this and we should just see the title of the page or not and that's because I forgot await and there we go so Traverse Media Learn Web Development so we can get the title if we want to get let's say all of the text on the page, we could do that.

And we'll just say const text and set that to await page.evaluate.

So once again, we're going to use evaluate, pass in a function and we have access to the document object.

So I can simply say document dot body and then we can get the inner text.

And then we'll just we'll just console log text.

Okay so come back down here and run it, we should just get all the text.

You can see this is the stuff in my footer, Patreon, YouTube channel, so all the courses.

Now let's say I want to get all the links on the page so I'm going to comment that out and let's say const links and we're going to set that to a weight and again we're going to use whoops we're going to use page dot evaluate now where we're getting all the links, we're getting multiple elements, right?

So instead of using, like we could do query selector and get any single element, but we wanna get all the links.

So we're gonna be using query selector all.

It's the same way you would access the DOM in you know front-end JavaScript now query selector all will give us something called a node list so We're going to pass in our function here, and we want to wrap our query selector all in the array.from method, which will basically create a shallow copy of an array from an iterable object, such as a node list.

Now array.from takes in a second argument, which is going to be a function.

And we can pass in an argument here that represents the element.

And we want to say for each 1, we want to get the href or the URL.

So we can take that element and get href.

Okay, so that should just give us all the links.

And then let's just do a console log here of links.

And if we come down here and run that, there we go.

So these are all the links that are on that page.

Now we can do the same, basically the same thing to get all the courses, we just need to be a little bit more specific than just, you know, all the A tags.

So what I'll do is let's actually copy this and then we'll comment it out and then we'll paste that in and let's change links to courses, right?

And then we're still going to do the query selector all, but now we actually have to look at the structure of the website in order to scrape it because we need to know what to put in here.

So let's go to traversymedia.com and I'm going to open up my dev tools and let's just take a look because we know we want to get these courses.

So if we look at the HTML, you can see there's a section with the ID of courses.

So they're all contained in that and if we look further into it, every course has a, and down here as well where it says more, every course has a class of card around it.

And if we look into that, every card has a class of card body and card footer.

Now the data we want is in both of these divs.

So in the card body, we have the title, which is in an H3, and we have the level, which is in a div with the class of level.

Then in the footer is the, let's see, is the link that we want, the Udemy link.

And then in this div with the class of promo code is our promo code which is actually in another class called promo.

Yes we want to get that as well so you really have to kind of you know go into the structure of whatever website that you're that you're scraping.

So now that we know that let's jump back in here and instead of just all the a tags let's let's get the let's say courses so the idea of courses member that section wraps around all of them, and then we want to get all the cards that are in that.

And then here, instead of just returning all the href, I want to return an object of courses.

So since I want to return an object, I'm going to put some parentheses here, otherwise it's going to just be looked at as a code block.

So I'm going to put then my object in here like that.

So whatever properties I put in here will get returned.

Now inside this card, we have 2 other elements that are wrapping the fields we want.

So here what I'm going to do is take the element which is going to be the card, right, so say E, and then we want to then use query selector on that because we want to go into the card body, so card dash body, and then we know the title is in an H3 in the card body, so I want to get H3, and then finally I want the text, so I'm going to use inner text.

Alright, so that should give me the title, and we can actually, We can actually just try this out and let's console log courses and then we'll come down here and run this.

Okay, so it looks like I screwed something up.

Evaluation field cannot read properties, null, enter text.

Oh, I forgot the dot for the class right strata again and there we go so now we have in a ray that has all that that course titles which is pretty cool Now I also want to get the level.

Remember the level is also in the card body right here.

So what we can do is just copy this down and let's change title to level and change the h3 to the class of level.

So that will get us to level and then we'll just keep going here now the other 2 things we want the link and the promo coder in the footer so let's do URL so here we're going to do E.

And then query selector and this time we want to look in the card dash footer and we want the a tag that's in the footer and we want the link so I'm going to do dot href.

And then finally we want the promo code, so let's call this promo.

And we're going to say e.querySelector and in the card footer we want the, let's say, promo-code class.

Now if I do that, if I just do that, promo-code, it's gonna also include this right here, this text of code and then the colon.

We wanna go deeper into it, into this promo class, which will contain the actual code.

So let's add onto this dot promo and that should do it.

So we'll save that and then let's run this again And there we go.

So we have the title, the level, the URL.

The promo code isn't where I forgot the inner text.

Because All that does is select the element.

We need to get the text from the element.

Now There is another syntax we can use to do this without using array.from.

So we'll say courses and we're still going to use our page object and instead of using evaluate we can do this double money sign and then eval.

And then we can put right in here what we're targeting, which is going to be the ID of courses and then all of the cards within.

All right, so we're doing this without the initial array.from and document query selector.

And then this takes in a second argument which is going to be a function.

And we're going to pass in here elements.

And then this is going to be, we're going to then have to take those elements and then map through them.

So we're just doing what we did above in a different way.

That's going to take in a function and this is where we want to return our object from so I'm going to have parentheses around the curly braces.

And then we should be able to do pretty much the same thing we did here so we can just copy those and then uncomment these.

And it should work the same way because we're passing in E here, so we have access to that.

And then we're going to use query selector.

We'll save, and then if I run npm start, we should get the same, yep, we get the same thing.

All right, now, instead of just logging these to the console, let's actually save these to a file.

Now, to do that with Node.js, we're going to use the FS or the file system module.

So let's say const FS and set that to require and we want to bring in the FS module.

And let's come back down here and then let's say save data to a JSON file.

So we can do that by saying FS and then we're going to use the write file method and pass in a couple things.

We want the name of our file, so courses.json.

And then it needs to be valid JSON before we save it, so we're going to run it through json.stringify.

And let's pass in here courses which is our array and then this will take in a callback with a possible error okay and then let's say whoops let's say if there's an error then we'll just, let's say throw error, and then we'll just do a console log here and say file saved.

Let's open up our sidebar here and let's run npm start.

It's going to go and fetch, or not really fetch, but scrape the data and then put it into that array and then it's gonna put it into this courses.json file.

And if I save that, Prettier will make it pretty.

Okay, and now I could do whatever I want with this data.

And you might just do it for personal reasons.

Maybe you wanna keep up to date with all the codes.

They're just like the course name or topic and then the month and year.

But if these were like crazy codes, you might wanna just be able to run a command and fetch all the codes there's just a million reasons why you might want to scrape data alright so that's pretty much it there's a lot more to puppeteer There's a lot more you can do.

You can fire off events and stuff like that.

Like I said, I think I said that I may do a more in-depth course, but I wanted this to just be kind of an introduction to data scraping so that if there is data you wanna get and use in some way, but there's no API, you have a way to do it yourself.

All right, thanks guys for watching and I'll see you next time.


Speaker 0: Hey, what's going on guys? So in this video, I'm going to give you an introduction to web scraping with a tool called Puppeteer. Now, if you're watching this video, I'm sure you know what a data API is, where you have different endpoints that you can hit with an HTTP request, and you can fetch some formatted data, whether that's from a public API or a private API. But what if the data you want isn't available in any kind of API? Well, in many cases, you can scrape that data yourself.

And there's a lot of different tools that you can do this with. But Puppeteer is extremely powerful. It's used for more than just web scraping. It's essentially a headless Chrome browser. So anything you can do in the browser normally, you can basically do that programmatically through Puppeteer.

So you have complete access to the DOM, you can fire off events, you can parse JavaScript, you can create screenshots and PDFs of websites. So it's really cool and I would encourage you guys to look further into it if you enjoy this video. Now what we're gonna do is scrape all of the courses from my homepage. So at TraverseeMedia.com, we're gonna take all the courses and get the title, the Udemy link, the course level, and the promo code for all the courses and put them into a JSON array and then save that to a file on my system. Alright so obviously there's different things you can do with different data but I'm just going to show you how you can how you can get that data.

Alright so let's get started. Alright guys so we're gonna get into this and I would recommend that you follow along. You don't have to, but I would recommend it. So this is the Puppeteer website where you can find all the documentation. It's pptr.dev.

Now what we want to scrape, the data we want to get is at traversymedia.com we have these courses. So all these courses right here, I want to get them. I want to get the title, the level, the URL for the Udemy link, and then also the promo code. So I want to get all that, put it into JSON objects in an array, and save it to a file. I'm going to show you a couple other things along the way that we can do with Puppeteer.

So I have VS Code open, just an empty folder called Courses Scrape. And of course you need Node.js installed. If you don't have it, just go to nodejs.org, download and install it, and you'll get Node as well as NPM, which is the Node Package Manager. So we're going to first run npm init and just add a dash y so we don't have to go through the questions and that will just initialize a package.json file which holds our scripts and dependencies and all that And then the only dependency we want to install with npm install or npm I is puppeteer. So let's do that and that will create our node modules folder with puppeteer and all its dependencies and it'll get added here.

Now, the file that, basically our entry point file, I'm gonna create that and call it index.js. You can call it whatever you'd like. And then I'm gonna create a start script for that. You don't have to, you can just run node index, but I'm just going to make it so we can run npm start. And here we want to run node index.

Alright, so if we save that and then in our index.js, we'll just do a console log. And then down here in our terminal, go NPM start and we should see that log Okay. Now the first thing we want to do of course is bring in puppeteer. So let's do that. We want to require the package and then all the all the objects that we're going to use, all the methods we're going to use are going to be asynchronous.

So we're going to have everything in an asynchronous function. So let's create an async function. You can call it whatever you want. I'm going to call it run and then we're going to run it. Now the first thing we want to do here is open the browser or launch a browser.

So we're going to create a variable called browser and then set that to await puppeteer.launch. So this is essentially just launching a browser programmatically so that we can do things like access different pages and elements on the pages, fire off events, things like that. Now to access a page we need to initialize a page variable and we can do that with await and then browser and then there's a new page method. Okay now to go to a specific page we want to await on page.goto and then we can pass in here the URL of the page that we want to go to which in this case is going to be traversymedia.com. Now before I do anything else I just want to go to the bottom here and make sure I close the browser and we do that with browser.close Okay and then within here is where we can access DOM elements and do pretty much anything we want.

Now there's some some other cool things I want to show you before we get into targeting content and data. So first of all we can create a screenshot and we can do that by a weight Page dot and then there's a screenshot Method that we can call and we just pass in an object with a path which is going to be Where we want this to go. So let's say example dot PNG Okay, so if I run npm start, it's going to run our index file. It takes 2 seconds or so, and then you can see up here example.png has been created. Now it's going to set it to a specific size, and it's not the whole page.

What you can do is you can add in another property here of full page and set that to true. If we do that, and then I run it again, it's going to overwrite the initial example PNG and now you can see we get the whole website as an image. Now you can do the same thing with a PDF. So I'll just copy this down, comment this 1 out and instead of page.screenshot we can do page.pdf. I'll change the extension here to pdf and then instead of full page we can put in a pdf format and we'll just do h, not H, A4 and if I save that we run this file again and now we can see we have a PDF so if I bring that over and open it up you can see it's not gonna have like the the same exact style or whatever but it has all the content so if you need to for some reason generate a PDF of a specific web page, you can do that.

Now I want to get into targeting some of the content now. So let's comment that PDF line out. And let's look at how to get the entire HTML of a page. So we'll create a variable called HTML and we're going to set that to await and then there's a method called content that we can call that will give us all of the HTML. So I'm just going to console log that and then we'll come down here and run and now you can see all the HTML for my home page is in the console right now and of course you can do whatever you want with this I'm just console logging it.

So that'll get the HTML, let's comment that out. Now if you want to get the title or really anything, if you want to target H3s or whatever, there's a method called evaluate on the page object. So let's say we want to get the title. So what we could do is say page dot evaluate, oops, evaluate and this is a high order function so we can pass in a function here and then we have access to the document object. So I can say just document.title and let's console.log title here.

Alright and then we'll come down we'll run this and we should just see the title of the page or not and that's because I forgot await and there we go so Traverse Media Learn Web Development so we can get the title if we want to get let's say all of the text on the page, we could do that. Let's comment that out. And we'll just say const text and set that to await page.evaluate. So once again, we're going to use evaluate, pass in a function and we have access to the document object. So I can simply say document dot body and then we can get the inner text.

And then we'll just we'll just console log text. Okay so come back down here and run it, we should just get all the text. You can see this is the stuff in my footer, Patreon, YouTube channel, so all the courses. So it's just all the text, right? Now let's say I want to get all the links on the page so I'm going to comment that out and let's say const links and we're going to set that to a weight and again we're going to use whoops we're going to use page dot evaluate now where we're getting all the links, we're getting multiple elements, right?

So instead of using, like we could do query selector and get any single element, but we wanna get all the links. So we're gonna be using query selector all. It's the same way you would access the DOM in you know front-end JavaScript now query selector all will give us something called a node list so We're going to pass in our function here, and we want to wrap our query selector all in the array.from method, which will basically create a shallow copy of an array from an iterable object, such as a node list. So in here, let's say document. And then query select all.

And we want to select all A tags. Let's close this up. So we'll pass in here A. Now array.from takes in a second argument, which is going to be a function. And we can pass in an argument here that represents the element.

And we want to say for each 1, we want to get the href or the URL. So we can take that element and get href. Okay, so that should just give us all the links. And then let's just do a console log here of links. And if we come down here and run that, there we go.

Make this a little bigger so we can see. So these are all the links that are on that page. Now we can do the same, basically the same thing to get all the courses, we just need to be a little bit more specific than just, you know, all the A tags. So what I'll do is let's actually copy this and then we'll comment it out and then we'll paste that in and let's change links to courses, right? And then we're still going to do the query selector all, but now we actually have to look at the structure of the website in order to scrape it because we need to know what to put in here.

So let's go to traversymedia.com and I'm going to open up my dev tools and let's just take a look because we know we want to get these courses. So if we look at the HTML, you can see there's a section with the ID of courses. So they're all contained in that and if we look further into it, every course has a, and down here as well where it says more, every course has a class of card around it. And if we look into that, every card has a class of card body and card footer. Now the data we want is in both of these divs.

So in the card body, we have the title, which is in an H3, and we have the level, which is in a div with the class of level. Then in the footer is the, let's see, is the link that we want, the Udemy link. And then in this div with the class of promo code is our promo code which is actually in another class called promo. Yes we want to get that as well so you really have to kind of you know go into the structure of whatever website that you're that you're scraping. So now that we know that let's jump back in here and instead of just all the a tags let's let's get the let's say courses so the idea of courses member that section wraps around all of them, and then we want to get all the cards that are in that.

And then here, instead of just returning all the href, I want to return an object of courses. So since I want to return an object, I'm going to put some parentheses here, otherwise it's going to just be looked at as a code block. So I'm going to put then my object in here like that. So whatever properties I put in here will get returned. So I want, we know we want the title.

Now inside this card, we have 2 other elements that are wrapping the fields we want. We have a card body and a card footer. So here what I'm going to do is take the element which is going to be the card, right, so say E, and then we want to then use query selector on that because we want to go into the card body, so card dash body, and then we know the title is in an H3 in the card body, so I want to get H3, and then finally I want the text, so I'm going to use inner text. Alright, so that should give me the title, and we can actually, We can actually just try this out and let's console log courses and then we'll come down here and run this. Okay, so it looks like I screwed something up.

Evaluation field cannot read properties, null, enter text. Oh, I forgot the dot for the class right strata again and there we go so now we have in a ray that has all that that course titles which is pretty cool Now I also want to get the level. Remember the level is also in the card body right here. It's in the class of level. So what we can do is just copy this down and let's change title to level and change the h3 to the class of level.

So that will get us to level and then we'll just keep going here now the other 2 things we want the link and the promo coder in the footer so let's do URL so here we're going to do E. And then query selector and this time we want to look in the card dash footer and we want the a tag that's in the footer and we want the link so I'm going to do dot href. Alright so that should give us the URL. And then finally we want the promo code, so let's call this promo. And we're going to say e.querySelector and in the card footer we want the, let's say, promo-code class.

Now if I do that, if I just do that, promo-code, it's gonna also include this right here, this text of code and then the colon. We wanna go deeper into it, into this promo class, which will contain the actual code. So let's add onto this dot promo and that should do it. So we'll save that and then let's run this again And there we go. So we have the title, the level, the URL.

The promo code isn't where I forgot the inner text. Because All that does is select the element. We need to get the text from the element. So now if we do that, it should show. There we go.

So now we have all of the promo codes. Now There is another syntax we can use to do this without using array.from. So I want to show you that as well. But it will get us the same thing. So we'll say courses and we're still going to use our page object and instead of using evaluate we can do this double money sign and then eval.

And then we can put right in here what we're targeting, which is going to be the ID of courses and then all of the cards within. All right, so we're doing this without the initial array.from and document query selector. And then this takes in a second argument which is going to be a function. And we're going to pass in here elements. And then this is going to be, we're going to then have to take those elements and then map through them.

So we're going to use dot map here. So we're just doing what we did above in a different way. And then we'll just pass in E. So for each element, we'll just say E. That's going to take in a function and this is where we want to return our object from so I'm going to have parentheses around the curly braces.

And then we should be able to do pretty much the same thing we did here so we can just copy those and then uncomment these. And it should work the same way because we're passing in E here, so we have access to that. And then we're going to use query selector. All right, so let's try that. We'll save, and then if I run npm start, we should get the same, yep, we get the same thing.

All right, now, instead of just logging these to the console, let's actually save these to a file. Now, to do that with Node.js, we're going to use the FS or the file system module. So let's say const FS and set that to require and we want to bring in the FS module. And let's come back down here and then let's say save data to a JSON file. So we can do that by saying FS and then we're going to use the write file method and pass in a couple things.

We want the name of our file, so courses.json. And then it needs to be valid JSON before we save it, so we're going to run it through json.stringify. And let's pass in here courses which is our array and then this will take in a callback with a possible error okay and then let's say whoops let's say if there's an error then we'll just, let's say throw error, and then we'll just do a console log here and say file saved. Alright so that should do it. Let's open up our sidebar here and let's run npm start.

It's going to go and fetch, or not really fetch, but scrape the data and then put it into that array and then it's gonna put it into this courses.json file. And if I save that, Prettier will make it pretty. Okay, and now I could do whatever I want with this data. And you might just do it for personal reasons. Maybe you wanna keep up to date with all the codes.

My codes are pretty easy to remember. They're just like the course name or topic and then the month and year. But if these were like crazy codes, you might wanna just be able to run a command and fetch all the codes there's just a million reasons why you might want to scrape data alright so that's pretty much it there's a lot more to puppeteer There's a lot more you can do. You can fire off events and stuff like that. Like I said, I think I said that I may do a more in-depth course, but I wanted this to just be kind of an introduction to data scraping so that if there is data you wanna get and use in some way, but there's no API, you have a way to do it yourself.

All right, thanks guys for watching and I'll see you next time.

See all Traversy Media transcripts on Youtube

Intro To Web Scraping With Puppeteer