Scrape Google Shopping Results

ByDarshan

In this post, we will learn to scrape Google Shopping Results using Node JS.

Scrape Google Shopping Results
Scrape Google Shopping Results

Requirements:

Web Parsing with CSS selectors


To search the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.

This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.

User Agents


User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.

You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.

If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Google.

Install Libraries


To scrape Google Shopping Results we need to install some NPM libraries so we can move forward.

  1. Unirest JS
  2. Cheerio JS

So before starting, we have to ensure that we have set up our Node JS project and installed both the packages — Unirest JS and Cheerio JS. You can install both packages from the above link.

Target:

We will target to scrape the shopping results of Nike shoes.

Let’s start scraping Google Shopping Results:


We have installed all the things which we will need for our scraper. Now we will hit our target URL using Unirest JS to get our HTML data and then we will parse our extracted HTML data with the help of Cheerio JS.
We will target this URL:

`https://www.google.com/search?q=nike shoes&tbm=shop&gl=us`

Look at the tbm parameter and its value(shop, here). This value shop will tell Google that we are looking for shopping results.

Open this URL in your browser. Inspect the code. You will see that every organic shopping result is inside this tag .sh-dgr__gr-auto.

Now, we will search the tags for title, product link, price, rating, reviews, delivery, and source.

The above images are in the pattern of two at the top and one at the bottom.

We have completed our search for tags of organic shopping results. Now, we will search for the tags of ad results.

If you inspect the ad results you will see that all the ad results are inside the tag .sh-np__click-target. This tag contains all the information about the title, link, price, and source.

All the above things make our code look like this:

const unirest = require("unirest");
    const cheerio = require("cheerio");
    
    const getShoppingData = () => {
      
      try
      {
      return unirest
        .get("https://www.google.com/search?q=nike shoes&tbm=shop&gl=us")
        .headers({
          "User-Agent":
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
        })
        .then((response) => {
          let $ = cheerio.load(response.body);
    
    
        let ads = [];
        
        $(".sh-np__click-target").each((i,el) => {
         ads.push({
            title: $(el).find(".sh-np__product-title").text(),
            link: "https://google.com" + $(el).attr("href"),
            source: $(el).find(".sh-np__seller-container").text(),
            price: $(el).find(".hn9kf").text(),
            delivery: $(el).find(".U6puSd").text(),
         })
         if($(el).find(".rz2LD").length)
         {
          let extensions = []
            extensions = $(el).find(".rz2LD").text()
            ads[i].extensions = extensions
         }
         })
    
        for (let i = 0; i < ads.length; i++) {
            Object.keys(ads[i]).forEach(key => ads[i][key] === "" ? delete ads[i][key] : {});  
        }
    
        let shopping_results = [];
    
        $(".sh-dgr__gr-auto").each((i,el) => {
            shopping_results.push({
                title: $(el).find(".Xjkr3b").text(),
                link: $(el).find(".zLPF4b .eaGTj a.shntl").attr("href").substring($(el).find("a.shntl").attr("href").indexOf("=")+1),
                source: $(el).find(".IuHnof").text(),
                price: $(el).find(".XrAfOe .a8Pemb").text(),
                rating: $(el).find(".Rsc7Yb").text(),
                reviews: $(el).find(".NzUzee div").attr("aria-label") ? $(el).find(".NzUzee div").attr("aria-label").substring(0,$(el).find(".NzUzee div").attr("aria-label").indexOf(" ")) : "",
                delivery: $(el).find(".vEjMR").text()
            })
            if($(el).find(".Ib8pOd").length)
            {
                let extensions = [];
                extensions = $(el).find(".Ib8pOd").text();
                shopping_results[i].extensions = extensions
            }
        })
    
        for (let i = 0; i < shopping_results.length; i++) {
            Object.keys(shopping_results[i]).forEach(key => shopping_results[i][key] === "" ? delete shopping_results[i][key] : {});  
         }
         
         console.log(ads)
         console.log(shopping_results)
        })
      }
     catch(e)
     {
        console.log(e)
     }
    }
    
    
    getShoppingData();

You can also check some of my other Google scrapers in my Git Repository: https://github.com/Darshan972/GoogleScrapingBlogs

Result:

Our result should look like this 👆🏻.

With Google Shopping API


If you don’t want to code and maintain the scraper in the long run, then you can try our Google Shopping API to scrape shopping results.

We also offer 100 free requests on the first sign-up.

const axios = require('axios');
    
    axios.get('https://api.serpdog.io/shopping?api_key=APIKEY&q=shoes&gl=us')
      .then(response => {
        console.log(response.data);
      })
      .catch(error => {
        console.log(error);
    });

Result:

Conclusion:


In this tutorial, we learned to scrape Google Shopping Results using Node JS. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources

  1. How to scrape Google Organic Search Results using Node JS?
  2. Scrape Google Images Results
  3. Scrape Google News Results
  4. Scrape Google Maps Reviews

Frequently Asked Questions

Q. How do I get Google Shopping results?


You can get Google Shopping Results by using Serpdog Google Shopping API without any problem with proxies and CAPTCHAs. This data is one of the great sources for data miners for competitor price tracking, sentimental analysis, etc.

Similar Posts