Scrape Google Shopping Results

This post will be a step-by-step guide for scraping Google Shopping Results using Node JS and its libraries.

Why Scrape Google Shopping?

Scraping Google Shopping is essential if you want to avail the following benefits:

Price Monitoring – Scrape Google Shopping to monitor the pricing of a particular product from various sources, and compare them to get the cheapest source, which also saves you money.

Product Information – Get detailed information about the products from Google Shopping, and compare their reviews and features with various other products to find the best from them.

Product Availability – You can use Google Shopping data to monitor the availability of a set of products instead of manually checking the product from different sources, which consumes your valuable time.

Let’s start scraping Google Shopping:

In this section, we will be scraping Google Shopping Results. But let us first complete the requirements for this project.

Web Parsing with CSS selectors

Searching for tags in HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.

This gadget can help you develop the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.

Install Libraries

To scrape Google Shopping Results we need to install some NPM libraries so we can move forward.

1. Unirest JS
2. Cheerio JS

So before starting, we have to ensure that we have set up our Node JS project and installed both the packages — Unirest JS and Cheerio JS. You can install both packages from the above link.

Process

We have installed all the things which we will need for our scraper. Now we will hit our target URL using Unirest JS to get our HTML data and then we will parse our extracted HTML data with the help of Cheerio JS.
We will target this URL:

`https://www.google.com/search?q=nike+shoes&tbm=shop&gl=us`

Look at the tbm parameter and its value(shop, here). This value shop will tell Google that we are looking for shopping results.

Open this URL in your browser. Inspect the code. You will see that every organic shopping result is inside this tag .sh-dgr__gr-auto.

Now, we will search the tags for title, product link, price, rating, reviews, delivery, and source.

The above images are in the pattern of two at the top and one at the bottom.

We have completed our search for tags of organic shopping results. Now, we will search for the tags of ad results.

If you inspect the ad results you will see that all the ad results are inside the tag .sh-np__click-target. This tag contains all the information about the title, link, price, and source.

All the above things make our code look like this:

const unirest = require("unirest");
    const cheerio = require("cheerio");
    
    const getShoppingData = () => {
      
      try
      {
      return unirest
        .get("https://www.google.com/search?q=nike+shoes&tbm=shop&gl=us")
        .headers({
          "User-Agent":
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
        })
        .then((response) => {
          let $ = cheerio.load(response.body);
    
    
        let ads = [];
        
        $(".sh-np__click-target").each((i,el) => {
         ads.push({
            title: $(el).find(".sh-np__product-title").text(),
            link: "https://google.com" + $(el).attr("href"),
            source: $(el).find(".sh-np__seller-container").text(),
            price: $(el).find(".hn9kf").text(),
            delivery: $(el).find(".U6puSd").text(),
         })
         if($(el).find(".rz2LD").length)
         {
          let extensions = []
            extensions = $(el).find(".rz2LD").text()
            ads[i].extensions = extensions
         }
         })
    
        for (let i = 0; i < ads.length; i++) {
            Object.keys(ads[i]).forEach(key => ads[i][key] === "" ? delete ads[i][key] : {});  
        }
    
        let shopping_results = [];
    
        $(".sh-dgr__gr-auto").each((i,el) => {
            shopping_results.push({
                title: $(el).find(".Xjkr3b").text(),
                link: $(el).find(".zLPF4b .eaGTj a.shntl").attr("href").substring($(el).find("a.shntl").attr("href").indexOf("=")+1),
                source: $(el).find(".IuHnof").text(),
                price: $(el).find(".XrAfOe .a8Pemb").text(),
                rating: $(el).find(".Rsc7Yb").text(),
                reviews: $(el).find(".NzUzee div").attr("aria-label") ? $(el).find(".NzUzee div").attr("aria-label").substring(0,$(el).find(".NzUzee div").attr("aria-label").indexOf(" ")) : "",
                delivery: $(el).find(".vEjMR").text()
            })
            if($(el).find(".Ib8pOd").length)
            {
                let extensions = [];
                extensions = $(el).find(".Ib8pOd").text();
                shopping_results[i].extensions = extensions
            }
        })
    
        for (let i = 0; i < shopping_results.length; i++) {
            Object.keys(shopping_results[i]).forEach(key => shopping_results[i][key] === "" ? delete shopping_results[i][key] : {});  
         }
         
         console.log(ads)
         console.log(shopping_results)
        })
      }
     catch(e)
     {
        console.log(e)
     }
    }
    
    
    getShoppingData();

You can also check some of my other Google scrapers in my Git Repository: https://github.com/Darshan972/GoogleScrapingBlogs

Results:

Our result should look like this 👆🏻.

With Google Shopping API

If you don’t want to code and maintain the scraper in the long run, then you can try our Google Shopping API to scrape shopping results.

We also offer 100 free requests on the first sign-up.

After registering successfully on Serpdog, embed your API Key in the below code you will be able to scrape Google Shopping Results without any blockage.

const axios = require('axios');
    
    axios.get('https://api.serpdog.io/shopping?api_key=APIKEY&q=shoes&gl=us')
      .then(response => {
        console.log(response.data);
      })
      .catch(error => {
        console.log(error);
    });

Conclusion:

In this tutorial, we learned to scrape Google Shopping Results using Node JS. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources

1. How to scrape Google Organic Search Results using Node JS?
2. Scrape Google Images Results
3. Scrape Google News Results
4. Scrape Google Maps Reviews

Frequently Asked Questions

Q. How do I get Google Shopping results?

You can get Google Shopping Results by using Serpdog Google Shopping API without any problem with proxies and CAPTCHAs. This data is one of the great sources for data miners for competitor price tracking, sentimental analysis, etc.