Web Scraping Google Maps

Google Maps Data is a crucial piece of data for review software companies, sentimental analysts, and data miners as it contains information like user ratings, user reviews, addresses, and images of a particular place.

In this tutorial, we will learn how to scrape this valuable Google Maps information with the help of Node JS.

Table of Contents

Why Scrape Google Maps?

Scraping Google Maps can provide you with various types of benefits:

Lead Generation – Scraping Google Maps can help you collect emails and phone numbers in large numbers from various places, and you can use them to create your database of leads which can sell at a high price in the market.

Sentimental Analysis – Google Maps data can be used for analyzing the sentiment of the public based on the average ratings and reviews given by the public.

Location-Based Services – One can scrape Google Maps data to provide their users with services like finding nearby businesses, restaurants, pubs, and cafes based on the location of the user.

Requirements for scraping Google Maps:

Web Parsing with CSS selectors

Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.

This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.

User Agents

User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.
You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.

If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Websites.

Install Libraries

Before we begin, install these libraries so we can move forward and prepare our scraper.

Puppeteer JS

Or you can type the below commands in your project terminal to install the libraries:

npm i puppeteer

Let’s start scraping Google Maps:

Copy the below target URL to extract the HTML data:

https://www.google.com/maps/search/coffee/@28.6559457,77.1404218,11z

Coffee is our query. After that, we have our latitudes and longitudes. The number before z at the end is nothing but the zooming intensity of Google Maps. You can decrease or increase it as per your choice. Its value ranges from 2.92, in which the map completely zooms out, to 21, in which the map completely zooms in.

Note: Latitudes and longitudes are required to pass in the URL. But the zoom parameter is optional.

We will use Puppeteer Infinite Scrolling Method to scrape the Google Maps Results. So, let us start preparing our scraper.

First, let us create a main function that will launch the browser and navigate to the target URL.

const getMapsData = async () => {
        browser = await puppeteer.launch({
            headless: false,
            args: ["--disabled-setuid-sandbox", "--no-sandbox"],
        });
        const page = await browser.newPage();
        await page.setExtraHTTPHeaders({
            "User-Agent":
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4882.194 Safari/537.36",
        })
        
        
        await page.goto("https://www.google.com/maps/search/Starbucks/@26.8484046,75.7215344,12z/data=!3m1!4b1" , {
            waitUntil: 'domcontentloaded',
            timeout: 60000
        })
        
        await page.waitForTimeout(3000);
        
        let data =  await scrollPage(page,".m6QErb[aria-label]",2)
        
        console.log(data)
        await browser.close();
        };

Step-by-step explanation:

puppeteer.launch() – This will launch the Chromium browser with the options we have set in our code. In our case, we are launching our browser in non-headless mode.
browser.newPage() – This will open a new page or tab in the browser.
page.setExtraHTTPHeaders() – It is used to pass HTTP headers with every request the page initiates.
page.goto() – This will navigate the page to the specified target URL.
page.waitForTimeout() – It will cause the page to wait for 3 seconds to do further operations.
scrollPage() – At last, we called our infinite scroller to extract the data we need with the page, the tag for the scroller div, and the number of items we want as parameters.

Now, let us prepare the infinite scroller.

const scrollPage = async(page, scrollContainer, itemTargetCount) => {
        let items = [];
        let previousHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
        while (itemTargetCount > items.length) {
            items = await extractItems(page);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollTo(0, document.querySelector("${scrollContainer}").scrollHeight)`);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight > ${previousHeight}`);
            await page.waitForTimeout(2000);
        }
        return items;
    }

Step-by-step explanation:

previousHeight – Scroll the height of the container.
extractItems() – Function to parse the scraped HTML.
In the next step, we just scrolled down the container to a height equal to previousHeight.
And in the last step, we waited for the container to scroll down until its height got greater than the previous height.

And, at last, we will talk about our parser.

const extractItems = async(page)  => {
        let maps_data = await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".Nv2PK")).map((el) => {
            const link = el.querySelector("a.hfpxzc").getAttribute("href");
            return {
            title: el.querySelector(".qBF1Pd")?.textContent.trim(),
            avg_rating: el.querySelector(".MW4etd")?.textContent.trim(),
            reviews: el.querySelector(".UY7F9")?.textContent.replace("(", "").replace(")", "").trim(),
            address: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            description: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(2)")?.textContent.replace("·", "").trim(),
            website: el.querySelector("a.lcr4fd")?.getAttribute("href"),
            category: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            timings: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            phone_num: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            extra_services: el.querySelector(".qty3Ue")?.textContent.replaceAll("·", "").replaceAll("  ", " ").trim(),
            latitude: link.split("!8m2!3d")[1].split("!4d")[0],
            longitude: link.split("!4d")[1].split("!16s")[0],
            link,
            dataId: link.split("1s")[1].split("!8m")[0],
            };
        });
        });
        return maps_data;
        }

Step-by-step explanation:

document.querySelectorAll() – It will return all the elements that match the specified CSS selector. In our case, it is Nv2PK.
getAttribute() -This will return the attribute value of the specified element.
textContent – It returns the text content inside the selected HTML element.
split() – Used to split a string into substrings with the help of a specified separator and return them as an array.
trim() – Removes the spaces from the starting and end of the string.
replaceAll() – Replaces the specified pattern from the whole string.

Here is the full code:

const puppeteer = require('puppeteer');
const extractItems = async(page)  => {
        let maps_data = await page.evaluate(() => {
        return Array.from(document.querySelectorAll(".Nv2PK")).map((el) => {
            const link = el.querySelector("a.hfpxzc").getAttribute("href");
            return {
            title: el.querySelector(".qBF1Pd")?.textContent.trim(),
            avg_rating: el.querySelector(".MW4etd")?.textContent.trim(),
            reviews: el.querySelector(".UY7F9")?.textContent.replace("(", "").replace(")", "").trim(),
            address: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            description: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(2)")?.textContent.replace("·", "").trim(),
            website: el.querySelector("a.lcr4fd")?.getAttribute("href"),
            category: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(1) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            timings: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:first-child")?.textContent.replaceAll("·", "").trim(),
            phone_num: el.querySelector(".W4Efsd:last-child > .W4Efsd:nth-of-type(3) > span:last-child")?.textContent.replaceAll("·", "").trim(),
            extra_services: el.querySelector(".qty3Ue")?.textContent.replaceAll("·", "").replaceAll("  ", " ").trim(),
            latitude: link.split("!8m2!3d")[1].split("!4d")[0],
            longitude: link.split("!4d")[1].split("!16s")[0],
            link,
            dataId: link.split("1s")[1].split("!8m")[0],
            };
        });
        });
        return maps_data;
        }
    
        const scrollPage = async(page, scrollContainer, itemTargetCount) => {
        let items = [];
        let previousHeight = await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight`);
        while (itemTargetCount > items.length) {
            items = await extractItems(page);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollTo(0, document.querySelector("${scrollContainer}").scrollHeight)`);
            await page.evaluate(`document.querySelector("${scrollContainer}").scrollHeight > ${previousHeight}`);
            await page.waitForTimeout(2000);
        }
        return items;
        }
    
    
    
    const getMapsData = async () => {
        browser = await puppeteer.launch({
        headless: false,
        args: ["--disabled-setuid-sandbox", "--no-sandbox"],
        });
        const [page] = await browser.pages();
        await page.setExtraHTTPHeaders({
            "User-Agent":
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4882.194 Safari/537.36",
        })
        
    
        await page.goto("https://www.google.com/maps/search/Starbucks/@26.8484046,75.7215344,12z/data=!3m1!4b1" , {
            waitUntil: 'domcontentloaded',
            timeout: 60000
        })
    
        await page.waitForTimeout(5000)  
    
    let data =  await scrollPage(page,".m6QErb[aria-label]",2)
    
    console.log(data)
    await browser.close();
    };
    
    getMapsData();

Our result should look like this 👇🏻:

[
    {
        title: 'The Coffee Bean & Tea Leaf',
        avg_rating: '4.7',
        reviews: '79',
        address: 'The Coffee Bean & Tea Lea,Ground Floor, Epicuria Food Court, Plot No-10 Shivaji Place, Najafgarh Rd',
        description: 'Chain coffee bar known for frozen drinks',
        category: 'Coffee shop',
        timings: 'Open ⋅ Closes 11PM',
        phone_num: 'Open ⋅ Closes 11PM',
        extra_services: 'Dine-in   Drive-through   No-contact delivery      Reserve a table',
        latitude: '28.6511983',
        longitude: '77.1215014',
        link: 'https://www.google.com/maps/place/The+Coffee+Bean+%26+Tea+Leaf/data=!4m7!3m6!1s0x390ce3a69997ad37:0xff83fd9a57a7a71e!8m2!3d28.6511983!4d77.1215014!16s%2Fg%2F11sgxr14tq!19sChIJN62XmabjDDkRHqenV5r9g_8?authuser=0&hl=en&rclk=1',
        dataId: '0x390ce3a69997ad37:0xff83fd9a57a7a71e'
    },
    {
        title: 'The Coffee Bean & Tea Leaf',
        avg_rating: '4.0',
        reviews: '271',
        address: 'T320, Ambience Mall, Gurgaon - Delhi Expy',
        description: 'Chain coffee bar known for frozen drinks',
        category: 'Coffee shop',
        timings: 'Open ⋅ Closes 11PM',
        phone_num: 'Open ⋅ Closes 11PM',
        extra_services: 'Dine-in   Takeaway   No-contact delivery',
        latitude: '28.5041789',
        longitude: '77.0970538',
        link: 'https://www.google.com/maps/place/The+Coffee+Bean+%26+Tea+Leaf/data=!4m7!3m6!1s0x390d194c1a223247:0x611f25bf4fddaf08!8m2!3d28.5041789!4d77.0970538!16s%2Fg%2F11cs6ch67r!19sChIJRzIiGkwZDTkRCK_dT78lH2E?authuser=0&hl=en&rclk=1',
        dataId: '0x390d194c1a223247:0x611f25bf4fddaf08'
    },
    .....

Conclusion:

In this tutorial, we learned to scrape Google Maps Results using Node JS. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!

Why Scrape Google Maps?

Requirements for scraping Google Maps:

Web Parsing with CSS selectors

User Agents

Install Libraries

Let’s start scraping Google Maps:

Conclusion:

Additional Resources

Web Scraping – A Complete Guide

Web Scraping With Python

Web Scraping With Node JS – An Ultimate Guide

Web Scraping Amazon Product Data

Scrape Bing With Python

Scrape Google Scholar Using Python

Why Scrape Google Maps?

Requirements for scraping Google Maps:

Web Parsing with CSS selectors

User Agents

Install Libraries

Let’s start scraping Google Maps:

Conclusion:

Additional Resources

Similar Posts