Scrape Google Autocomplete Suggestions

In This Post, We Will Learn To Scrape Google Autocomplete Suggestions Using Node JS.

Scrape Google Autocomplete Results
Scrape Google Autocomplete Results

Requirements:

User Agents


User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.

You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.

If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Google.

Install Libraries


To scrape Google maps reviews we need to install some NPM libraries so we can move forward.

  1. Unirest JS
  2. Cheerio JS

So before starting, we have to ensure that we have set up our Node JS project and installed both the packages — Unirest JS and Cheerio JS. You can install both packages from the above link.

Target:

We will target to scrape the shopping results of Nike shoes.

Let’s start scraping Google Autocomplete Results


Now, all the things are installed that we will need to prepare our scraper. We will use an npm library Unirest JS to make a get request to our target URL so we can get our raw HTML data. Then we will use Cheerio JS for parsing the extracted HTML data.

We will target this URL:

https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome

Copy this URL into your browser and press enter. You will see a text file downloading in your browser by entering this URL in your browser. Open this file in your respective code editor. We will now first convert this JSON string into an object and get the respective data we need.

const unirest = require("unirest");
const searchSuggestions = async () => {
try {
const response = await unirest
.get("https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome")
.headers({
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
})

let data = JSON.parse(response.body);
let suggestions = [];
for (let i = 0; i < data[1].length; i++) {
suggestions.push({
value: data[1][i],
relevance: data[4]["google:suggestrelevance"][i],
type: data[4]["google:suggesttype"][i],
});
}
const verbatimRelevance = data[4]["google:verbatimrelevance"]

console.log(suggestions)
console.log(verbatimRelevance)

} catch (e) {
console.log("Error : " + e);
}
};

searchSuggestions();

You can also check some of my other Google scrapers on my Git Repository: https://github.com/Darshan972/GoogleScrapingBlogs

Result:

[
{ value: 'coffee near me', relevance: 1250, type: 'QUERY' },
{ value: 'coffee shops near me', relevance: 650, type: 'QUERY' },
{ value: 'coffee shop', relevance: 601, type: 'QUERY' },
{ value: 'coffee table', relevance: 600, type: 'QUERY' },
{ value: 'coffee maker', relevance: 553, type: 'QUERY' },
{ value: 'coffee bean', relevance: 552, type: 'QUERY' },
{ value: 'coffee grinder', relevance: 551, type: 'QUERY' },
{ value: 'coffee meets bagel', relevance: 550, type: 'QUERY' }
]
1300

Our result should look like this 👆🏻.

With Google Autocomplete API


If you want to scrape Google Autocomplete Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this API.

Serpdog also provides 100 free searches per month, and if you want to scale your request quota, you can buy paid plans also.

const axios = require('axios');
axios.get('https://api.serpdog.io/autocomplete?api_key=APIKEY&q=football&gl=us')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});

Results:

{
"meta": {
"api_key": "APIKEY",
"q": "football",
"gl": "us"
},
"suggestions": [
{
"value": "football cleats",
"relevance": 601,
"type": "QUERY"
},
{
"value": "football games",
"relevance": 600,
"type": "QUERY"
},
{
"value": "football wordle",
"relevance": 555,
"type": "QUERY"
},
{
"value": "football today",
"relevance": 554,
"type": "QUERY"
},
{
"value": "football gloves",
"relevance": 553,
"type": "QUERY"
},
{
"value": "football field",
"relevance": 552,
"type": "QUERY"
},
{
"value": "football movies",
"relevance": 551,
"type": "QUERY"
},
{
"value": "football positions",
"relevance": 550,
"type": "QUERY"
}
],
"verbatim_relevance": 1300
}

Conclusion:


In this tutorial, we learned to scrape Google Autocomplete Suggestions using Node JS. Feel free to message me anything you need clarification on. 

Follow me on Twitter. Thanks for reading!

Additional Resources

  1. How to scrape Google Organic Search Results using Node JS?
  2. Scrape Google Images Results
  3. Scrape Google News Results
  4. Scrape Google Maps Reviews

Similar Posts