Scrape Google Autocomplete Suggestions
In This Post, We Will Learn To Scrape Google Autocomplete Suggestions Using Node JS.
Requirements:
User Agents
User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.
You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.
If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Google.
Install Libraries
To scrape Google maps reviews we need to install some NPM libraries so we can move forward.
So before starting, we have to ensure that we have set up our Node JS project and installed both the packages — Unirest JS and Cheerio JS. You can install both packages from the above link.
Target:
We will target to scrape the shopping results of Nike shoes.
Let’s start scraping Google Autocomplete Results
Now, all the things are installed that we will need to prepare our scraper. We will use an npm library Unirest JS to make a get request to our target URL so we can get our raw HTML data. Then we will use Cheerio JS for parsing the extracted HTML data.
We will target this URL:
https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome
Copy this URL into your browser and press enter. You will see a text file downloading in your browser by entering this URL in your browser. Open this file in your respective code editor. We will now first convert this JSON string into an object and get the respective data we need.
const unirest = require("unirest");
const searchSuggestions = async () => {
try {
const response = await unirest
.get("https://www.google.com/complete/search?&hl=en&q=coffee&gl=us&client=chrome")
.headers({
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
})
let data = JSON.parse(response.body);
let suggestions = [];
for (let i = 0; i < data[1].length; i++) {
suggestions.push({
value: data[1][i],
relevance: data[4]["google:suggestrelevance"][i],
type: data[4]["google:suggesttype"][i],
});
}
const verbatimRelevance = data[4]["google:verbatimrelevance"]
console.log(suggestions)
console.log(verbatimRelevance)
} catch (e) {
console.log("Error : " + e);
}
};
searchSuggestions();
You can also check some of my other Google scrapers on my Git Repository: https://github.com/Darshan972/GoogleScrapingBlogs
Result:
[
{ value: 'coffee near me', relevance: 1250, type: 'QUERY' },
{ value: 'coffee shops near me', relevance: 650, type: 'QUERY' },
{ value: 'coffee shop', relevance: 601, type: 'QUERY' },
{ value: 'coffee table', relevance: 600, type: 'QUERY' },
{ value: 'coffee maker', relevance: 553, type: 'QUERY' },
{ value: 'coffee bean', relevance: 552, type: 'QUERY' },
{ value: 'coffee grinder', relevance: 551, type: 'QUERY' },
{ value: 'coffee meets bagel', relevance: 550, type: 'QUERY' }
]
1300
Our result should look like this 👆🏻.
With Google Autocomplete API
If you want to scrape Google Autocomplete Results easily, without making a scraper, as scraping can take a lot of time sometimes, you can try this API.
Serpdog also provides 100 free searches per month, and if you want to scale your request quota, you can buy paid plans also.
const axios = require('axios');
axios.get('https://api.serpdog.io/autocomplete?api_key=APIKEY&q=football&gl=us')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Results:
{
"meta": {
"api_key": "APIKEY",
"q": "football",
"gl": "us"
},
"suggestions": [
{
"value": "football cleats",
"relevance": 601,
"type": "QUERY"
},
{
"value": "football games",
"relevance": 600,
"type": "QUERY"
},
{
"value": "football wordle",
"relevance": 555,
"type": "QUERY"
},
{
"value": "football today",
"relevance": 554,
"type": "QUERY"
},
{
"value": "football gloves",
"relevance": 553,
"type": "QUERY"
},
{
"value": "football field",
"relevance": 552,
"type": "QUERY"
},
{
"value": "football movies",
"relevance": 551,
"type": "QUERY"
},
{
"value": "football positions",
"relevance": 550,
"type": "QUERY"
}
],
"verbatim_relevance": 1300
}
Conclusion:
In this tutorial, we learned to scrape Google Autocomplete Suggestions using Node JS. Feel free to message me anything you need clarification on.
Follow me on Twitter. Thanks for reading!
