How to web scrape Yelp data?
Yelp is one of the biggest crowd-sourced ratings and review websites for local businesses. It is a trusted review website because of the limited amount of spam and ads. With more than 200 million public reviews on its platform makes Yelp, a data-rich website for data miners.
In this tutorial, we are going to scrape Yelp Search Results using different Node JS libraries. And at the end, to make things simple, I will suggest you a Yelp Search API, which you can use to scrape data from Yelp easily.
Why Scrape Yelp?
Yelp has a mighty base of 90 million visitors per month across its website and mobile app, with users and businesses contributing to this platform day-to-day.
Scraping Yelp can help you to get information about your competitors, their rating, and reviews. And using that way, you can analyze where your business stands in the market and the weak points that are becoming problems in your business expansion.
Yelp’s Ultra-Big Size Business Directory is a great source of Quality Leads For Your Business. You Can Also Collect Addresses, Phone Numbers, And Other Details By Scraping Yelp.
Before we start with the tutorial, let me explain some requirements for this tutorial.
Requirements:
Web Parsing with CSS selectors
Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.
This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.
User Agents
User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.
You can also rotate User Agents, read more about this in this article: How to fake and rotate User Agents using Python 3.
If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Websites.
Install Libraries
To start scraping Yelp Search Results we need to install some NPM libraries so that we can move forward.
So before starting, we have to ensure that we have set up our Node JS project and installed both the packages - Unirest JS and Cheerio JS. You can install both packages from the above link.
Scraping Yelp Business Listings
We will scrape the individual business listings on Yelp for a particular query and location. And here is an important piece of data that we are going to scrape:
- Name of the business
- Ratings and Reviews
- Service Options Offered By Businesses
- Highlights
Scraping Yelp Search Results:
The Yelp Search Result scraping can be divided into two parts:
- Making the HTTP request on the target URL to extract the raw HTML data.
- Parsing the HTML data to extract the required data.
Now, let’s start scraping Yelp by making a GET request on the target URL using Unirest to get the raw HTML data.
const unirest = require("unirest")
const cheerio = require("cheerio")
const getData = async () => {
let url = "https://www.yelp.com/search?find_desc=burger&find_loc=San+Francisco%2C+CA"
const selectRandom = () => {
const userAgents = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36",
];
var randomNumber = Math.floor(Math.random() * userAgents.length); return userAgents[randomNumber];
}
let user_agent = selectRandom();
let header = {
"User-Agent": `${user_agent}`
}
const response = await unirest
.get(url)
.headers(header)
In the above code, I just made a function to select a random User Agent on every request and then passed this User Agent as a header with a GET request on the URL using Unirest.
Now, we have completed the part for scraping raw HTML.
Then, we will parse this raw HTML using Cheerio.
const $ = cheerio.load(response.body)
If you use the CSS selector Gadget correctly, you will find all these results are under the tag .padding-13__09f24__IOjKY.
Similarly, you can find the tags for other respective data also. For example, the tag for the title is .css-1m051bw, the tag for the review is is .margin-t0-5__09f24__gboxT .css-chan6m, etc.
And this is what our parser looks like:
const $ = cheerio.load(response.body);
let results = [];
$(".padding-l3__09f24__IOjKY").each((i, el) => {
results.push({
title: $(el).find(".css-1m051bw").text(),
rating: $(el).find(".css-gutk1c").text(),
reviews: $(el).find(".margin-t0-5__09f24__gboxT .css-chan6m").text(),
snippet: $(el).find(".css-16lklrv").text(),
neighbourhood: $(el).find(".css-dzq7l1 .css-chan6m").text(),
thumbnail: $(el).find("img").attr("src")
})
if ($(el).find(".mobile-text-medium__09f24__MZ1v6").length) {
let highlights = [];
$(el).find(".mobile-text-medium__09f24__MZ1v6").each((i, el) => {
highlights[i] = $(el).text().replace("")
})
results[i].highlights = highlights;
}
if ($(el).find(".css-1italjr").length && !$(el).find(".content__09f24__vViKh").length) {
let button = {};
button = {
text: $(el).find(".css-1italjr span").text(),
link: "https://www.yelp.com" + $(el).find("a.css-1italjr").attr("href")
}
results[i].button = button;
}
if ($(el).find(".css-11bijt4").length) {
let categories = [];
$(el).find("a.css-abnp9g").each((i, el) => {
categories.push({
title: $(el).find(".css-11bijt4").text(),
link: "https://www.yelp.com" + $(el).attr("href")
})
})
results[i].categories = categories;
}
if ($(el).find(".raw__09f24__T4Ezm").length) {
let service_options = [];
$(el).find(".raw__09f24__T4Ezm").each((i, el) => {
service_options[i] = $(el).text()
})
results[i].service_options = service_options
}
})
Here are the results:
{
title: 'Applebee’s Grill + Bar',
neighbourhood: "Fisherman's Wharf",
thumbnail: 'https://s3-media0.fl.yelpcdn.com/offerphoto/G3FUB2Xr7o20mZA5aB2xxg/ls.jpg',
categories: [
{
title: 'Sports Bars',
link: 'https://www.yelp.com/search?cflt=sportsbars&find_loc=San+Francisco%2C+CA'
},
{
title: 'American (Traditional)',
link: 'https://www.yelp.com/search?cflt=tradamerican&find_loc=San+Francisco%2C+CA'
},
{
title: 'Burgers',
link: 'https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA'
}
],
service_options: [ 'Outdoor seating', 'Delivery', 'Takeout' ]
}
{
title: 'Burger King',
snippet: "“OK, so it's late at night, you are hungry, and you want some deep fried grease or a burger. I am sure this place probably only deserves 2 stars because Burger King is not very high…” more",
neighbourhood: 'Fillmore',
thumbnail: 'https://s3-media0.fl.yelpcdn.com/bphoto/XrU8Vnn6B_OnboZpOyJBZQ/348s.jpg',
button: {
text: 'Get Directions',
link: 'https://www.yelp.com/adredir?ad_business_id=X9WCWO4eikSkhuy6y2VMcA&click_origin=search_results_direction&placement=above_search&placement_slot=1&redirect_url=https%3A%2F%2Fwww.yelp.com%2Fmap%2Fburger-king-san-francisco-9&request_id=695c641556a7a93c&signature=4075594fe886f6c488ae7e78dae88bb58598e70828613aa7453e7bbf24e949f1&slot=1'
},
categories: [
{
title: 'Burgers',
link: 'https://www.yelp.com/search?cflt=burgers&find_loc=San+Francisco%2C+CA'
},
{
title: 'Fast Food',
link: 'https://www.yelp.com/search?cflt=hotdogs&find_loc=San+Francisco%2C+CA'
}
],
service_options: [ 'Delivery', 'Takeout' ]
}
Here is the complete code:
const unirest = require("unirest")
const cheerio = require("cheerio")
const getData = async () => {
let url = "https://www.yelp.com/search?find_desc=burger&find_loc=San+Francisco%2C+CA"
const selectRandom = () => {
const userAgents = ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36",
];
var randomNumber = Math.floor(Math.random() * userAgents.length); return userAgents[randomNumber];
}
let user_agent = selectRandom();
let header = {
"User-Agent": `${user_agent}`
}
const response = await unirest
.get(url)
.headers(header)
const $ = cheerio.load(response.body);
let results = [];
$(".padding-l3__09f24__IOjKY").each((i, el) => {
results.push({
title: $(el).find(".css-1m051bw").text(),
rating: $(el).find(".css-gutk1c").text(),
reviews: $(el).find(".margin-t0-5__09f24__gboxT .css-chan6m").text(),
snippet: $(el).find(".css-16lklrv").text(),
neighbourhood: $(el).find(".css-dzq7l1 .css-chan6m").text(),
thumbnail: $(el).find("img").attr("src")
})
if ($(el).find(".mobile-text-medium__09f24__MZ1v6").length) {
let highlights = [];
$(el).find(".mobile-text-medium__09f24__MZ1v6").each((i, el) => {
highlights[i] = $(el).text().replace("")
})
results[i].highlights = highlights;
}
if ($(el).find(".css-1italjr").length && !$(el).find(".content__09f24__vViKh").length) {
let button = {};
button = {
text: $(el).find(".css-1italjr span").text(),
link: "https://www.yelp.com" + $(el).find("a.css-1italjr").attr("href")
}
results[i].button = button;
}
if ($(el).find(".css-11bijt4").length) {
let categories = [];
$(el).find("a.css-abnp9g").each((i, el) => {
categories.push({
title: $(el).find(".css-11bijt4").text(),
link: "https://www.yelp.com" + $(el).attr("href")
})
})
results[i].categories = categories;
}
if ($(el).find(".raw__09f24__T4Ezm").length) {
let service_options = [];
$(el).find(".raw__09f24__T4Ezm").each((i, el) => {
service_options[i] = $(el).text()
})
results[i].service_options = service_options
}
})
for (let i = 0; i < results.length; i++) {
Object.keys(results[i]).forEach(key => results[i][key] === "" || results[i][key] === undefined ? delete results[i][key] : {});
}
for (let i = 0; i < results.length; i++) {
Object.keys(results[i]).forEach(key => results[i][key] === "" || results[i][key] === undefined ? delete results[i][key] : {});
}
console.log(results)
}
getData();
And that’s what a basic scraper of Yelp looks like. Similarly, this same process of scraping and selecting tags can be followed in other programming languages.
Another method for scraping Yelp data is by utilizing Serpdog’s Yelp Search API.
With Yelp Search API
Maintaining the scraper according to the frequently changing HTML structure of websites can be a frustrating task. It is not only time-consuming but it also requires continuous updates and maintenance to ensure effective data collection.
To resolve this issue, you can leverage Serpdog’s Yelp Search API which provides a fast and reliable API to retrieve real-time data from Yelp without worrying about any blockage and HTML changes.
Moreover, Serpdog also offers 1000 free requests credits on registration on our website.
Frequently Asked Questions
How to scrape Yelp data for free?
You Can Use Serpdog’s Yelp Search API For Scraping Yelp Data For Free. Serpdog Provides 100 Credits To Users On Their First Sign-Up.
Can You Scrape Yelp Data?
Yes, You Can Scrape Yelp Data With The Help Of Any Scraping API Present In The Market.
Is It Legal To Scrape Yelp Data?
You Can Scrape Any Data Publicly Available On The Internet Without Any Legal Problem.
Conclusion:
In this tutorial, we learned to scrape Yelp Data by making a basic scraper with the help of Node JS. We also saw how Serpdog | Yelp Search API can help you with scraping the results.
I hope you enjoyed the tutorial. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!
Additional Resources
Want to learn more about web scraping? Not a problem! We have already prepared the list of tutorials so you can kickstart your web scraping journey.
