Google Jobs Listings provides valuable data that enterprises can use to enrich their job database, conduct market research, and streamline recruitment purposes. It is a job aggregator which collects job posts from multiple platforms. This allows job seekers and researchers to save time and effort by not looking into the same posts from different platforms.
In this article, we will learn to scrape Google Jobs Using Node JS. We will also explore its benefits and how this data can be scraped more easily and effectively.
What is Google Jobs?
The Google Jobs Search Page is a webpage that aggregates job listings from various employment sites such as LinkedIn, Glassdoor, and Indeed. It allows users to filter jobs by title, location, industry, and more and directs users to the original postings for the application.
The left side consists of job listings from different sources, while the right side contains the job description, including application links, highlights, location, salary, and other information.
In this guide, we will be targeting these data points from the above job search page:
- Job title
- Company name
- Job location
- Job posted via [platform]
- Job listing date
Why scrape Google Jobs?
Scraping Google Jobs can provide you with various benefits:
Job Listings – Google Jobs Listings are aggregated at a large scale from different sources including job boards, career pages, etc which allow users to access an amplitude of opportunities in a single place.
Job Trends – Scraping Google Jobs can help you analyze various job trends and market demands in the industries. You can use this data to get an insight into the jobs that have become popular recently and when these jobs can reach a peak according to the information you have gained by data extraction.
Job Openings – You can also use Google Jobs data to access a huge volume of job openings that you might have not seen otherwise.
Let’s start scraping Google Jobs
In this section, we will prepare a basic script to scrape the first 10 Google Jobs listings, including their title, location, and other details.
For this, we have first to install some libraries so we can move forward:
- Unirest JS – For scraping raw HTML data.
- Cheerio JS – For parsing the raw HTML data.
We will also download a CSS selector agent to easily select the elements from the HTML so we don’t have to deal with the complex HTML structure.
Please open the link below in your browser so that we can select the HTML tags for the required elements.
https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en
Let’s begin the process of scraping the Google Jobs Results. We will be using Unirest JS to extract the raw HTML data and parse this data with the help of Cheerio JS.
Now, make a GET request using Unirest JS on the target URL.
const unirest = require("unirest");
const cheerio = require("cheerio");
const getJobsData = async () => {
const url = "https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en";
const response = await unirest
.get(url)
.header({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"})
const $ = cheerio.load(response.body);
Step-by-step explanation:
- In the fifth line, we made a GET request to the target URL.
- In the following line, we passed User-Agent as a header with the URL, so our bot can mimic a real organic user.
- Next, we load the response in a Cheerio instance variable.
Now, we will prepare our parser by searching the tags with the help CSS selector gadget, as stated above in the Requirements section.
All the jobs tabs you saw above are inside an HTML tag li.iFjolb . So its parser will look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
})
})
Then we will start selecting the tags using the CSS SelectorGadget to scrape the required data.
After adding this tag to our scraper, it looks like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text()
})
})
Then we will parse the company name.
From the above image, we found the tag for the company name as .vNEEBe.
Similarly, we can find the tags for the other required data fields, which makes our parser look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text(),
company_name: $(el).find(".vNEEBe").text(),
location: $(el).find(".vNEEBe+ .Qk80Jf").text(),
via: $(el).find(".Qk80Jf+ .Qk80Jf").text(),
})
if($(el).find(".KKh3md").length)
{
jobs_results[i].extensions = [];
$(el).find(".KKh3md .LL4CDc").each((j,el) => {
jobs_results[i].extensions[j] = $(el).text()
})
jobs_results[i].extensions = jobs_results[i].extensions?.slice(jobs_results[i].extensions?.length/2)
}
})
Results:
[
{
title: 'Web Developer',
company_name: 'DigiPlusIT',
location: 'Mumbai, Maharashtra',
via: 'via LinkedIn',
extensions: [ '1 day ago', 'Full–time' ]
},
{
title: 'Web Developer',
company_name: 'Mumbai Educational Trust',
location: 'Mumbai, Maharashtra',
via: 'via Glassdoor',
extensions: [ '13 hours ago', 'Full–time' ]
},
......
Using Google Jobs API to Scrape Jobs Listings
Scraping Jobs can be difficult in the long term if you are not able to maintain and optimize the scraper to avoid frequent IP bans and often changing HTML structure. To avoid this hassle, you can try this Google Jobs API.
Serpdog has a massive network of 30M+ residential proxies allowing developers to scrape Google Jobs Results smoothly. Also, the pre-cooked structured JSON data can save you a lot of time.
Here is how you can use our Google Jobs API to aggregate job results published on trusted platforms across the world.
const axios = require('axios');
axios.get('https://api.serpdog.io/jobs?q=jobs+in+london&api_key=APIKEY')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Results:
{
"meta": {
"gl": "us",
"q": "jobs in london",
"api_key": "APIKEY"
},
"jobs_results": [
{
"title": "Barista | London",
"company_name": "Blank Street",
"location": "London, UK",
"via": "via Greenhouse",
"description": "About Blank Street\n\nAt Blank Street, we believe great coffee should be an everyday ritual. With shops across London, Brooklyn, Manhattan, Boston and DC, we’re the first-ever brand to offer affordable high-quality coffee.\n\nBlank Street originated in an effort to change the specialty coffee status quo. Starting with small-format shops and continuing with a limited menu and top-of-the-line tech, we’ve always been focused on simplifying the coffee experience. We partner with amazing local vendors and brands, and have some of the best baristas out there on our team.\nLove coffee and customer service? Keep reading.\n\nWhat's Brewing...\n\nBaristas at Blank Street must be able to work independently, while also thriving in a team environment. The right person for this role is a proven people-person, taking pride in giving amazing service experiences. You are customer-centric and believe in serving a carefully curated menu with passion and precision, making every effort to give your customers... exactly what they need, every day, in their way.\n\nOur Values\n• Magic is in the Details: in everything we do we value attention to detail, going the extra mile & thinking about everything that makes a moment meaningful\n• Move as One: We look for inclusive and respectful team members who strive to be the best team player & who over-communicates to ensure understanding\n• “My House is your House” Hospitality: We look for individuals who welcome guests into the cafe the same way they would invite in a friend, who are obsessed with presentation, making sure our cafes are always clean and welcoming & who provide stellar service, even when things are busy or stressful\n\nWho you are:\n• A friendly and enthusiastic team player with a passion for excellent customer service; you’re always ready to make your customers’ day\n• A reliable employee who is able to effectively manage time and priorities, including during busy rush periods where a sense of urgency is necessary\n• Passionate about creating delicious coffee and other cafe beverages\n• A strong independent problem solver with proven multi-tasking and communication skills\n• Someone who is curious, adaptable and always willing to learn\n• Comfortable working in a team or independently\n• Comfortable maintaining store safety\n\nWhat you'll own:\n• Delight our customers with consistent, welcoming and engaging customer service, taking the opportunity to turn every customer into a regular\n• Work within a world-class coffee program using the best coffee equipment (the Eversys cameos and shot masters) and products in the industry to prepare the tastiest drinks\n• Complete all training to ensure proper measures are in place to achieve the correct handling of food and beverage to retain its freshness and quality\n• Have systems in place to avoid wastage of product with both food and beverage, following and adjusting par levels where needed in line with business peaks and lows\n• Adhere to Health and Safety Regulations and Food Safety Standards at all times\n• Full and complete knowledge and adherence to all product, service and brand training playbooks\n• Participates in all initiatives with the Operations and marketing teams to increase foot traffic, new customers, daily sales, and operational profitability and develop a loyal neighbourhood customer base\n\nRequirements:\n• Experience in the customer service or hospitality industry\n• Availability that meets the needs of our cafes\n• Part-Time: 24 hours per week, 3 days of availability required and must be available to work 2 peak days (Peak days: Friday, Saturday, Sunday)\n• Full Time: 35 hours per week, 4-5 days of availability required and must be available to work 2 peak days (Peak days: Friday, Saturday, Sunday)\n• Weekend and holiday availability preferred\n• You must be authorised to work in the UK\n\nPerks:\n• £12.00 - £13.00\n• Industry-leading pay, bi-weekly payments, pension scheme\n• Barista training program and exciting growth opportunities\n• Unlimited drinks and free food - when on shift\n• 50% off all drinks and food - when you are off your shift\n• Great work-life balance\n• Referral incentive scheme - £250 when you refer a friend\n\nApplication & Interview Process:\n• Online Questionnaire\n• Recruiter Phone Interview\n• Hiring Manager In-Person Interview\n• Offer",
"url": "https://www.google.com/search?ibp=htl;jobs&q=jobs+in+london&htidocid=5efcMuKLmz4lae9WAAAAAA%3D%3D&hl=en-US&kgs=1a84be9da335179c&shndl=-1&shem=ssic&source=sh/x/im/textlists/detail/m1/1#fpstate=tldetail&htivrt=jobs&htiq=jobs+in+london&htidocid=5efcMuKLmz4lae9WAAAAAA%3D%3D",
"extensions": [
"Full-time and Part-time",
"No degree mentioned"
],
"apply_links": [
{
"title": "Apply on Greenhouse",
"link": "https://boards.greenhouse.io/blankstreet/jobs/5210435003?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on LinkedIn",
"link": "https://uk.linkedin.com/jobs/view/barista-london-at-blank-street-3667913102?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Monster.co.uk",
"link": "https://www.monster.co.uk/job-openings/barista-london-eng--4c52d2ff-80e3-48a4-ae79-0e6935f36fd6?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Left Lane Job Board",
"link": "https://jobs.leftlanecap.com/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Base10 Job Board",
"link": "https://careers.base10.vc/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on General Catalyst Job Board",
"link": "https://jobs.generalcatalyst.com/companies/blank-street/jobs/15383388-barista-london?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on Talentify",
"link": "https://www.talentify.io/job/barista-london-england-blank-street-5210435003?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
},
{
"title": "Apply on ZipRecruiter UK",
"link": "https://www.ziprecruiter.co.uk/jobs/321842903-barista-london-at-blank-street?utm_campaign=google_jobs_apply&utm_source=google_jobs_apply&utm_medium=organic"
}
]
},
Conclusion:
In a nutshell, Google Jobs is a vast repository of job posts from various trusted sources that can be used for identifying industry trends and staying informed about the latest market demands around the world.
In this tutorial, we learned to scrape Google Jobs Results with Node JS. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!