Google Jobs Listings provides valuable data that enterprises can use to enrich their job database, conduct market research, and streamline recruitment purposes. It is a job aggregator which collects job posts from multiple platforms. This allows job seekers and researchers to save time and effort by not looking into the same posts from different platforms.
In this article, we will learn to scrape Google Jobs Using Node JS. We will also explore its benefits and how this data can be scraped more easily and effectively.
Why scrape Google Jobs?
Scraping Google Jobs can provide you with various benefits:
Job Listings – Google Jobs Listings are aggregated at a large scale from different sources including job boards, career pages, etc which allow users to access an amplitude of opportunities in a single place.
Job Trends – Scraping Google Jobs can help you analyze various job trends and market demands in the industries. You can use this data to get an insight into the jobs that have become popular recently and when these jobs can reach a peak according to the information you have gained by data extraction.
Job Openings – You can also use Google Jobs data to access a huge volume of job openings that you might have not seen otherwise.
Let’s start scraping Google Jobs
In this section, we will prepare a basic script to scrape the first 10 Google Jobs listings, including their title, location, and other details.
For this, we have first to install some libraries so we can move forward:
- Unirest JS – For scraping raw HTML data.
- Cheerio JS – For parsing the raw HTML data.
We will also download a CSS selector agent to easily select the elements from the HTML so we don’t have to deal with the complex HTML structure.
Please open the link below in your browser so that we can select the HTML tags for the required elements.
https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en
Let’s begin the process of scraping the Google Jobs Results. We will be using Unirest JS to extract the raw HTML data and parse this data with the help of Cheerio JS.
Now, make a GET request using Unirest JS on the target URL.
const unirest = require("unirest");
const cheerio = require("cheerio");
const getJobsData = async () => {
const url = "https://www.google.com/search?q=web+developer+in+mumbai&ibp=htl;jobs&hl=en";
const response = await unirest
.get(url)
.header({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"})
const $ = cheerio.load(response.body);
Step-by-step explanation:
- In the fifth line, we made a GET request to the target URL.
- In the following line, we passed User-Agent as a header with the URL, so our bot can mimic a real organic user.
- Next, we load the response in a Cheerio instance variable.
Now, we will prepare our parser by searching the tags with the help CSS selector gadget, as stated above in the Requirements section.
All the jobs tabs you saw above are inside an HTML tag li.iFjolb . So its parser will look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
})
})
Then we will start selecting the tags using the CSS SelectorGadget to scrape the required data.
After adding this tag to our scraper, it looks like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text()
})
})
Then we will parse the company name.
From the above image, we found the tag for the company name as .vNEEBe.
Similarly, we can find the tags for the other required data fields, which makes our parser look like this:
let jobs_results = [];
$(".iFjolb").each((i,el) => {
jobs_results.push({
title: $(el).find(".PUpOsf").text(),
company_name: $(el).find(".vNEEBe").text(),
location: $(el).find(".vNEEBe+ .Qk80Jf").text(),
via: $(el).find(".Qk80Jf+ .Qk80Jf").text(),
})
if($(el).find(".KKh3md").length)
{
jobs_results[i].extensions = [];
$(el).find(".KKh3md .LL4CDc").each((j,el) => {
jobs_results[i].extensions[j] = $(el).text()
})
jobs_results[i].extensions = jobs_results[i].extensions?.slice(jobs_results[i].extensions?.length/2)
}
})
Results:
[
{
title: 'Web Developer',
company_name: 'DigiPlusIT',
location: 'Mumbai, Maharashtra',
via: 'via LinkedIn',
extensions: [ '1 day ago', 'Full–time' ]
},
{
title: 'Web Developer',
company_name: 'Mumbai Educational Trust',
location: 'Mumbai, Maharashtra',
via: 'via Glassdoor',
extensions: [ '13 hours ago', 'Full–time' ]
},
......
Using Google Jobs API to Scrape Jobs Listings
Scraping Jobs can be difficult in the long term if you are not able to maintain and optimize the scraper to avoid frequent IP bans and often changing HTML structure. To avoid this hassle, you can try this Google Jobs API.
Serpdog has a massive network of 10M+ residential proxies allowing developers to scrape Google Jobs Results smoothly. Also, the pre-cooked structured JSON data can save you a lot of time.
Here is how you can use our Google Jobs API to aggregate job results published on trusted platforms across the world.
const axios = require('axios');
axios.get('https://api.serpdog.io/jobs?q=jobs+in+london&api_key=APIKEY')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Results:
"jobs_results": [
{
"title": "Front Office Manager",
"company_name": "Accor Hotels",
"location": "London, UK",
"via": "via Accor Careers",
"description": "Job Description\n\nAre you an enthusiastic leader ready for your next exciting opportunity?!\n\nWe are currently recruiting a Front Office Manager to join our fantastic team Ibis London ExCeL Docklands!\n\nYour missions will be:\n\n• Ensures all Front Office policies and procedures are adhered\n\n• Coordinates management of the Front Office department while on duty to ensure a smooth operation and the highest level of guest satisfaction seeking feedback on guest satisfaction and responding directly back to guests who experience any issues\n\n• Strong & effective communication with all other departments\n\n• Provides guidance and motivation to the Front Office team, creating an environment that allows colleagues to achieve job fulfilment and providing a path for career development with AccorHotels\n\n• Responsible to balance operational, administrative and colleague's needs\n\n• Oversees group booking and responsible for meeting room booking alongside the F&B Manager\n\n• Develops, implements and... maintains new incentives through the promotion of our loyalty program ALL to motivate colleagues and maximise hotel revenue\n\n• Ensures effective utilisation & productivity of all colleagues through staff planning, hiring, scheduling & adhering to budget\n\n• Adheres to and promotes the Company’s Health & safety policies to ensure a safe work environment and knowledgeable all safety & emergency procedures ensuring a clean and safe working environment\n\nWork Experience\n\nWho are we looking for?\n\n• Previous experience in similar roles is preferable\n\n• Friendly, approachable with excellent customer service skills\n\n• Attention to detail and exceptional organisational skills\n\n• Strong team player and positive attitude\n\n• Ability to work effectively under pressure\n\n• Essential to be flexible with working hours\n\n• Must already have the right to work in the UK\n\nBenefits\n\nWhat do we offer?\n\nExcellent training and development opportunities\n\nAnnual Bonus Scheme (up to one month additional salary)\n\nWorldwide staff discounts\n\nFree stays at AccorHotels UK & Ireland\n\nMeals on duty\n\nUniform and dry cleaning",
"extensions": [
"Full-time"
]
},
......
Conclusion:
In a nutshell, Google Jobs is a vast repository of job posts from various trusted sources that can be used for identifying industry trends and staying informed about the latest market demands around the world.
In this tutorial, we learned to scrape Google Jobs Results with Node JS. Feel free to message me if I missed something. Follow me on Twitter. Thanks for reading!