Web Scraping Google News Using Python

Python is the most popular language for web scraping. This language is widely used in creating automated web scrapers to extract the precious data available on the internet for various purposes like Data analysis, SEO monitoring, news, and media monitoring.

In this post, we’ll create a web scraping tool to scrape Google News Results with Python.

Why Scrape Google News Results? 

Scraping Google News Results provide several benefits, including:

Brand Monitoring — Scraping Google News Results can help you monitor the media and public perspective about your brand. It helps to keep a check on any issue or negative publicity about your company that can affect your business.

Keeps You Updated — News Results keep you updated about the current political events occurring in the world. It also helps you to keep a check on the current advancements taking place in your areas of interest.

Market Research — Google News Results can help you study various historical trends in your industry and the data can also be used for research-based purposes like consumer sentiment, competitor analysis, etc.

Competitor Analysis — You can utilize the news data to monitor the latest developments and new product launches by your competitor. You can also study their media strategy to identify any loopholes in your tactics while dealing with media marketing.

Building Awareness – Scrape Google News Data to build awareness among the public on particular topics such as political science, GK, economics, etc.

Let’s Start Scraping Google News Using Python

In this blog post, we’ll create a Python script to extract the first 100 Google News results including, the title, description, link, source, and date. 

Requirements

To scrape Google News, we will be installing these two libraries: 

1. Beautiful Soup — Used for parsing the raw HTML data.
2. Requests — Used for making HTTP requests.

Or you can directly install these libraries by running the below commands in your terminal:

pip install requests
pip install beautifulsoup4

Process:

Before starting, I assume you have set up your Python project on your device. So, open the project file in your respective code editor and import these two libraries, which we will use in this tutorial. 

import json
import requests
from bs4 import BeautifulSoup

Now, let’s create a function to scrape the Google News Results: 

def getNewsData():
    headers = {
        "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
    }
    response = requests.get(
        "https://www.google.com/search?q=amazon&gl=us&tbm=nws&num=100", headers=headers
    )
    soup = BeautifulSoup(response.content, "html.parser")
    news_results = []

First, we set the header to the User Agent, which will help us to make our scraping bot make an organic visit to Google. Then we made an HTTP request on the target URL using the request library we imported above and stored the extracted HTML in the response variable. And in the last line, we created an instance of the BeautifulSoup library to parse the HTML data. 

Let us now search for the tags from the HTML to extract the required data.

If you inspect the HTML file, you will find every result or news article is contained inside this div.SoaBEf tag. And after further searching in the container, you will find the tag for the title as div.MBeuO, the description as .GI74Re, and the source as .NUnG9d span and the date as .LfVVr.

Now, add these tags in your parser: 

    for el in soup.select("div.SoaBEf"):
        news_results.append(
            {
                "link": el.find("a")["href"],
                "title": el.select_one("div.MBeuO").get_text(),
                "snippet": el.select_one(".GI74Re").get_text(),
                "date": el.select_one(".LfVVr").get_text(),
                "source": el.select_one(".NUnG9d span").get_text()
            }
        )

    print(json.dumps(news_results, indent=2))

getNewsData()

Ok, so let us now run this code in our terminal to see the results:

Hurray🥳🥳!!! We have successfully scraped the news data from Google News. Let us look at another method to help you scrape the news results without getting blocked.

Using Google News API to Scrape News

Scraping news results can be difficult for an individual with changing HTML structure. Also, one should have a large pool of residential proxies and User Agents to avoid any blockage from Google for smooth scraping. 

What if you were provided with a simple and streamlined solution to scrape Google News Results?

Yes, you heard right! Our Google News API is powered by a network of more than 10M+ proxies and allows you to scrape Google News Results at scale without any fear of blockage.

We also offer 100 free requests on the first sign-up.

After getting registered on our website, you will get an API Key. Embed this API Key in the code below, and you will be able to scrape Google News Results at a much faster speed.

import requests
payload = {'api_key': 'APIKEY', 'q':'football' , 'gl':'us'}
resp = requests.get('https://api.serpdog.io/news', params=payload)
print (resp.text)

Conclusion:

This tutorial taught us to scrape Google News Results using Node JS. Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources

1. How to scrape Google Organic Search Results using Node JS?
2. Scrape Google Images Results
3. Scrape Google Shopping Results
4. Scrape Google Maps Reviews

Frequently Asked Questions