...

Scrape Google Finance With Node JS

Scrape Google Finance

Google Finance Page is a data-rich page for traders and investors to get access to international exchanges, real-time financial news, and financial analysis to keep you updated with the current market scenario.

This type of data-rich website always has a couple of advantages:

  • It helps traders and investors in pricing analysis.
  • The financial data of a company can help people decide which stock to purchase in the future.
  • Technical indicators and analysis are used to research trends in the historical pricing of stock and predict the future trend of share prices.

Why scrape Google Finance?

Scraping Google Finance can provide you with various benefits:

Keeps You Updated: Scraping Google Finance keeps you updated about the current situation of the market around the world and helps you change your strategies according to the possible future trends in the market.

Data Analysis: The Google Finance data can provide you with insights into current market trends of an industry, its performance, and much more which can be used to discover any opportunities for investment in the market.

News analysis: You can use Google Finance data to scrape finance-related news around the world which can be used for sentimental analysis, tracking news trends, and much more.

Requirements for scraping Google Finance:

Web Parsing with CSS selectors

Searching the tags from the HTML files is not only a difficult thing to do but also a time-consuming process. It is better to use the CSS Selectors Gadget for selecting the perfect tags to make your web scraping journey easier.

This gadget can help you to come up with the perfect CSS selector for your need. Here is the link to the tutorial, which will teach you to use this gadget for selecting the best CSS selectors according to your needs.

User Agents

User-Agent is used to identify the application, operating system, vendor, and version of the requesting user agent, which can save help in making a fake visit to Google by acting as a real user.

You can also rotate User Agents, Read more about this in this article: How to fake and rotate User Agents using Python 3.

If you want to further safeguard your IP from being blocked by Google, you can try these 10 Tips to avoid getting Blocked while Scraping Google.

Install Libraries

To start scraping Google Finance Results we need to install some NPM libraries so that we can move forward.

  1. Node JS
  2. Unirest JS
  3. Cheerio JS

So before starting, we have to ensure that we have set up our Node JS project and installed both the packages — Unirest JS and Cheerio JS. You can install both packages from the above link.

Let’s start scraping Google Finance:

Google Finance Web Page
Web Scraping Google Finance With Node JS 2

Let’s start scraping the Google Finance Page. We will be using Unirest JS to extract the raw HTML data and parse this data with the help of Cheerio JS.

Copy this link and open it in your browser so we can start selecting the tags to parse the required data.

https://www.google.com/finance/?hl=en

Now, we will make a GET request to our target URL using Unirest JS to extract the HTML data.

const unirest = require("unirest");
const cheerio = require("cheerio");const getFinanceData = async () => {const url = "https://www.google.com/finance/?hl=en";const response = await unirest
.get(url)
.header({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"})
const $ = cheerio.load(response.body);

Step-by-step explanation:

  1. In the sixth line, we made a GET request to the target URL.
  2. In the next line, we passed User-Agent as a header with the URL, so our bot can mimic a real organic user.
  3. Next, we load the response in a Cheerio instance variable.

Now, we will prepare our parser by searching the tags with the help CSS selector gadget, as stated above in the Requirements section.

Stocks on Google Finance

As you can see, the above tabs are under the tag .H8Chl .SxcTic. So, its parser will look like this:

let interested_top = [];$(".H8Ch1 .SxcTic").each((i,el) => {
interested_results.push({
stock_name: $(el).find(".ZvmM7").text(),
price: $(el).find(".YMlKec").text(),
change_in_price: $(el).find(".P2Luy").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

Now, we will parse the financial news results.

Google Finance News
Web Scraping Google Finance With Node JS 4

So, we found the tag for financial news results. This is how we parse it:

  let financial_news = [];
  $(".yY3Lee").each((i,el) => {
    financial_news.push({
        title: $(el).find(".Yfwt5").text(),
        thumbnail: $(el).find(".Z4idke").attr("src"),
        link: $(el).find("a").attr("href"),
        source: $(el).find(".sfyJob").text(),
        time: $(el).find(".Adak").text()
    })
  })

Similarly, if we follow the same process for finding the tags of the other blocks and tabs, it makes our whole parser look like this:

let interested_top = [];
$(".H8Ch1 .SxcTic").each((i,el) => {
interested_top.push({
stock_name: $(el).find(".ZvmM7").text(),
price: $(el).find(".YMlKec").text(),
change_in_price: $(el).find(".P2Luy").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

let financial_news = [];
//sometimes the scraper may return you empty news, so you have to 
//run the program once again
$(".yY3Lee").each((i,el) => {
    financial_news.push({
        title: $(el).find(".Yfwt5").text(),
        thumbnail: $(el).find(".Z4idke").attr("src"),
        link: $(el).find("a").attr("href"),
        source: $(el).find(".sfyJob").text(),
        time: $(el).find(".Adak").text()
    })
  })

let market_trends = [];
$(".gR2U6").each((i,el) => {
market_trends[i] = $(el).text();
})

let interested_bottom = [];
$(".tOzDHb").each((i,el) => {
interested_bottom.push({
stock_name: $(el).find(".RwFyvf").text(),
price: $(el).find(".YMlKec").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

let calendar_results = [];
$(".kQQz8e").each((i,el) => {
calendar_results.push({
stock_name: $(el).find(".qNqwJf").text(),
date_and_time: $(el).find(".fbt0Xc").text(),
link: $(el).find("a").attr("href")
 })
})

let most_followed_on_google = [];
$(".NaLFgc").each((i,el) => {
most_followed_on_google.push({
stock_name: $(el).find(".TwnKPb").text(),
following: $(el).find(".Iap8Fc").text().replace(" following", ""),
change_in_percentage: $(el).find(".JwB6zf").text(),
link: $(el).find("a").attr("href")
 })
})

console.log("interested_top:", interested_top)
console.log("financial_news:" ,financial_news)
console.log("market_trends:", market_trends)
console.log("interested_bottom:", interested_bottom)
console.log("calendar_results:", calendar_results)
console.log("most_followed_on_google:", most_followed_on_google)

Here are the results:

interested_top: [
  {
    stock_name: 'BSE Ltd',
    price: '₹586.75',
    change_in_price: '-₹14.25',
    change_in_percentage: '2.37%'
  },
  {
    stock_name: 'ITC Ltd',
    price: '₹360.70',
    change_in_price: '+₹7.20',
    change_in_percentage: '2.04%'
  },
  {
    stock_name: 'S&P 500',
    price: '3,766.18',
    change_in_price: '-61.93',
    change_in_percentage: '1.62%'
  },
  {
    stock_name: 'Dow Jones Industrial Average',
    price: '32,648.88',
    change_in_price: '-511.95',
    change_in_percentage: '1.54%'
  },
  {
    stock_name: 'NuStar Energy L.P.',
    price: '$16.27',
    change_in_price: '-$0.22',
    change_in_percentage: '1.33%'
  },
  {
    stock_name: 'Reliance Industries Ltd',
    price: '₹2,593.70',
    change_in_price: '-₹12.90',
    change_in_percentage: '0.49%'
  }
]
financial_news: [
  {
    title: 'RBI’s status quo and an indication that rate cuts would have to wait: \n' +       
      'What’s in store for borrowers, depositors',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcR31ko4m3eb2aup1eqEF4y7BykaoV4kXh25YX782BvgQ6VxWFwkoMARf7WDPQA',
    link: 'https://indianexpress.com/article/explained/explained-economics/rbi-mpc-status-quo-rate-impact-8970844/',
    source: 'The Indian Express',
    time: '4 hours ago'
  },
  {
    title: 'No more GNCAP crash certified cars for India: What will happen to safety \n' +    
      'ratings now explained',
    thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTdb21ydJZxp5HK4TCBfCmQmEKgwx8j6mkXdf3DDfP6qGvOkoUanncdaRe3T9k',
    link: 'https://m.timesofindia.com/auto/news/no-more-gncap-crash-certified-cars-for-india-what-will-happen-to-safety-ratings-now-explained/articleshow/104205701.cms',
    source: 'The Times of India',
    time: '4 hours ago'
  },
  {
    title: 'Long bonds’ historic 46% meltdown rivals burst of dot-com bubble',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQrKkHYPDCaEP7cn3M06a3X9adPoiqAxgT4xY-jsALdjl9T7Fm3pOOHE1QNnw4',
    link: 'https://m.economictimes.com/markets/bonds/long-bonds-historic-46-meltdown-rivals-burst-of-dot-com-bubble/articleshow/104200896.cms',
    source: 'The Economic Times',
    time: '7 hours ago'
  },
  {
    title: 'Stocks in news: Adani Wilmar, Bajaj Finance, Valiant Labs, Tata Motors, \n' +     
      'IndiGo, PB Fintech',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQpWDdGFeF5Se3vzLUSANZYkda9at4DvYG3e_xlA0QisCMe7bUmf-KwqzfgBrk',
    link: 'https://m.economictimes.com/markets/stocks/news/stocks-in-news-adani-wilmar-bajaj-finance-valiant-labs-tata-motors-indigo-pb-fintech/articleshow/104194195.cms',
    source: 'The Economic Times',
    time: '8 hours ago'
  },
  {
    title: 'Know the pros and cons of investing in your child’s name | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQzIN8xmhmctsXWZvrcGP-ZyrqFnkyvzIvrNhB9m2DWtFkrPsxvCth7TPoIVy4',
    link: 'https://www.livemint.com/money/personal-finance/start-planning-for-your-children-s-future-investments-tax-benefits-and-more-11696534535835.html',
    source: 'Mint',
    time: '14 hours ago'
  },
  {
    title: 'VigyanSpace: The Hindustan Times National School Science Quiz witnesses \n' +     
      '11,500+ registrations | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTpT1PwQJKMg8b8keElsUr_O5cO8PcihdunD12-mSYnjYuXFrrrvRtC5erUlG0',
    link: 'https://www.livemint.com/science/news/vigyanspace-the-hindustan-times-national-school-science-quiz-witnesses-11-500-registrations-11696502611691.html',
    source: 'Mint',
    time: '23 hours ago'
  },
  {
    title: 'NCLAT gives a third lessor the right to inspect Go First aircraft | Mint',        
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTSC56Q4kpZhZWkEuW2Jh0HGPckk9X7pDWXAutc0iRchRml7Cm3hQ0QMsj_zdM',
    link: 'https://www.livemint.com/news/india/nclat-gives-a-third-lessor-the-right-to-inspect-go-first-aircraft-11696493136191.html',
    source: 'Mint',
    time: '1 day ago'
  },
  {
    title: 'Is Sam Bankman-Fried Guilty? It’s Their Job to Convince the Jury. | Mint',        
    thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTQ7k5Lr8iX15o5iW8B0i-ceGC0L9tUFYLy_NC3hEQqlCUYZEhbULD3Tg-pZkY',
    link: 'https://www.livemint.com/technology/is-sam-bankman-fried-guilty-it-s-their-job-to-convince-the-jury-11696525789382.html',
    source: 'Mint',
    time: '17 hours ago'
  },
  {
    title: "Elon Musk plans to have three subscription tiers for X. Here's what you \n" +     
      'should know | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRhEKsT6n5uI4A8OP17G1qgIB_TEAt7TRIXiwyTsAl2kwIN8ruL-nU3NT4C_AM',
    link: 'https://www.livemint.com/technology/tech-news/elon-musks-x-is-testing-new-premium-subscription-plans-ceo-linda-yaccarino-confirms-11696557434632.html',
    source: 'Mint',
    time: '7 hours ago'
  }
]
market_trends: [ 'Market indexes', 'Climate leaders', 'Crypto', 'Currencies' ]
interested_bottom: [
  {
    stock_name: 'BSE SENSEX',
    price: '61,033.55',
    change_in_percentage: '0.25%'
  },
  {
    stock_name: 'NIFTY 50',
    price: '18,157.00',
    change_in_percentage: '0.25%'
  },
  {
    stock_name: 'Dow Jones Industrial Average',
    price: '32,648.88',
    change_in_percentage: '1.54%'
  },
  {
    stock_name: 'BSE Ltd',
    price: '₹586.75',
    change_in_percentage: '2.37%'
  },
  {
    stock_name: 'S&P 500',
    price: '3,766.18',
    change_in_percentage: '1.62%'
  },
  {
    stock_name: 'Reliance Industries Ltd',
    price: '₹2,593.70',
    change_in_percentage: '0.49%'
  },
  {
    stock_name: 'Yes Bank Limited',
    price: '₹16.55',
    change_in_percentage: '0.00%'
  },
  {
    stock_name: 'State Bank of India',
    price: '₹614.60',
    change_in_percentage: '0.073%'
  },
  {
    stock_name: 'ITC Ltd',
    price: '₹360.70',
    change_in_percentage: '2.04%'
  },
  {
    stock_name: 'NuStar Energy L.P.',
    price: '$16.27',
    change_in_percentage: '1.33%'
  },
  {
    stock_name: 'Mahindra And Mahindra Ltd',
    price: '₹1,334.70',
    change_in_percentage: '1.23%'
  },
  {
    stock_name: 'Future Retail Ltd',
    price: '₹3.35',
    change_in_percentage: '4.69%'
  },
  {
    stock_name: 'Vodafone Idea Ltd',
    price: '₹8.50',
    change_in_percentage: '1.80%'
  },
  {
    stock_name: 'OFS Capital Corp',
    price: '$10.48',
    change_in_percentage: '0.73%'
  },
  { stock_name: 'VIX', price: '26.13', change_in_percentage: '2.31%' },
  {
    stock_name: 'NCC Ltd',
    price: '₹72.05',
    change_in_percentage: '0.14%'
  },
  {
    stock_name: 'Indusind Bank Ltd',
    price: '₹1,149.00',
    change_in_percentage: '0.42%'
  },
  {
    stock_name: 'Nasdaq Composite',
    price: '10,391.74',
    change_in_percentage: '2.11%'
  }
]
calendar_results: [
  {
    stock_name: 'PI Industries Ltd.',
    date_and_time: 'Nov 9, 2022, 2:30 PM',
    link: 'https://www.google.com/finance/quote/PIIND:NSE'
  },
  {
    stock_name: 'Tata Motors',
    date_and_time: 'Nov 9, 2022, 6:30 PM',
    link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
  },
  {
    stock_name: 'Occidental Petroleum',
    date_and_time: 'Nov 9, 2022, 11:30 PM',
    link: 'https://www.google.com/finance/quote/OXY:NYSE'
  },
  {
    stock_name: 'Adani Green Energy',
    date_and_time: 'Nov 10, 2022, 12:08 PM',
    link: 'https://www.google.com/finance/quote/ADANIGREEN:NSE'
  },
  {
    stock_name: 'Hindalco Industries',
    date_and_time: 'Nov 11, 2022, 4:00 PM',
    link: 'https://www.google.com/finance/quote/HINDALCO:NSE'
  },
  {
    stock_name: 'Zomato',
    date_and_time: 'Nov 11, 2022, 5:00 PM',
    link: 'https://www.google.com/finance/quote/ZOMATO:NSE'
  }
]
most_followed_on_google: [
  {
    stock_name: 'Reliance Industries Ltd',
    following: '306K',
    change_in_percentage: '0.49%',
    link: 'https://www.google.com/finance/quote/RELIANCE:NSE'
  },
  {
    stock_name: 'State Bank of India',
    following: '272K',
    change_in_percentage: '0.07%',
    link: 'https://www.google.com/finance/quote/SBIN:NSE'
  },
  {
    stock_name: 'Yes Bank Limited',
    following: '228K',
    change_in_percentage: '0.00%',
    link: 'https://www.google.com/finance/quote/YESBANK:NSE'
  },
  {
    stock_name: 'Tata Motors Limited Fully Paid Ord. Shrs',
    following: '197K',
    change_in_percentage: '0.44%',
    link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
  },
  {
    stock_name: 'Infosys Ltd',
    following: '164K',
    change_in_percentage: '0.14%',
    link: 'https://www.google.com/finance/quote/INFY:NSE'
  },
  {
    stock_name: 'Tata Consultancy Services Limited',
    following: '156K',
    change_in_percentage: '0.73%',
    link: 'https://www.google.com/finance/quote/TCS:NSE'
  }
]

Using Google Finance API

Scraping Google Finance using the above method might work initially for a few requests. However, if you attempt to extract data at scale, there’s a strong likelihood that Google may block your IP address.

If your IP gets restricted due to scraping data at scale, it might obstruct your access to data for further analysis or usage. To overcome this problem, you need a reliable solution that can manage IP blockage and ensure continuous access to valuable data.

Serpdog’s Google Finance API is one of the most effective and robust solutions available in the market that allows seamless extraction of data through its scalable data pipeline.

Google Finance API

It allows its users to automate the extraction and organization of Finance Data by rotating millions of residential proxies at its backend.

We also offer 100 free requests on the first sign-up.

After getting registered on our website, you will get an API Key. Embed this API Key in the below code, you will be able to scrape Google Finance Results at a much faster speed.

const axios = require('axios');

axios.get('https://api.serpdog.io/finance?api_key=APIKEY&q=NIFTY_50:INDEXNSE&hl=en')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });

Conclusion:

Web Scraping for finance data is a valuable asset for financial researchers to monitor and identify potential future trends in the market, predicting economic trends and enabling businesses to track the financial condition of their competitors. 

In this tutorial, we employed a combination of Unirest and Cheerio JS to extract financial data from the main tab of the Google Finance webpage. Furthermore, we also explored the significance of scraping financial data and highlighted how Serpdog’s Google Finance API can assist in ensuring a consistent data pipeline from Google.

Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources:

  1. Web Scraping Google With Node JS — A Complete Guide
  2. Web Scraping Google Without Getting Blocked
  3. Scrape Google Organic Search Results
  4. Scrape Google Maps Reviews
  5. Web Scraping Google Maps