...

Scrape Google Finance With Node JS

by darshankhandelwal12
Scrape Google Finance

Google Finance Page is a data-rich page for traders and investors to get access to international exchanges, real-time financial news, and financial analysis to keep you updated with the current market scenario.

This type of data-rich website always has a couple of advantages:

  • It helps traders and investors in pricing analysis.
  • The financial data of a company can help people decide which stock to purchase in the future.
  • Technical indicators and analysis are used to research trends in the historical pricing of stock and predict the future trend of share prices.

Why scrape Google Finance?

Scraping Google Finance can provide you with various benefits:

Keeps You Updated: Scraping Google Finance keeps you updated about the current situation of the market around the world and helps you change your strategies according to the possible future trends in the market.

Data Analysis: The Google Finance data can provide you with insights into current market trends of an industry, its performance, and much more which can be used to discover any opportunities for investment in the market.

News analysis: You can use Google Finance data to scrape finance-related news around the world which can be used for sentimental analysis, tracking news trends, and much more.

Let’s start scraping Google Finance

We will be targeting this Google Finance Web Page link to get the stock market data.

Google Finance Web Page
Web Scraping Google Finance With Node JS 2

But before starting scraping Google Finance Results we need to install some NPM libraries so that we can move forward.

  1. Unirest JS
  2. Cheerio JS

You can also install both packages from the below line.

npm i unirest
npm i cheerio

We will be using Unirest JS to extract the raw HTML data and Cheerio for parsing the extracted data.

const unirest = require("unirest");
const cheerio = require("cheerio");
const getFinanceData = async () => {
const url = "https://www.google.com/finance/?hl=en";
const response = await unirest
.get(url)
.header({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"})
const $ = cheerio.load(response.body);

Step-by-step explanation:

  1. In the sixth line, we made a GET request to the target URL.
  2. In the next line, we passed User-Agent as a header with the URL, so our bot can mimic a real organic user.
  3. Next, we load the response in a Cheerio instance variable.

Now, we will prepare our parser by searching the tags with the help of the Inspect tools in our browser.

As you can see, the above tabs are under the tag .H8Chl .SxcTic. So, its parser will look like this:

let interested_top = [];$(".H8Ch1 .SxcTic").each((i,el) => {
interested_results.push({
stock_name: $(el).find(".ZvmM7").text(),
price: $(el).find(".YMlKec").text(),
change_in_price: $(el).find(".P2Luy").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

Now, we will parse the financial news results.

Each financial news is contained inside the tag yY3Lee. We will run a loop over it to obtain the required data points.

  let financial_news = [];
  $(".yY3Lee").each((i,el) => {
    financial_news.push({
        title: $(el).find(".Yfwt5").text(),
        thumbnail: $(el).find(".Z4idke").attr("src"),
        link: $(el).find("a").attr("href"),
        source: $(el).find(".sfyJob").text(),
        time: $(el).find(".Adak").text()
    })
  })

Similarly, if we follow the same process for finding the tags of the other blocks and tabs, it makes our whole parser look like this:

let interested_top = [];
$(".H8Ch1 .SxcTic").each((i,el) => {
interested_top.push({
stock_name: $(el).find(".ZvmM7").text(),
price: $(el).find(".YMlKec").text(),
change_in_price: $(el).find(".P2Luy").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

let financial_news = [];
//sometimes the scraper may return you empty news, so you have to 
//run the program once again
$(".yY3Lee").each((i,el) => {
    financial_news.push({
        title: $(el).find(".Yfwt5").text(),
        thumbnail: $(el).find(".Z4idke").attr("src"),
        link: $(el).find("a").attr("href"),
        source: $(el).find(".sfyJob").text(),
        time: $(el).find(".Adak").text()
    })
  })

let market_trends = [];
$(".gR2U6").each((i,el) => {
market_trends[i] = $(el).text();
})

let interested_bottom = [];
$(".tOzDHb").each((i,el) => {
interested_bottom.push({
stock_name: $(el).find(".RwFyvf").text(),
price: $(el).find(".YMlKec").text(),
change_in_percentage: $(el).find(".JwB6zf").text()
 })
})

let calendar_results = [];
$(".kQQz8e").each((i,el) => {
calendar_results.push({
stock_name: $(el).find(".qNqwJf").text(),
date_and_time: $(el).find(".fbt0Xc").text(),
link: $(el).find("a").attr("href")
 })
})

let most_followed_on_google = [];
$(".NaLFgc").each((i,el) => {
most_followed_on_google.push({
stock_name: $(el).find(".TwnKPb").text(),
following: $(el).find(".Iap8Fc").text().replace(" following", ""),
change_in_percentage: $(el).find(".JwB6zf").text(),
link: $(el).find("a").attr("href")
 })
})

console.log("interested_top:", interested_top)
console.log("financial_news:" ,financial_news)
console.log("market_trends:", market_trends)
console.log("interested_bottom:", interested_bottom)
console.log("calendar_results:", calendar_results)
console.log("most_followed_on_google:", most_followed_on_google)

Here are the results:

interested_top: [
  {
    stock_name: 'BSE Ltd',
    price: '₹586.75',
    change_in_price: '-₹14.25',
    change_in_percentage: '2.37%'
  },
  {
    stock_name: 'ITC Ltd',
    price: '₹360.70',
    change_in_price: '+₹7.20',
    change_in_percentage: '2.04%'
  },
  {
    stock_name: 'S&P 500',
    price: '3,766.18',
    change_in_price: '-61.93',
    change_in_percentage: '1.62%'
  },
  {
    stock_name: 'Dow Jones Industrial Average',
    price: '32,648.88',
    change_in_price: '-511.95',
    change_in_percentage: '1.54%'
  },
  {
    stock_name: 'NuStar Energy L.P.',
    price: '$16.27',
    change_in_price: '-$0.22',
    change_in_percentage: '1.33%'
  },
  {
    stock_name: 'Reliance Industries Ltd',
    price: '₹2,593.70',
    change_in_price: '-₹12.90',
    change_in_percentage: '0.49%'
  }
]
financial_news: [
  {
    title: 'RBI’s status quo and an indication that rate cuts would have to wait: \n' +       
      'What’s in store for borrowers, depositors',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcR31ko4m3eb2aup1eqEF4y7BykaoV4kXh25YX782BvgQ6VxWFwkoMARf7WDPQA',
    link: 'https://indianexpress.com/article/explained/explained-economics/rbi-mpc-status-quo-rate-impact-8970844/',
    source: 'The Indian Express',
    time: '4 hours ago'
  },
  {
    title: 'No more GNCAP crash certified cars for India: What will happen to safety \n' +    
      'ratings now explained',
    thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTdb21ydJZxp5HK4TCBfCmQmEKgwx8j6mkXdf3DDfP6qGvOkoUanncdaRe3T9k',
    link: 'https://m.timesofindia.com/auto/news/no-more-gncap-crash-certified-cars-for-india-what-will-happen-to-safety-ratings-now-explained/articleshow/104205701.cms',
    source: 'The Times of India',
    time: '4 hours ago'
  },
  {
    title: 'Long bonds’ historic 46% meltdown rivals burst of dot-com bubble',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQrKkHYPDCaEP7cn3M06a3X9adPoiqAxgT4xY-jsALdjl9T7Fm3pOOHE1QNnw4',
    link: 'https://m.economictimes.com/markets/bonds/long-bonds-historic-46-meltdown-rivals-burst-of-dot-com-bubble/articleshow/104200896.cms',
    source: 'The Economic Times',
    time: '7 hours ago'
  },
  {
    title: 'Stocks in news: Adani Wilmar, Bajaj Finance, Valiant Labs, Tata Motors, \n' +     
      'IndiGo, PB Fintech',
    thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQpWDdGFeF5Se3vzLUSANZYkda9at4DvYG3e_xlA0QisCMe7bUmf-KwqzfgBrk',
    link: 'https://m.economictimes.com/markets/stocks/news/stocks-in-news-adani-wilmar-bajaj-finance-valiant-labs-tata-motors-indigo-pb-fintech/articleshow/104194195.cms',
    source: 'The Economic Times',
    time: '8 hours ago'
  },
  {
    title: 'Know the pros and cons of investing in your child’s name | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQzIN8xmhmctsXWZvrcGP-ZyrqFnkyvzIvrNhB9m2DWtFkrPsxvCth7TPoIVy4',
    link: 'https://www.livemint.com/money/personal-finance/start-planning-for-your-children-s-future-investments-tax-benefits-and-more-11696534535835.html',
    source: 'Mint',
    time: '14 hours ago'
  },
  {
    title: 'VigyanSpace: The Hindustan Times National School Science Quiz witnesses \n' +     
      '11,500+ registrations | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTpT1PwQJKMg8b8keElsUr_O5cO8PcihdunD12-mSYnjYuXFrrrvRtC5erUlG0',
    link: 'https://www.livemint.com/science/news/vigyanspace-the-hindustan-times-national-school-science-quiz-witnesses-11-500-registrations-11696502611691.html',
    source: 'Mint',
    time: '23 hours ago'
  },
  {
    title: 'NCLAT gives a third lessor the right to inspect Go First aircraft | Mint',        
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTSC56Q4kpZhZWkEuW2Jh0HGPckk9X7pDWXAutc0iRchRml7Cm3hQ0QMsj_zdM',
    link: 'https://www.livemint.com/news/india/nclat-gives-a-third-lessor-the-right-to-inspect-go-first-aircraft-11696493136191.html',
    source: 'Mint',
    time: '1 day ago'
  },
  {
    title: 'Is Sam Bankman-Fried Guilty? It’s Their Job to Convince the Jury. | Mint',        
    thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTQ7k5Lr8iX15o5iW8B0i-ceGC0L9tUFYLy_NC3hEQqlCUYZEhbULD3Tg-pZkY',
    link: 'https://www.livemint.com/technology/is-sam-bankman-fried-guilty-it-s-their-job-to-convince-the-jury-11696525789382.html',
    source: 'Mint',
    time: '17 hours ago'
  },
  {
    title: "Elon Musk plans to have three subscription tiers for X. Here's what you \n" +     
      'should know | Mint',
    thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRhEKsT6n5uI4A8OP17G1qgIB_TEAt7TRIXiwyTsAl2kwIN8ruL-nU3NT4C_AM',
    link: 'https://www.livemint.com/technology/tech-news/elon-musks-x-is-testing-new-premium-subscription-plans-ceo-linda-yaccarino-confirms-11696557434632.html',
    source: 'Mint',
    time: '7 hours ago'
  }
]
market_trends: [ 'Market indexes', 'Climate leaders', 'Crypto', 'Currencies' ]
interested_bottom: [
  {
    stock_name: 'BSE SENSEX',
    price: '61,033.55',
    change_in_percentage: '0.25%'
  },
  {
    stock_name: 'NIFTY 50',
    price: '18,157.00',
    change_in_percentage: '0.25%'
  },
  {
    stock_name: 'Dow Jones Industrial Average',
    price: '32,648.88',
    change_in_percentage: '1.54%'
  },
  {
    stock_name: 'BSE Ltd',
    price: '₹586.75',
    change_in_percentage: '2.37%'
  },
  {
    stock_name: 'S&P 500',
    price: '3,766.18',
    change_in_percentage: '1.62%'
  },
  {
    stock_name: 'Reliance Industries Ltd',
    price: '₹2,593.70',
    change_in_percentage: '0.49%'
  },
  {
    stock_name: 'Yes Bank Limited',
    price: '₹16.55',
    change_in_percentage: '0.00%'
  },
  {
    stock_name: 'State Bank of India',
    price: '₹614.60',
    change_in_percentage: '0.073%'
  },
  {
    stock_name: 'ITC Ltd',
    price: '₹360.70',
    change_in_percentage: '2.04%'
  },
  {
    stock_name: 'NuStar Energy L.P.',
    price: '$16.27',
    change_in_percentage: '1.33%'
  },
  {
    stock_name: 'Mahindra And Mahindra Ltd',
    price: '₹1,334.70',
    change_in_percentage: '1.23%'
  },
  {
    stock_name: 'Future Retail Ltd',
    price: '₹3.35',
    change_in_percentage: '4.69%'
  },
  {
    stock_name: 'Vodafone Idea Ltd',
    price: '₹8.50',
    change_in_percentage: '1.80%'
  },
  {
    stock_name: 'OFS Capital Corp',
    price: '$10.48',
    change_in_percentage: '0.73%'
  },
  { stock_name: 'VIX', price: '26.13', change_in_percentage: '2.31%' },
  {
    stock_name: 'NCC Ltd',
    price: '₹72.05',
    change_in_percentage: '0.14%'
  },
  {
    stock_name: 'Indusind Bank Ltd',
    price: '₹1,149.00',
    change_in_percentage: '0.42%'
  },
  {
    stock_name: 'Nasdaq Composite',
    price: '10,391.74',
    change_in_percentage: '2.11%'
  }
]
calendar_results: [
  {
    stock_name: 'PI Industries Ltd.',
    date_and_time: 'Nov 9, 2022, 2:30 PM',
    link: 'https://www.google.com/finance/quote/PIIND:NSE'
  },
  {
    stock_name: 'Tata Motors',
    date_and_time: 'Nov 9, 2022, 6:30 PM',
    link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
  },
  {
    stock_name: 'Occidental Petroleum',
    date_and_time: 'Nov 9, 2022, 11:30 PM',
    link: 'https://www.google.com/finance/quote/OXY:NYSE'
  },
  {
    stock_name: 'Adani Green Energy',
    date_and_time: 'Nov 10, 2022, 12:08 PM',
    link: 'https://www.google.com/finance/quote/ADANIGREEN:NSE'
  },
  {
    stock_name: 'Hindalco Industries',
    date_and_time: 'Nov 11, 2022, 4:00 PM',
    link: 'https://www.google.com/finance/quote/HINDALCO:NSE'
  },
  {
    stock_name: 'Zomato',
    date_and_time: 'Nov 11, 2022, 5:00 PM',
    link: 'https://www.google.com/finance/quote/ZOMATO:NSE'
  }
]
most_followed_on_google: [
  {
    stock_name: 'Reliance Industries Ltd',
    following: '306K',
    change_in_percentage: '0.49%',
    link: 'https://www.google.com/finance/quote/RELIANCE:NSE'
  },
  {
    stock_name: 'State Bank of India',
    following: '272K',
    change_in_percentage: '0.07%',
    link: 'https://www.google.com/finance/quote/SBIN:NSE'
  },
  {
    stock_name: 'Yes Bank Limited',
    following: '228K',
    change_in_percentage: '0.00%',
    link: 'https://www.google.com/finance/quote/YESBANK:NSE'
  },
  {
    stock_name: 'Tata Motors Limited Fully Paid Ord. Shrs',
    following: '197K',
    change_in_percentage: '0.44%',
    link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
  },
  {
    stock_name: 'Infosys Ltd',
    following: '164K',
    change_in_percentage: '0.14%',
    link: 'https://www.google.com/finance/quote/INFY:NSE'
  },
  {
    stock_name: 'Tata Consultancy Services Limited',
    following: '156K',
    change_in_percentage: '0.73%',
    link: 'https://www.google.com/finance/quote/TCS:NSE'
  }
]

Using Google Finance API

Scraping Google Finance using the above method might work initially for a few requests. However, if you attempt to extract data at scale, there’s a strong likelihood that Google may block your IP address.

If your IP gets restricted due to scraping data at scale, it might obstruct your access to data for further analysis or usage. To overcome this problem, you need a reliable solution that can manage IP blockage and ensure continuous access to valuable data.

Serpdog’s Google Finance API is one of the most effective and robust solutions available in the market that allows seamless extraction of data through its scalable data pipeline.

Google Finance API

It allows its users to automate the extraction and organization of Finance Data by rotating millions of residential proxies at its backend.

We also offer 100 free requests on the first sign-up.

Try Google Finance API For Free!

After getting registered on our website, you will get an API Key. Embed this API Key in the below code, you will be able to scrape Google Finance Results at a much faster speed.

const axios = require('axios');

axios.get('https://api.serpdog.io/finance?api_key=APIKEY&q=NIFTY_50:INDEXNSE&hl=en')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });

Conclusion:

Web Scraping for finance data is a valuable asset for financial researchers to monitor and identify potential future trends in the market, predicting economic trends and enabling businesses to track the financial condition of their competitors. 

In this tutorial, we employed a combination of Unirest and Cheerio JS to extract financial data from the main tab of the Google Finance webpage. Furthermore, we also explored the significance of scraping financial data and highlighted how Serpdog’s Google Finance API can assist in ensuring a consistent data pipeline from Google.

Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!

Additional Resources:

  1. Web Scraping Google With Node JS — A Complete Guide
  2. Web Scraping Google Without Getting Blocked
  3. Scrape Google Organic Search Results
  4. Scrape Google Maps Reviews
  5. Web Scraping Google Maps