Google Finance Page is a data-rich page for traders and investors to get access to international exchanges, real-time financial news, and financial analysis to keep you updated with the current market scenario.
This type of data-rich website always has a couple of advantages:
- It helps traders and investors in pricing analysis.
- The financial data of a company can help people decide which stock to purchase in the future.
- Technical indicators and analysis are used to research trends in the historical pricing of stock and predict the future trend of share prices.
Why scrape Google Finance?
Aggregating finance data can provide you with various benefits:
Keeps You Updated: It keeps you updated about the current situation of the market around the world and helps you change your strategies according to the possible future trends in the market.
Data Analysis: The financial data can provide you with insights into current market trends of an industry, its performance, and much more which can be used to discover any opportunities for investment in the market.
News analysis: You can use financial data to get finance-related news around the world which can be used for sentimental analysis, tracking news trends, and much more.
Let’s start scraping
We will be targeting this Google Finance Main Page to get the stock market data.
But before starting, we will install the below Python libraries to move to the next step.
You can install both packages from the below line.
pip install requests
pip install beautifulsoup4
We will be using Requests and BS4 to extract and parse the raw HTML data.
import requests
from bs4 import BeautifulSoup
url = "https://www.google.com/finance/?hl=en"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")
Step-by-step explanation:
- After importing the libraries, we declared our target URL and header to be passed with the GET request.
- In the next line, we created an HTTP GET connection.
- Next, we created an instance of BS4 to parse the HTML.
Now, we will prepare our parser by searching the tags with the help of the Inspect tools in our browser.
As you can see, the above tabs are under the tag .H8Chl .SxcTic. So, its parser will look like this:
interested_top = []
for el in soup.select(".H8Ch1 .SxcTic"):
interested_top.append({
"stock_name": el.select_one(".ZvmM7").get_text(),
"price": el.select_one(".YMlKec").get_text(),
"change_in_price": el.select_one(".P2Luy").get_text(),
"change_in_percentage": el.select_one(".JwB6zf").get_text()
})
Now, we will parse the financial news results.
Each financial news is contained inside the tag yY3Lee. We will run a loop over it to obtain the required data points.
financial_news = []
for el in soup.select(".yY3Lee"):
financial_news.append({
"title": el.select_one(".Yfwt5").get_text(),
"thumbnail": el.select_one(".Z4idke").get("src"),
"link": el.select_one("a").get("href"),
"source": el.select_one(".sfyJob").get_text(),
"time": el.select_one(".Adak").get_text()
})
Similarly, if we follow the same process for finding the tags of the other blocks and tabs, it makes our whole parser look like this:
interested_top = []
for el in soup.select(".H8Ch1 .SxcTic"):
interested_top.append({
"stock_name": el.select_one(".ZvmM7").get_text(),
"price": el.select_one(".YMlKec").get_text(),
"change_in_price": el.select_one(".P2Luy").get_text(),
"change_in_percentage": el.select_one(".JwB6zf").get_text()
})
financial_news = []
for el in soup.select(".yY3Lee"):
financial_news.append({
"title": el.select_one(".Yfwt5").get_text(),
"thumbnail": el.select_one(".Z4idke").get("src"),
"link": el.select_one("a").get("href"),
"source": el.select_one(".sfyJob").get_text(),
"time": el.select_one(".Adak").get_text()
})
market_trends = [el.get_text() for el in soup.select(".gR2U6")]
interested_bottom = []
for el in soup.select(".tOzDHb"):
interested_bottom.append({
"stock_name": el.select_one(".RwFyvf").get_text(),
"price": el.select_one(".YMlKec").get_text(),
"change_in_percentage": el.select_one(".JwB6zf").get_text()
})
calendar_results = []
for el in soup.select(".kQQz8e"):
calendar_results.append({
"stock_name": el.select_one(".qNqwJf").get_text(),
"date_and_time": el.select_one(".fbt0Xc").get_text(),
"link": el.select_one("a").get("href")
})
most_followed_on_google = []
for el in soup.select(".NaLFgc"):
most_followed_on_google.append({
"stock_name": el.select_one(".TwnKPb").get_text(),
"following": el.select_one(".Iap8Fc").get_text().replace(" following", ""),
"change_in_percentage": el.select_one(".JwB6zf").get_text(),
"link": el.select_one("a").get("href")
})
print("interested_top:", interested_top)
print("financial_news:", financial_news)
print("market_trends:", market_trends)
print("interested_bottom:", interested_bottom)
print("calendar_results:", calendar_results)
print("most_followed_on_google:", most_followed_on_google)
Here are the results:
interested_top: [
{
stock_name: 'BSE Ltd',
price: '₹586.75',
change_in_price: '-₹14.25',
change_in_percentage: '2.37%'
},
{
stock_name: 'ITC Ltd',
price: '₹360.70',
change_in_price: '+₹7.20',
change_in_percentage: '2.04%'
},
{
stock_name: 'S&P 500',
price: '3,766.18',
change_in_price: '-61.93',
change_in_percentage: '1.62%'
},
{
stock_name: 'Dow Jones Industrial Average',
price: '32,648.88',
change_in_price: '-511.95',
change_in_percentage: '1.54%'
},
{
stock_name: 'NuStar Energy L.P.',
price: '$16.27',
change_in_price: '-$0.22',
change_in_percentage: '1.33%'
},
{
stock_name: 'Reliance Industries Ltd',
price: '₹2,593.70',
change_in_price: '-₹12.90',
change_in_percentage: '0.49%'
}
]
financial_news: [
{
title: 'RBI’s status quo and an indication that rate cuts would have to wait: \n' +
'What’s in store for borrowers, depositors',
thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcR31ko4m3eb2aup1eqEF4y7BykaoV4kXh25YX782BvgQ6VxWFwkoMARf7WDPQA',
link: 'https://indianexpress.com/article/explained/explained-economics/rbi-mpc-status-quo-rate-impact-8970844/',
source: 'The Indian Express',
time: '4 hours ago'
},
{
title: 'No more GNCAP crash certified cars for India: What will happen to safety \n' +
'ratings now explained',
thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTdb21ydJZxp5HK4TCBfCmQmEKgwx8j6mkXdf3DDfP6qGvOkoUanncdaRe3T9k',
link: 'https://m.timesofindia.com/auto/news/no-more-gncap-crash-certified-cars-for-india-what-will-happen-to-safety-ratings-now-explained/articleshow/104205701.cms',
source: 'The Times of India',
time: '4 hours ago'
},
{
title: 'Long bonds’ historic 46% meltdown rivals burst of dot-com bubble',
thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQrKkHYPDCaEP7cn3M06a3X9adPoiqAxgT4xY-jsALdjl9T7Fm3pOOHE1QNnw4',
link: 'https://m.economictimes.com/markets/bonds/long-bonds-historic-46-meltdown-rivals-burst-of-dot-com-bubble/articleshow/104200896.cms',
source: 'The Economic Times',
time: '7 hours ago'
},
{
title: 'Stocks in news: Adani Wilmar, Bajaj Finance, Valiant Labs, Tata Motors, \n' +
'IndiGo, PB Fintech',
thumbnail: 'https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcQpWDdGFeF5Se3vzLUSANZYkda9at4DvYG3e_xlA0QisCMe7bUmf-KwqzfgBrk',
link: 'https://m.economictimes.com/markets/stocks/news/stocks-in-news-adani-wilmar-bajaj-finance-valiant-labs-tata-motors-indigo-pb-fintech/articleshow/104194195.cms',
source: 'The Economic Times',
time: '8 hours ago'
},
{
title: 'Know the pros and cons of investing in your child’s name | Mint',
thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQzIN8xmhmctsXWZvrcGP-ZyrqFnkyvzIvrNhB9m2DWtFkrPsxvCth7TPoIVy4',
link: 'https://www.livemint.com/money/personal-finance/start-planning-for-your-children-s-future-investments-tax-benefits-and-more-11696534535835.html',
source: 'Mint',
time: '14 hours ago'
},
{
title: 'VigyanSpace: The Hindustan Times National School Science Quiz witnesses \n' +
'11,500+ registrations | Mint',
thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTpT1PwQJKMg8b8keElsUr_O5cO8PcihdunD12-mSYnjYuXFrrrvRtC5erUlG0',
link: 'https://www.livemint.com/science/news/vigyanspace-the-hindustan-times-national-school-science-quiz-witnesses-11-500-registrations-11696502611691.html',
source: 'Mint',
time: '23 hours ago'
},
{
title: 'NCLAT gives a third lessor the right to inspect Go First aircraft | Mint',
thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTSC56Q4kpZhZWkEuW2Jh0HGPckk9X7pDWXAutc0iRchRml7Cm3hQ0QMsj_zdM',
link: 'https://www.livemint.com/news/india/nclat-gives-a-third-lessor-the-right-to-inspect-go-first-aircraft-11696493136191.html',
source: 'Mint',
time: '1 day ago'
},
{
title: 'Is Sam Bankman-Fried Guilty? It’s Their Job to Convince the Jury. | Mint',
thumbnail: 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTQ7k5Lr8iX15o5iW8B0i-ceGC0L9tUFYLy_NC3hEQqlCUYZEhbULD3Tg-pZkY',
link: 'https://www.livemint.com/technology/is-sam-bankman-fried-guilty-it-s-their-job-to-convince-the-jury-11696525789382.html',
source: 'Mint',
time: '17 hours ago'
},
{
title: "Elon Musk plans to have three subscription tiers for X. Here's what you \n" +
'should know | Mint',
thumbnail: 'https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcRhEKsT6n5uI4A8OP17G1qgIB_TEAt7TRIXiwyTsAl2kwIN8ruL-nU3NT4C_AM',
link: 'https://www.livemint.com/technology/tech-news/elon-musks-x-is-testing-new-premium-subscription-plans-ceo-linda-yaccarino-confirms-11696557434632.html',
source: 'Mint',
time: '7 hours ago'
}
]
market_trends: [ 'Market indexes', 'Climate leaders', 'Crypto', 'Currencies' ]
interested_bottom: [
{
stock_name: 'BSE SENSEX',
price: '61,033.55',
change_in_percentage: '0.25%'
},
{
stock_name: 'NIFTY 50',
price: '18,157.00',
change_in_percentage: '0.25%'
},
{
stock_name: 'Dow Jones Industrial Average',
price: '32,648.88',
change_in_percentage: '1.54%'
},
{
stock_name: 'BSE Ltd',
price: '₹586.75',
change_in_percentage: '2.37%'
},
{
stock_name: 'S&P 500',
price: '3,766.18',
change_in_percentage: '1.62%'
},
{
stock_name: 'Reliance Industries Ltd',
price: '₹2,593.70',
change_in_percentage: '0.49%'
},
{
stock_name: 'Yes Bank Limited',
price: '₹16.55',
change_in_percentage: '0.00%'
},
{
stock_name: 'State Bank of India',
price: '₹614.60',
change_in_percentage: '0.073%'
},
{
stock_name: 'ITC Ltd',
price: '₹360.70',
change_in_percentage: '2.04%'
},
{
stock_name: 'NuStar Energy L.P.',
price: '$16.27',
change_in_percentage: '1.33%'
},
{
stock_name: 'Mahindra And Mahindra Ltd',
price: '₹1,334.70',
change_in_percentage: '1.23%'
},
{
stock_name: 'Future Retail Ltd',
price: '₹3.35',
change_in_percentage: '4.69%'
},
{
stock_name: 'Vodafone Idea Ltd',
price: '₹8.50',
change_in_percentage: '1.80%'
},
{
stock_name: 'OFS Capital Corp',
price: '$10.48',
change_in_percentage: '0.73%'
},
{ stock_name: 'VIX', price: '26.13', change_in_percentage: '2.31%' },
{
stock_name: 'NCC Ltd',
price: '₹72.05',
change_in_percentage: '0.14%'
},
{
stock_name: 'Indusind Bank Ltd',
price: '₹1,149.00',
change_in_percentage: '0.42%'
},
{
stock_name: 'Nasdaq Composite',
price: '10,391.74',
change_in_percentage: '2.11%'
}
]
calendar_results: [
{
stock_name: 'PI Industries Ltd.',
date_and_time: 'Nov 9, 2022, 2:30 PM',
link: 'https://www.google.com/finance/quote/PIIND:NSE'
},
{
stock_name: 'Tata Motors',
date_and_time: 'Nov 9, 2022, 6:30 PM',
link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
},
{
stock_name: 'Occidental Petroleum',
date_and_time: 'Nov 9, 2022, 11:30 PM',
link: 'https://www.google.com/finance/quote/OXY:NYSE'
},
{
stock_name: 'Adani Green Energy',
date_and_time: 'Nov 10, 2022, 12:08 PM',
link: 'https://www.google.com/finance/quote/ADANIGREEN:NSE'
},
{
stock_name: 'Hindalco Industries',
date_and_time: 'Nov 11, 2022, 4:00 PM',
link: 'https://www.google.com/finance/quote/HINDALCO:NSE'
},
{
stock_name: 'Zomato',
date_and_time: 'Nov 11, 2022, 5:00 PM',
link: 'https://www.google.com/finance/quote/ZOMATO:NSE'
}
]
most_followed_on_google: [
{
stock_name: 'Reliance Industries Ltd',
following: '306K',
change_in_percentage: '0.49%',
link: 'https://www.google.com/finance/quote/RELIANCE:NSE'
},
{
stock_name: 'State Bank of India',
following: '272K',
change_in_percentage: '0.07%',
link: 'https://www.google.com/finance/quote/SBIN:NSE'
},
{
stock_name: 'Yes Bank Limited',
following: '228K',
change_in_percentage: '0.00%',
link: 'https://www.google.com/finance/quote/YESBANK:NSE'
},
{
stock_name: 'Tata Motors Limited Fully Paid Ord. Shrs',
following: '197K',
change_in_percentage: '0.44%',
link: 'https://www.google.com/finance/quote/TATAMOTORS:NSE'
},
{
stock_name: 'Infosys Ltd',
following: '164K',
change_in_percentage: '0.14%',
link: 'https://www.google.com/finance/quote/INFY:NSE'
},
{
stock_name: 'Tata Consultancy Services Limited',
following: '156K',
change_in_percentage: '0.73%',
link: 'https://www.google.com/finance/quote/TCS:NSE'
}
]
Using Google Finance API
Extracting financial data using the above method might work initially for a few requests. However, if you attempt to extract data at scale, there’s a strong likelihood that Google may block your IP address.
If your IP gets restricted due to scraping data at scale, it might obstruct your access to data for further analysis or usage. To overcome this problem, you need a reliable solution that can manage IP blockage and ensure continuous access to valuable data.
Serpdog’s Google Finance API is one of the most effective and robust solutions available in the market that allows seamless extraction of data through its scalable data pipeline.
It allows its users to automate the extraction and organization of Finance Data by rotating millions of residential proxies at its backend.
We also offer 100 free requests on the first sign-up.
After getting registered on our website, you will get an API Key. Embed this API Key in the below code, you will be able to scrape finance Results at a much faster speed.
import requests
payload = {'api_key': 'APIKEY', 'q':'NIFTY_50:INDEXNSE' , 'hl':'en'}
resp = requests.get('https://api.serpdog.io/finance', params=payload)
print (resp.text)
Conclusion:
Financial data extraction is a valuable asset for researchers to monitor and identify potential future trends in the market, predicting economic trends and enabling businesses to track the financial condition of their competitors.
In this tutorial, we employed a combination of Requests and BS4 to extract financial data from the main tab of the Google Finance webpage. Furthermore, we also explored the significance of scraping financial data and highlighted how Serpdog’s Google Finance API can assist in ensuring a consistent data pipeline from Google.
Feel free to message me anything you need clarification on. Follow me on Twitter. Thanks for reading!