Getting Historical Price Data for Stocks & Crypto Using Yfinance and Finnhub

in LeoFinance4 years ago (edited)

yfinance-finnhub.png

One of the easiest ways to get historical price data for stocks and some of the cryptocurrencies is using Yahoo Finance API. However, few years ago Yahoo Finances discontinued its official API. Even Google Finance doesn't seem to provide and API for historical data either. Most website with historical price data seem to make it difficult to get the data from their websites programmatically. Fortunately, Yahoo Finance website for stock price quotes continues to display the data in a table format and makes easy to download.

Ran Aroussi has created a yfinance python module which makes it easy to retrieve historical price data from Yahoo Finance. This is the module I have been using for my trading related python projects. It seems it uses web scraping tools like sellenium or something to get the data from the website. It is not an official Yahoo Finance API. But it works. It is also integrated with pandas datareader that makes simpler to use and receive results as a pandas dataframe to do further needed computations.

After pip installing yfinance and pandas_datareader we can write this simple code to get the data we need.

import yfinance as yf
from pandas_datareader import data as pdr
import datetime as dt

yf.pdr_override()

end_date = dt.datetime.today()
start_date = end_date - dt.timedelta(days=365)
stock = 'BTC-USD'
df = pdr.get_data_yahoo(stock, start_date, end_date)

This simple codes provides a starting point to create more complex scripts to develop trading strategies, backtesting, records keeping, etc. It is great. It works. However, I felt a need to explore other alternatives for retrieving historical price data for several reasons.

With yfinance we can only get daily prices. While it is possible to get weekly data as well a little bit of tweaking or even web scraping from the Yahoo Finance website itself, it is not possible to get data for intraday timeframes like 4-hour, hourly, 15 minutes, etc. For some strategies daily data would be enough, for others intraday timeframes are needed.

Another reason to consider alternative options is that, since yfinance is not an official API, the way Yahoo Finance displays their data may change in the future. Like Google Finance they may choose to make it difficult to get the data programmatically in the future. Hopefully, they won't.

There are several paid services that provide historical price and market data. For experimental and personal use projects like mine paying for this kind of data may not justify the needs. For Apps, Websites, or more professional use paying for data probably would be a good idea to get reliable data. As I was exploring various options, I came across Finnhub. In comparison with other services Finnhub seems to be more interesting, well documented, and easy to use.

Finnhub provides free API calls, as well as paid options. Their free option definitely beats their competitors. To my understanding user can make 60 API calls per minute. In my opinion it is really good. Not only they provide historic price data for stocks and cryptocurrencies, they also provide all kinds of market and company related data.

To use Finnhub with python, we need to pip install finnhub module. Then we need to get an API key on their website which is provided quickly and for free. Here is a simple python code to achieve the same with finnhub as we did above with yfinance.

import finnhub
import pandas as pd

api_key: 'xxxxxxxxxxxxxxxx'
stock = 'SQ'
resolution = 'D'
end_date = dt.datetime.now()
start_date = end_date - dt.timedelta(days=365)
end = int(end_date.timestamp())
start = int(start_date.timestamp())

finnhub_client = finnhub.Client(api_key=api_key)
result = finnhub_client.stock_candles(stock, resolution, start, end)
df = pd.DataFrame(result)

The are a few differences the way finnhub retrieves that data compared to yfinance. First, the start and end date arguments do not take actual dates in form of year-month-day. Instead these values need to be passed in form of milliseconds. We can convert dates into milliseconds form using timestamp() method.

Second, the returned data is not in pandas dataframe format, instead it is in json format. For some using json might be more convenient. In my case I prefer pandas dataframe, which makes further computation easier. So, we can convert the result data to pandas dataframe.

Resolution represents the timeframe we can choose for the data. The above example uses 'D' for daily. We can change that to intraday timeframes like '5', '15', '30', '60' minutes or use 'W', 'M' for weekly and monthly data. Now this is really helpful. What I noticed though, when intraday resolutions/timeframes are chosen finnhub returns data covering less periods. For example if we wanted hourly data for the whole year, the returned data will not be for the whole year but rather about 3 months or even less. Also this changes with types of assets like stocks or crypto. Still very useful and enough data for most use cases.

As I was comparing the two modules and data returned by them, I saw a little discrepancy in the date representation of the stock/crypto prices. You can see on the below screenshot returned dates, prices and volume for Bitcoin by both yfinance and finnhub. First is finnhub, second is yfinance. You can see how prices and volumes match. However the dates are different.

s.png

I am not entirely sure why the dates do not match. My guess is finnhub goes with the start of the day for the candlestick, while yfinance goes with the end of the day. Overall, I don't think it is a big issue when each used on their own. But something important to keep in mind when comparing prices and strategies when using in combination with other platforms like tradingview.

I will continue using both modules, as both have a lot to offer. Especially finnhub, seems to have a lot of other important market data easily available and worth exploring.

Posted Using LeoFinance

Sort:  

I've spent quite a bit of time downloading data from yahoo finance. I might give this a try before I pull some price history next time around rather than go to their site for the CSV

Posted Using LeoFinance

Manually downloading files like that is definitely time consuming. Especially if you need to repeat the same thing dozens of times. I remember I would easily spend half hour daily downloading some work related pdf files. Then someone helped me to automate the process and 30 minute manual work was reduced to a minute.

I was planning to write a code to automatically download those csv files. I need weekly ones there and it seems very simple to do and useful. Especially when I need data for thousands of stocks. Will work on it next couple days.

Posted Using LeoFinance

Well from the looks of it finnhub seems quite documentated and have many options when you're using it, but then you said yfinance don't provide intradaydata? What other advantages does it have over yfinance?

For what I am doing right now, daily data is good enough. Due to its simplicity I am primarily using yfinance. But finnhub seems to be more powerful. There is so much more data they provide beyond just prices like company history, finances, etc. I didn't get to experiment with that yet. But once I finish current projects I do intend to play with finnhub more.

Since Finnhub provides paid services too, I think they will be around for a long time and keep improving their services. So for more complex projects and reliable data Finnhub would be a better choice.

Some of my scripts evaluate thousands of stocks, that's why using yfinance is better. There is not api call limit. With finnhub there is 60 api calls limit per minute. So the code will have to evaluate 50-60 stocks at a time not to exceed the call limits. To evaluate thousands stocks the code would have to run for 20 minutes. With yfinance all can be done much faster.

Posted Using LeoFinance

Yeah in the long run it seems finnhub will be the better choices that's if they keep adding cool features and improving a whole lot. Well a for now you're still stuck using yfinance till your project is over. Hmmm I get it now, thanks for explaining that further.

will you be creating dashboards with the data and if so where I can I get a hold of them :P

No, I am not planning on creating dashboards or anything like that at this time. I am just using these tools to test trading strategies.

Don't forget to tag this with #STEMGEEKS or #TECHNOLOGY, it certainly fits.
You have up to an hour to add the tag to get it included on https://STEMGeeks.net.

I didn't know. Thank you.

Posted Using LeoFinance

Upvoted by GITPLAIT!

We have a curation trial on Hive.vote. you can earn a passive income by delegating to @gitplait
We share 80 % of the curation rewards with the delegators.


To delegate, use the links or adjust 10HIVE, 20HIVE, 50HIVE, 100HIVE, 200HIVE, 500HIVE, 1,000HIVE, 10,000HIVE, 100,000HIVE


Join the Community and chat with us on Discord let’s solve problems & build together.

I have picked your post for my daily hive voting initiative, Keep it up and Hive On!!