You have two options here:
- Check out one of the data providers online. Quandl offers (mostly) free historical data for a variety of pairs and exchanges. Alternatively, if you have bucks to spare, Coinigy offers high-quality data sets on a per-month pricing model.
- Roll your own data crawler, using a programming language of your choice and the various APIs available for exchanges.
I'll discuss the two options further below.
Acquiring Data from a Data Provider
Obviously, this option is the most convenient, however also possibly most expensive, depending on your required quality of the data.
there is also no guarantee that the exchange, pair and/or time range you're looking for is present in the data base.
To Sum up:
- Least time-consuming
- Data is likely to be cleaned and formatted
- Many data providers offer a unified layout of data across several data sets, making it easy to compare them.
- Paid data is expensive for hobbyists (Market data at coinigy is at 30$ / Month, as of the time of writing)
- Free data is often provided as is, leaving you with clean-up duty and post-processing tasks
- If data is cleaned, artifacts can occur without you knowing, tempering with your research results.
Rolling your own Data Crawler
With just a little experience in programming and basic understanding of how an API works, you can quickly set up your own data acquisition tool. There's varying degrees of difficulty, of course, but at the very heart of it, the matter is quite simple:
- Choose an Exchange
- Choose an API
- Hook into API using code wizardry
- Start downloading data
I'll walk you through the necessary steps (sans the code - I feel like this is a topic for another StackExchange; I will, however, link you to libraries that will help you get started).
1. Choose an Exchange
I'm guessing you've done your research and thusly, will only point out a few things when considering the exchanges you want to fetch data for:
- Make sure they have stable servers and connections (Exchanges in Asia, accessed from, for example, Europe, have notoriously unstable connections). Otherwise you'll have pot holes all over your data, perhaps even rendering it useless.
- if you absolutely require the data, consider renting a server closer to the exchange server center's physical location.
- Make sure they have meaningful volume. If there's only little volume changing owners over the course of a set time frame (usually, the 24h volume metric is always available), the exchange is likely not a very good choice. It skew the picture of the market.
- Test their customer support first. If you happen to require their assistance some time later during crawling, a great support staff can make all the difference. Besides, it gives you a small indication of the exchange infrastructure's quality.
2. Choose an
Choosing the right Application Programming Interface (
API) depends on 2 things:
- How granular you need your data to be,
APIs the exchange has to offer (obviously).
If you're fine with, say, 1-Minute snapshots of data (i.e. tickers, order_books, trades etc), requesting data via a Representational State Transfer (REST) API is sufficient. This allows you to send a
http request to a specified URL, and receive a response containing the data requested (usually in
The convenient thing about
REST APIs is that they function mostly the same across exchanges - requests are sent, responses received and data evaluated by you. Some exchanges require you to use different request methods (such as
POST) instead of the usual
GET requests; however, this is usually only required for private endpoints (private meaning that you need to authenticate yourself first, before receiving data like your account's balance, etc), so if you're just after market data you shouldn't encounter them often.
A limitation of
REST APIs is that they usually feature a request limit. The most commonly seen limit is
60 requests per minute, but can be more strict (or more loose - Bitfinex allows unlimited
requests per minute). Some exchanges also employ a request counter - this means that certain requests increase your
IP-connected counter by a certain value. Depending on your status with the exchange this counter decreases over time by a pre-determined value.
In addition to this (if you're thinking 'Well, hey! Why not fetch data every second then?'), market data is often cached. A snapshot of an API endpoint is stored for a set interval, before actually being updated on the server side. Hence, you might send a request every second, but nonetheless receive identical data until the server cache is refreshed.
WS) and WebSocketSecure (
WSS) APIs are full-duplex connections, which allow the user to receive close-to-real-time updates of one or several
API endpoints to which they've subscribed.
Full-duplex simply means that you can not only receive, but also send data to the
WebSocket connection. This is predominantly useful when running a bot that is supposed to trade on your behalf, as the feedback is usually faster.
In principle, you open a connection to the
WS API, and subscribe to the endpoints you want data on. Typically, this is separated by endpoint and pair, but make sure to consult the
API documentation - some exchanges do not employ a
channel subscription model, and data comes flooding right in without further configuration.
The implementation of
WebSockets ranges from trivial to complex, so expect to do some reasearch on how to access some of the exchanges'
WS APIs. While the principle system remains identical across all of them, many exchanges use different protocols and/or services to provide their data - for example the
WAMP protocol and Pusher, which add an additional layer of complexity, since they require a specific client to connect.
However, if you require tick-by-tick data, such as for real-time order book building, this is what you want to go with.
Financial Information eXchange (
FIX) Protocol is a standard started in 1992, and now commonly used by institutions and brokers in the financial markets. It is by far the least available
API at exchanges, with even fewer actually sending market data. It is most commonly used to place or cancel orders (for example at CoinbasePro). However, some exchanges offer real-time market data via FIX.
Keep in mind that you'll need extensive knowledge of the FIX protocol, such as setting up a FIX engine and proxies to connect to the exchange. This may, in some cases, incur monthly-recurring charges.
3. Hook into API using Code
Depending on what API you chose, as well as which programming language you intend to use, you have a wide variety of options.
Many exchanges provide client libraries for their APIs, some developed by the exchange dev team themselves, others are contributed by users (which are usually revised by the developer team before being quoted on the exchange's website).
There are many other libraries out there, for likely any language (for example BitEx for Python 3.x (to indulge in some shame-less self-promotion), which unifies core methods across a variety of
APIs, as well as some
Being a Python developer myself, I can only recommend using it. Especially if you're not bound to an environment.
requests library, for example, you can query an exchange with ease:
# Get a list of all asset pairs at Kraken.com
4.Start downloading data
Once you've set up your data crawlers, you need to release them! I personally run the REST crawler via
cron jobs, and
Websocket crawlers as
daemons. But this is up to you.
It will of course take some time to acquire a meaningful set of data, but alas, you either pay with money or with time for the data you want.
Feel free to add, correct or update any of this content in the comments below! Thanks.