4

I'm looking for sources of crypto currency data, preferably historical, which I can use to back-test my bots and/or analyze the market.

Is there a reliable source for this online ? Do exchanges offer historical data by any chance?

If not, how do I go about acquiring this data?

Improve this question

6 Answers 6

Reset to default
10

Acquiring Data

You have two options here:

  1. Check out one of the data providers online. Quandl offers (mostly) free historical data for a variety of pairs and exchanges. Alternatively, if you have bucks to spare, Coinigy offers high-quality data sets on a per-month pricing model.
  2. Roll your own data crawler, using a programming language of your choice and the various APIs available for exchanges.

I'll discuss the two options further below.

Acquiring Data from a Data Provider

Obviously, this option is the most convenient, however also possibly most expensive, depending on your required quality of the data. there is also no guarantee that the exchange, pair and/or time range you're looking for is present in the data base.

To Sum up:

Pro:

  • Least time-consuming
  • Data is likely to be cleaned and formatted
  • Many data providers offer a unified layout of data across several data sets, making it easy to compare them.

Cons:

  • Paid data is expensive for hobbyists (Market data at coinigy is at 30$ / Month, as of the time of writing)
  • Free data is often provided as is, leaving you with clean-up duty and post-processing tasks
  • If data is cleaned, artifacts can occur without you knowing, tempering with your research results.

Rolling your own Data Crawler

With just a little experience in programming and basic understanding of how an API works, you can quickly set up your own data acquisition tool. There's varying degrees of difficulty, of course, but at the very heart of it, the matter is quite simple:

  1. Choose an Exchange
  2. Choose an API
  3. Hook into API using code wizardry
  4. Start downloading data

I'll walk you through the necessary steps (sans the code - I feel like this is a topic for another StackExchange; I will, however, link you to libraries that will help you get started).

1. Choose an Exchange

I'm guessing you've done your research and thusly, will only point out a few things when considering the exchanges you want to fetch data for:

  • Make sure they have stable servers and connections (Exchanges in Asia, accessed from, for example, Europe, have notoriously unstable connections). Otherwise you'll have pot holes all over your data, perhaps even rendering it useless.
    • if you absolutely require the data, consider renting a server closer to the exchange server center's physical location.
  • Make sure they have meaningful volume. If there's only little volume changing owners over the course of a set time frame (usually, the 24h volume metric is always available), the exchange is likely not a very good choice. It skew the picture of the market.
  • Test their customer support first. If you happen to require their assistance some time later during crawling, a great support staff can make all the difference. Besides, it gives you a small indication of the exchange infrastructure's quality.

2. Choose an API

Choosing the right Application Programming Interface (API) depends on 2 things:

  1. How granular you need your data to be,

and

  1. what APIs the exchange has to offer (obviously).

REST APIs

If you're fine with, say, 1-Minute snapshots of data (i.e. tickers, order_books, trades etc), requesting data via a Representational State Transfer (REST) API is sufficient. This allows you to send a http request to a specified URL, and receive a response containing the data requested (usually in JSON-format).

The convenient thing about REST APIs is that they function mostly the same across exchanges - requests are sent, responses received and data evaluated by you. Some exchanges require you to use different request methods (such as POST) instead of the usual GET requests; however, this is usually only required for private endpoints (private meaning that you need to authenticate yourself first, before receiving data like your account's balance, etc), so if you're just after market data you shouldn't encounter them often.

A limitation of REST APIs is that they usually feature a request limit. The most commonly seen limit is 60 requests per minute, but can be more strict (or more loose - Bitfinex allows unlimited requests per minute). Some exchanges also employ a request counter - this means that certain requests increase your IP-connected counter by a certain value. Depending on your status with the exchange this counter decreases over time by a pre-determined value.

In addition to this (if you're thinking 'Well, hey! Why not fetch data every second then?'), market data is often cached. A snapshot of an API endpoint is stored for a set interval, before actually being updated on the server side. Hence, you might send a request every second, but nonetheless receive identical data until the server cache is refreshed.

WS APIs

WebSocket (WS) and WebSocketSecure (WSS) APIs are full-duplex connections, which allow the user to receive close-to-real-time updates of one or several API endpoints to which they've subscribed. Full-duplex simply means that you can not only receive, but also send data to the WebSocket connection. This is predominantly useful when running a bot that is supposed to trade on your behalf, as the feedback is usually faster.

In principle, you open a connection to the WS API, and subscribe to the endpoints you want data on. Typically, this is separated by endpoint and pair, but make sure to consult the API documentation - some exchanges do not employ a channel subscription model, and data comes flooding right in without further configuration.

The implementation of WebSockets ranges from trivial to complex, so expect to do some reasearch on how to access some of the exchanges' WS APIs. While the principle system remains identical across all of them, many exchanges use different protocols and/or services to provide their data - for example the WAMP protocol and Pusher, which add an additional layer of complexity, since they require a specific client to connect.

However, if you require tick-by-tick data, such as for real-time order book building, this is what you want to go with.

FIX APIs

The Financial Information eXchange (FIX) Protocol is a standard started in 1992, and now commonly used by institutions and brokers in the financial markets. It is by far the least available API at exchanges, with even fewer actually sending market data. It is most commonly used to place or cancel orders (for example at CoinbasePro). However, some exchanges offer real-time market data via FIX.

Keep in mind that you'll need extensive knowledge of the FIX protocol, such as setting up a FIX engine and proxies to connect to the exchange. This may, in some cases, incur monthly-recurring charges.

3. Hook into API using Code

Depending on what API you chose, as well as which programming language you intend to use, you have a wide variety of options.

Many exchanges provide client libraries for their APIs, some developed by the exchange dev team themselves, others are contributed by users (which are usually revised by the developer team before being quoted on the exchange's website).

There are many other libraries out there, for likely any language (for example BitEx for Python 3.x (to indulge in some shame-less self-promotion), which unifies core methods across a variety of REST APIs, as well as some WebSocket APIs).

Being a Python developer myself, I can only recommend using it. Especially if you're not bound to an environment. Using the requests library, for example, you can query an exchange with ease:

import requests

# Get a list of all asset pairs at Kraken.com 
print(requests.get('https://api.kraken.com/0/public/AssetPairs').json())

4.Start downloading data

Once you've set up your data crawlers, you need to release them! I personally run the REST crawler via cron jobs, and Websocket crawlers as daemons. But this is up to you.

It will of course take some time to acquire a meaningful set of data, but alas, you either pay with money or with time for the data you want.

Useful Links

API References

Data Sources

Feel free to add, correct or update any of this content in the comments below! Thanks.

Improve this answer
8
  • upvoted! any suggestions on how to go about websocket clients in python if you wanted real time data from say one or more of these APIs. I am a newbie and quite overwhelmed with the options available 1) tornado 2) twisted asyncio 3) asyncio 4) socket_io client 5) gevent 6) eventlet 7) threading with event loops per thread, what would be the right way to say fetch streams from several sources for your flask app, i am already using celery, thanks for the answer in advance!
    – PirateApp
    Commented Jun 25, 2018 at 13:08
  • 1
    It depends a bit on the API and what protocol it uses (pusher, pure wss, WAMP, etc) but generally I just went with what was easiest to get started with. I've used the websocket-client library most of the time, as that gets the job done well enough for me. Each of the clients run in a separate process and I pull everything together using ZMQ. As to what you should go with: roll a dice ? :D they're all good choices.
    – deepbrook
    Commented Jun 25, 2018 at 13:56
  • 1
    As I haven't used socket.io, I don't know - but in principle, WSS and pusher are just protocols - they still run via websockets, but require a specific procedure when interacting with the API. github.com/Crypto-toolbox/Thoth has a pusher and WSS client and using zmq to offer the data. Usually there's a few libraries linked on the respective exchange's api doc site which offer working clients, though.
    – deepbrook
    Commented Jun 26, 2018 at 7:36
  • 1
    I connect to the WSS and use ZMQ with a Pub/Sub pattern, where subscribers have a load balancer (several workers to handle incoming streams). I do write data to disk, but I do so at an interval defined by 400Kib cached or 5minutes passed since last write, whatever comes to pass first. I don't care about disconnects - If data is missing, there is nothing I can do about it anyway, although so far I haven't had the case that my socket disconnected due to heavy load, since all it does is relay it.
    – deepbrook
    Commented Jun 27, 2018 at 7:01
  • 1
    Yes, you can compare them to celery workers. And I do use python - ultimately though, it doesn't really matter. ZMQ is pretty much language agnostic and has wrappers for all kinds of languages - I stuck with python mostly because I work with it daily.
    – deepbrook
    Commented Jun 27, 2018 at 7:42
2

You can download the historical data of coinmarketcap.com with my developed crawler: https://github.com/roNn23/coinmarketcap-historical-data-crawler. But it's only getting the snapshots of coinmarketcap.com, maybe it's to wide for your needs.

Improve this answer
1

You can use my website www.cryptodatasets.com which provides exactly what you need. tick by tick historical BTC, ETH, and LTC prices and volume from Bitfinex and Hitbtc. It's exactly what I use to backtest my strategies/bots.

Improve this answer
2
  • neat! Do you plan on extending the available exchanges?
    – deepbrook
    Commented Nov 23, 2017 at 9:23
  • Absolutely, working on it in my free time!
    – MegaHotel
    Commented Nov 24, 2017 at 15:40
1

Just put up a website cryptoarchive.com.au with the data sets I use for my own modeling. One minute OHLVC data is available for free, tick-level data - very reasonably priced. Data from Binance for now.

Improve this answer
0

For historical data about crypto exchange rates and volumes you can check:

Improve this answer
1
  • This answer adds no new information, unfortunately.
    – deepbrook
    Commented Apr 5, 2019 at 10:28
-1

If you're especially looking at quality of data, Coinscious has the most accurate data. They offer both historical and real-time market data through flat files or API access. Their terminal also has good technical analytics on different coins, exchanges and indicators.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.