Has anyone stored the historical market depth & tick data from MtGox? Is it available publically? Would be useful if you were making a bot.
-
Related, there is a BitcoinCharts API which provides some timeseries information bitcoincharts.com/about/markets-api– ACyclicOct 19, 2011 at 0:30
-
I have emailed MtGox and TradeHill about this, have not received a reply. Clark Moody doesn't store historic market depth information as there's too much data.– ACyclicOct 26, 2011 at 3:00
-
I think that's the main problem. It really is too much data.– nmatOct 26, 2011 at 14:22
-
1I grabbed a simple script I had and I got it fetching market depth every 5 seconds and saving the differences (I should use websockets, but I didn't have a script ready). After 2 days I have ~5MB of data so maybe this is not that hard to do. The data could be even saved in a more compressed format (I am just dumping what MtGox gives me). It's just a matter of setting up a server running 24/7 with a script like this.– nmatOct 28, 2011 at 14:49
Add a comment
|
4 Answers
Current market depth available at
All trades which happened in the last 24 hours available at
Retrieves 100 trades that happend after specified transaction
- https://mtgox.com/code/data/getTrades.php?since="transaction tid"
A script is required to retrive all historical data, like the one posted by mizerydearia. The idea is to cycle download request while incrementing "transaction tid" aka "lasttrade" to download all data in steps.
#!/bin/bash
outputfile=data
lasttrade=0
while test -n "$lasttrade";do
wget --quiet https://mtgox.com/code/data/getTrades.php?since=$lasttrade
if test "`cat getTrades.php\?since\=$lasttrade`" != "[]";then
cat getTrades.php\?since\=* >> $outputfile;
echo >> $outputfile;
lasttrade=`cat getTrades.php\?since\=$lasttrade|sed "s/,/\n/g"|grep tid|tail -n 1|cut -d ":" -f 2|cut -d "\"" -f 2`;
lasttradets=`cat getTrades.php\?since\=*|sed "s/,/\n/g"|grep date|tail -n 1|cut -d ":" -f 2`;
echo "$(date -d @$lasttradets) tid $lasttrade"
else
echo Finished
fi
rm getTrades.php\?since\=*;
done
See MtGox - API for more http API calls.
-
Any idea how big this file is today? I'm at about 3.5 gb right now and still going. Feb 22, 2013 at 16:01
-
-
I'm running a similar script -- it's currently up to 10/20/2011 @ ~ 155 MB uncompressed -- I'll let you know when it completes, and see if I can dump it up online somewhere. Jan 13, 2014 at 4:32
-
Well, I got to 3/11/2013 -- and it's only about ~670 MB so far. So there must be a bug in the above script. Probably related to the fact that it's parsing the JSON with Regex. Maybe an error is being thrown every so often, and it's starting over at the error code (using it as a TID) or something? Jan 13, 2014 at 9:46
-
I got up to date (1/14/2014) this morning -- completely uncompressed, the JSON is about 1.33 GB -- compressed in a regular .zip file, it's about 160 MB. I used the URL and the approach described above (slightly different implementation), and although it does not mention USD in the URL, the results appear to be in USD only (maybe that happened when they changed the API? This could also be the reason for the much smaller file size). At any rate, I've posted my dump here: bit.ly/1dMzqhJ Jan 14, 2014 at 12:15
To get ALL mtgoxUSD trades since the start you can use the following url:
www.bitcoincharts.com/t/trades.csv?symbol=mtgoxUSD&start=0
WARNING. This generates a 49mb csv at the moment and can take some time, so please only follow the link if you actually need the data.
Visit http://bitcoincharts.com/about/markets-api/ for more information.
Just noting here that MtGox tid's are not included in the data you get from bitcoincharts in that csv call API, and they switched from numerical indexing to microsecond timestamping a while back. In trying to maintain a db with integrity, you need the unique tid data... The history file is now over 100MB also.
Original MtGox Shared Dataset
Once upon a time there was a Google BigQuery dataset that was updated often with MtGox's official trade record. That's no longer being published because... ask someone who should explain it.
Data on Google BigQuery
The BigQuery shared MtGox Trades dataset is (@2014-04-20) available at https://bigquery.cloud.google.com/dataset/931531665892:mtgox
The BitTorrent Version
If you'd rather not BigQuery, but you still want that data, there's a torrent, here's the full-metal-magnet.
magnet:?xt=urn:btih:c87cd62f9484993f0153a47fc51f06fb43ed71ee&dn=931531665892-mtgox-trades.csv&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A80The MtGox Leak Database
In 2014 the Magic The Gathering Online eXchange (MtGox) server was compromised. Source code, user account details and trade data were made public.
For academic research the trade data is useful, here's a magnet to the original "public service release" file, MtGox2014Leak.zip. BEWARE: The TibanneBackOffice.zip file contains a trojan
magnet:?xt=urn:btih:b6545ecc7db8d44c8cbc4e93989edf8221af75f5&dn=2014+Mt.+Gox+Leak&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A6969&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337Alternatively, cryptoanarchic.me have kindly provided an HTTP mirrored version of MtGox2014Leak.zip.
Those are the two data sources I'm using in my research.
