{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# The `fxcmpy_data_reader`\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This section is about the `fxcmpy_data_reader` classes\n", "\n", "* `fxcmpy_tick_data_reader` for historical tick data and\n", "* `fxcmpy_candles_data_reader` for historicial candles data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The `fxcmpy_tick_data_reader`" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import importlib\n", "import datetime as dt\n", "from fxcmpy import fxcmpy_tick_data_reader as tdr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `fxcmpy_tick_data_reader` reads tick data from the **FXCM** servers for a given symbol and a given time window." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Available Symbols " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is tick data for the following symbols (currency pairs) available." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('AUDCAD', 'AUDCHF', 'AUDJPY', 'AUDNZD', 'CADCHF', 'EURAUD', 'EURCHF', 'EURGBP', 'EURJPY', 'EURUSD', 'GBPCHF', 'GBPJPY', 'GBPNZD', 'GBPUSD', 'GBPCHF', 'GBPJPY', 'GBPNZD', 'NZDCAD', 'NZDCHF', 'NZDJPY', 'NZDUSD', 'USDCAD', 'USDCHF', 'USDJPY')\n" ] } ], "source": [ "print(tdr.get_available_symbols())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading Tick Data " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, start and end dates need to be specified." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "start = dt.datetime(2018, 2, 1)\n", "end = dt.datetime(2018, 2, 15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note, that the tick data is stored in weekly packages." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fetching data from: https://tickdata.fxcorporate.com/EURUSD/2018/5.csv.gz\n", "Fetching data from: https://tickdata.fxcorporate.com/EURUSD/2018/6.csv.gz\n", "Fetching data from: https://tickdata.fxcorporate.com/EURUSD/2018/7.csv.gz\n" ] } ], "source": [ "dr = tdr('EURUSD', start, end, verbosity=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The resulting object is of type `fxcmpy_tick_data_reader` and has the methods `get_data()` and `get_raw_data()`." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "fxcmpy.fxcmpy_data_reader.fxcmpy_tick_data_reader" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(dr)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "raw_data= dr.get_raw_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`get_raw_data()` returns the data in a `DataFrame` object with a string representation of the dates as index. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Index: 5643477 entries, 01/28/2018 22:00:46.425 to 02/16/2018 21:59:08.423\n", "Data columns (total 2 columns):\n", "Bid float64\n", "Ask float64\n", "dtypes: float64(2)\n", "memory usage: 129.2+ MB\n" ] } ], "source": [ "raw_data.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`get_data()` returns the same `DataFrame` object but with a `DatetimeIndex` (note that the conversion is time consuming for large data sets)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "DatetimeIndex: 5643477 entries, 2018-01-28 22:00:46.425000 to 2018-02-16 21:59:08.423000\n", "Data columns (total 2 columns):\n", "Bid float64\n", "Ask float64\n", "dtypes: float64(2)\n", "memory usage: 129.2 MB\n" ] } ], "source": [ "dr.get_data().info()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
BidAsk
2018-01-28 22:00:46.4251.242391.24282
2018-01-28 22:00:46.4381.242361.24282
2018-01-28 22:00:59.1171.242361.24267
2018-01-28 22:00:59.1341.242291.24267
2018-01-28 22:01:25.6681.242281.24266
2018-01-28 22:01:29.1571.242591.24266
2018-01-28 22:01:30.1641.242581.24266
2018-01-28 22:01:33.1371.242431.24266
2018-01-28 22:01:33.1531.242281.24266
2018-01-28 22:02:04.4961.242281.24261
\n", "
" ], "text/plain": [ " Bid Ask\n", "2018-01-28 22:00:46.425 1.24239 1.24282\n", "2018-01-28 22:00:46.438 1.24236 1.24282\n", "2018-01-28 22:00:59.117 1.24236 1.24267\n", "2018-01-28 22:00:59.134 1.24229 1.24267\n", "2018-01-28 22:01:25.668 1.24228 1.24266\n", "2018-01-28 22:01:29.157 1.24259 1.24266\n", "2018-01-28 22:01:30.164 1.24258 1.24266\n", "2018-01-28 22:01:33.137 1.24243 1.24266\n", "2018-01-28 22:01:33.153 1.24228 1.24266\n", "2018-01-28 22:02:04.496 1.24228 1.24261" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dr.get_data().head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The `fxcmpy_candles_data _reader`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `fxcmpy_candles_data_reader` returns candles from the **FXCM** servers for a given symbol. The basic usage is as before." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from fxcmpy import fxcmpy_candles_data_reader as cdr" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "('AUDCAD', 'AUDCHF', 'AUDJPY', 'AUDNZD', 'CADCHF', 'EURAUD', 'EURCHF', 'EURGBP', 'EURJPY', 'EURUSD', 'GBPCHF', 'GBPJPY', 'GBPNZD', 'GBPUSD', 'GBPCHF', 'GBPJPY', 'GBPNZD', 'NZDCAD', 'NZDCHF', 'NZDJPY', 'NZDUSD', 'USDCAD', 'USDCHF', 'USDJPY')\n" ] } ], "source": [ "print(cdr.get_available_symbols())" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "start = dt.datetime(2018, 2, 1)\n", "end = dt.datetime(2018, 3, 1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`period` defines the granularity of the data and must be one of `m1`, `h1` or `D1`." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "period = 'm1'" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fetching data from: https://candledata.fxcorporate.com/m1/EURGBP/2018/5.csv.gz\n", "Fetching data from: https://candledata.fxcorporate.com/m1/EURGBP/2018/6.csv.gz\n", "Fetching data from: https://candledata.fxcorporate.com/m1/EURGBP/2018/7.csv.gz\n", "Fetching data from: https://candledata.fxcorporate.com/m1/EURGBP/2018/8.csv.gz\n", "Fetching data from: https://candledata.fxcorporate.com/m1/EURGBP/2018/9.csv.gz\n" ] } ], "source": [ "candles = cdr('EURGBP', start, end, period, verbosity=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The resulting object is of type `fxcmpy_candles_data_reader` and has, similar to `fxcmpy_tick_data_reader` the methods `get_data()` and `get_raw_data()`." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "fxcmpy.fxcmpy_data_reader.fxcmpy_candles_data_reader" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(candles)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "raw_data = candles.get_raw_data()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Index: 35932 entries, 01/28/2018 22:00:00.000 to 03/02/2018 21:59:00.000\n", "Data columns (total 8 columns):\n", "BidOpen 35932 non-null float64\n", "BidHigh 35932 non-null float64\n", "BidLow 35932 non-null float64\n", "BidClose 35932 non-null float64\n", "AskOpen 35932 non-null float64\n", "AskHigh 35932 non-null float64\n", "AskLow 35932 non-null float64\n", "AskClose 35932 non-null float64\n", "dtypes: float64(8)\n", "memory usage: 2.5+ MB\n" ] } ], "source": [ "raw_data.info()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
BidOpenBidHighBidLowBidCloseAskOpenAskHighAskLowAskClose
DateTime
01/28/2018 22:00:00.0000.878490.878610.878490.878580.879380.879380.879120.87912
01/28/2018 22:01:00.0000.878580.878580.878330.878330.879120.879300.879120.87923
01/28/2018 22:03:00.0000.878330.878450.878330.878420.879230.879300.879190.87929
01/28/2018 22:04:00.0000.878420.878430.878420.878420.879290.879290.879290.87929
01/28/2018 22:05:00.0000.878420.878770.878420.878580.879290.879420.879130.87913
\n", "
" ], "text/plain": [ " BidOpen BidHigh BidLow BidClose AskOpen \\\n", "DateTime \n", "01/28/2018 22:00:00.000 0.87849 0.87861 0.87849 0.87858 0.87938 \n", "01/28/2018 22:01:00.000 0.87858 0.87858 0.87833 0.87833 0.87912 \n", "01/28/2018 22:03:00.000 0.87833 0.87845 0.87833 0.87842 0.87923 \n", "01/28/2018 22:04:00.000 0.87842 0.87843 0.87842 0.87842 0.87929 \n", "01/28/2018 22:05:00.000 0.87842 0.87877 0.87842 0.87858 0.87929 \n", "\n", " AskHigh AskLow AskClose \n", "DateTime \n", "01/28/2018 22:00:00.000 0.87938 0.87912 0.87912 \n", "01/28/2018 22:01:00.000 0.87930 0.87912 0.87923 \n", "01/28/2018 22:03:00.000 0.87930 0.87919 0.87929 \n", "01/28/2018 22:04:00.000 0.87929 0.87929 0.87929 \n", "01/28/2018 22:05:00.000 0.87942 0.87913 0.87913 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "raw_data.head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "data = candles.get_data()" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "DatetimeIndex: 35932 entries, 2018-01-28 22:00:00 to 2018-03-02 21:59:00\n", "Data columns (total 8 columns):\n", "BidOpen 35932 non-null float64\n", "BidHigh 35932 non-null float64\n", "BidLow 35932 non-null float64\n", "BidClose 35932 non-null float64\n", "AskOpen 35932 non-null float64\n", "AskHigh 35932 non-null float64\n", "AskLow 35932 non-null float64\n", "AskClose 35932 non-null float64\n", "dtypes: float64(8)\n", "memory usage: 2.5 MB\n" ] } ], "source": [ "data.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`D1` for period only works for time windows before the current year" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "Candles with period 'D1' are restricted to years before 2018", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0mperiod\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'D1'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcandles\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcdr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'EURGBP'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mperiod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m/notebooks/pyalgo/schwed/www/fxcmpy-docu/fxcmpy/fxcmpy_data_reader.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, symbol, start, end, period, verbosity)\u001b[0m\n\u001b[1;32m 166\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcodec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 167\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'https://candledata.fxcorporate.com/%s/%s/%s/%s.csv.gz'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 168\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__fetch_data__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 169\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 170\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m/notebooks/pyalgo/schwed/www/fxcmpy-docu/fxcmpy/fxcmpy_data_reader.py\u001b[0m in \u001b[0;36m__fetch_data__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 190\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mstop\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mdt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0myear\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 191\u001b[0m \u001b[0mmsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Candles with period 'D1' are restricted to years before %s\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 192\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mdt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0myear\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 193\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0myear\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstop\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 194\u001b[0m \u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'https://candledata.fxcorporate.com/%s/%s/%s.csv.gz'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: Candles with period 'D1' are restricted to years before 2018" ] } ], "source": [ "period = 'D1'\n", "candles = cdr('EURGBP', start, end, period)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, they can be used for years including 2017." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true }, "outputs": [], "source": [ "start = dt.datetime(2017, 7, 1)\n", "end = dt.datetime(2017, 11, 1)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "period = 'D1'\n", "candles = cdr('EURGBP', start, end, period)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "DatetimeIndex: 309 entries, 2017-01-02 22:00:00 to 2017-12-31 22:00:00\n", "Data columns (total 8 columns):\n", "BidOpen 309 non-null float64\n", "BidHigh 309 non-null float64\n", "BidLow 309 non-null float64\n", "BidClose 309 non-null float64\n", "AskOpen 309 non-null float64\n", "AskHigh 309 non-null float64\n", "AskLow 309 non-null float64\n", "AskClose 309 non-null float64\n", "dtypes: float64(8)\n", "memory usage: 21.7 KB\n" ] } ], "source": [ "candles.get_data().info()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }