{"action":"created","issue":{"url":"https://api.github.com/repos/freqtrade/freqtrade/issues/1025","repository_url":"https://api.github.com/repos/freqtrade/freqtrade","labels_url":"https://api.github.com/repos/freqtrade/freqtrade/issues/1025/labels{/name}","comments_url":"https://api.github.com/repos/freqtrade/freqtrade/issues/1025/comments","events_url":"https://api.github.com/repos/freqtrade/freqtrade/issues/1025/events","html_url":"https://github.com/freqtrade/freqtrade/issues/1025","id":340995380,"node_id":"MDU6SXNzdWUzNDA5OTUzODA=","number":1025,"title":"backslap - sample(working) code to process backtest inframe. buy sell, stop in and tested - 750x quicker  (0.02sec to process 60 pairs) ","user":{"login":"creslinux","id":34645187,"node_id":"MDQ6VXNlcjM0NjQ1MTg3","avatar_url":"https://avatars0.githubusercontent.com/u/34645187?v=4","gravatar_id":"","url":"https://api.github.com/users/creslinux","html_url":"https://github.com/creslinux","followers_url":"https://api.github.com/users/creslinux/followers","following_url":"https://api.github.com/users/creslinux/following{/other_user}","gists_url":"https://api.github.com/users/creslinux/gists{/gist_id}","starred_url":"https://api.github.com/users/creslinux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/creslinux/subscriptions","organizations_url":"https://api.github.com/users/creslinux/orgs","repos_url":"https://api.github.com/users/creslinux/repos","events_url":"https://api.github.com/users/creslinux/events{/privacy}","received_events_url":"https://api.github.com/users/creslinux/received_events","type":"User","site_admin":false},"labels":[{"id":957613485,"node_id":"MDU6TGFiZWw5NTc2MTM0ODU=","url":"https://api.github.com/repos/freqtrade/freqtrade/labels/backtest","name":"backtest","color":"a9fc6f","default":false},{"id":607535495,"node_id":"MDU6TGFiZWw2MDc1MzU0OTU=","url":"https://api.github.com/repos/freqtrade/freqtrade/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-07-13T12:10:25Z","updated_at":"2018-07-16T18:32:27Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"The below scratch loads 60 dataframes from disk that were harvested from backtest.py.backtest\r\nThe only delta is i modified backtest.py to also include high and low columns. \r\n\r\nThe code is a proof of concept for managing BT in frame \r\nThe driver is processing row by row is hellishly inefficient when we\r\nhave all the data in frame. \r\n\r\nThe performance is 2-3 milliseconds per frame,\r\n to process all 60 5 minute 12 days dataframes 0.015 seconds \r\nThe same frames processed in current BT takes 10seconds\r\n\r\nThe code is incomplete it will process all BUY, SELL, STOP,  \r\nIf has not yet been updated to include ROI and Dynamic Stop or write out a trades/return table\r\nThough these do not pose any challenge\r\n\r\nThe code is also parameterised, so setting stop to be triggered on low and use stop price is setting two variables only.\r\n\r\n\r\n``` python\r\nimport logging\r\n\r\nimport pandas as pd\r\nimport numpy as np\r\nimport utils_find_1st as utf1st\r\nimport timeit\r\n\r\nlogger = logging.getLogger(__name__)\r\n\r\n### Dependencies\r\n# find_1st - numpy C extension to find first in array\r\n# https://pypi.org/project/py_find_1st/\r\n# pip3 install py_find_1st\r\n\r\n\r\n###### Backslap Scratch\r\n# Back slap is a science project to move\r\n# backtest processing \"in frame\" to determine\r\n# pro's and cons\r\n# The spur is currently FT loops in a nested manner\r\n# each pair/row/row and takes 5-10seconds typically a pair\r\n#\r\n# The hope is 1 or 2 orders or magnitude can be gained\r\n# and simpler application of trade profit logic\r\n\r\n##### Test env conf/tools\r\ndebug = True\r\nif debug:\r\n    from pandas import set_option\r\n\r\n    set_option('display.max_rows', 5000)\r\n    set_option('display.max_columns', 8)\r\n    pd.set_option('display.width', 1000)\r\n\r\ndef s():\r\n    st = timeit.default_timer()\r\n    return st\r\ndef f(st):\r\n    return(timeit.default_timer() - st)\r\n\r\n#### Test/Stub environment\r\n\r\n#### End test environment\r\n### Config\r\nstop = 0.01 #stop from config\r\np_stop = (-stop +1) #What stop really means, e.g 0.01 is 0.99 of price\r\n\r\ntrailing_stop = \"xxx\"\r\nbto = buys_triggered_on = \"close\"\r\nsto = stops_triggered_on = \"close\"  ## Should be low, FT uses close\r\nsco = stops_calulated_on = \"close\"  ## Should be stop, FT uses close\r\n\r\n### End Config\r\n\r\n### load all the dumped DFs from directory (60 in all)\r\n\r\nfrom os import listdir\r\nfrom os.path import isfile, join\r\nfiles_in_dir = [ f for f in listdir('/Users/creslin/PycharmProjects/freqtrade_new/frames/')\r\n                 if isfile(join('/Users/creslin/PycharmProjects/freqtrade_new/frames/',f)) ]\r\n\r\nfor file in files_in_dir:\r\n    print(\"data frame is\", file, \"\\n\")\r\n    csv = \"/Users/creslin/PycharmProjects/freqtrade_new/frames/\" + file\r\n    ticker_data = pd.DataFrame.from_csv(csv, sep='\\t', encoding='utf-8')\r\n\r\n    bslap = ticker_data\r\n    bslap_len = len(bslap)\r\n    t_exit_ind = 0 # Start loop from first index\r\n    t_exit_last = 0\r\n    st = s() # Start timer for processing dataframe\r\n    while t_exit_ind < bslap_len:\r\n        '''\r\n        Dev phases\r\n        Phase 1\r\n         1) Manage buy, sell, stop enter/exit\r\n          a) Find first buy index\r\n          b) Discover first stop and sell hit after buy index\r\n          c) Chose first intance as trade exit\r\n\r\n        Phase 2\r\n         2) Manage dynamic Stop and ROI Exit\r\n          a) Create trade slice from 1\r\n          b) search within trade slice for dynamice stop hit\r\n          c) search within trade slice for ROI hit\r\n        '''\r\n\r\n        '''\r\n        Finds index for first buy = 1 flag, use .values numpy array for speed\r\n\r\n        Create a slice, from first buy index onwards.\r\n        Slice will be used to find exit conditions after trade open\r\n        '''\r\n        print(t_exit_ind)\r\n        t_open_slice = bslap.iloc[t_exit_ind:]\r\n        t_open_ind = utf1st.find_1st(t_open_slice['buy'].values, 1, utf1st.cmp_equal)\r\n        t_open_slice = bslap.iloc[t_open_ind:]\r\n\r\n        '''\r\n        Calculate t_stop_pri (Trade Stop Price) based on our buy price\r\n        bto (buys triggered on) Default is close.\r\n        TODO - remove column - only useful for human debug\r\n        '''\r\n        t_stop_pri = (bslap.iloc[t_open_ind][bto] * p_stop)\r\n        bslap['t_stop_pri'] = t_stop_pri\r\n\r\n        '''\r\n        Find first stop index after Trade Open:\r\n\r\n        First index in t_open_slice\r\n        where [sto] (stop tiggered on variable: \"close\", \"low\" etc) < t_stop_pri\r\n        '''\r\n        t_stop_ind = utf1st.find_1st(t_open_slice[sto].values,\r\n                                     t_stop_pri,\r\n                                     utf1st.cmp_smaller) \\\r\n                     + t_open_ind + 1\r\n\r\n        '''\r\n        Find first sell index after trade open\r\n\r\n        First index in t_open_slice where ['sell'] = 1\r\n        '''\r\n        t_sell_ind = utf1st.find_1st(t_open_slice['sell'].values,\r\n                                     1 , utf1st.cmp_equal)\\\r\n                     + t_open_ind + 1\r\n\r\n\r\n        '''\r\n        Determine which was hit first stop or sell, use as exit\r\n\r\n        STOP takes priority over SELL as would be 'in candle' from tick data\r\n        Sell would use Open from Next candle.\r\n        So in a draw Stop would be hit first on ticker data in live\r\n        '''\r\n        if t_stop_ind <= t_sell_ind:\r\n            t_exit_ind = t_stop_ind\r\n            t_exit_type = 'stop'\r\n            t_exit_pri = sco #Stops Calculated on variable, FT uses close, should use Stop\r\n        else:\r\n            # move sell onto next candle, we only look back on sell\r\n            # will use the open price later.\r\n            t_exit_ind = t_sell_ind +1\r\n            t_exit_type = 'sell'\r\n            t_exit_pri  = 'open' # open looking back.\r\n\r\n\r\n        if debug:\r\n            '''\r\n            Print out the buys, stops, sells\r\n            Include Line before and after to for easy\r\n            Human verification\r\n            '''\r\n            # Buy\r\n            print(bslap.iloc[t_open_ind], \"\\n\")\r\n            print(\"Open Trade Index\", t_open_ind + 1 )\r\n            op_is = t_open_ind -1 # Print open index start, line before\r\n            op_if = t_open_ind +2 # Print open index finish, line after\r\n            print(bslap.iloc[op_is:op_if], \"\\n\")\r\n\r\n            # Stop\r\n            print(bslap.iloc[t_stop_ind], \"\\n\")\r\n            print(\"First Stop Index after Trade open\", t_stop_ind )\r\n            st_is = t_stop_ind -2 # Print stop index start, line before\r\n            st_if = t_stop_ind +1 # Print stop index finish, line after\r\n            print(bslap.iloc[st_is:st_if], \"\\n\")\r\n\r\n            # Sell\r\n            print(bslap.iloc[t_sell_ind], \"\\n\")\r\n            print(\"First Sell Index after Trade open\", t_sell_ind )\r\n            print(\"HINT, if sell exit(not stop) trade should use open from\", t_sell_ind +1, \"\\n\" )\r\n            sl_is = t_sell_ind - 2  # Print sell index start, line before\r\n            sl_if = t_sell_ind + 1  # Print sell index finish, line after\r\n            print(bslap.iloc[sl_is:sl_if], \"\\n\")\r\n\r\n            #Chosen Exit (stop or sell)\r\n            print(\"Exit type is :\", t_exit_type)\r\n            print((bslap.iloc[t_exit_ind], \"\\n\"))\r\n            print(\"trade exit price field is\", t_exit_pri, \"\\n\" )\r\n\r\n        '''\r\n        Trade entry is always the next candles \"open\" price\r\n        We trigger on close, so cannot see that till after\r\n        its closed.\r\n        '''\r\n        trade_enter_price = bslap.iloc[t_open_ind +1]['open']\r\n        trade_exit_price = bslap.iloc[t_exit_ind][t_exit_pri]\r\n\r\n        print (\"trade enter\", trade_enter_price)\r\n        print (\"trade exit\", trade_exit_price)\r\n\r\n        # Loop control - catch no closed trades.\r\n        print(\"exit cond\", t_exit_last, t_exit_ind )\r\n        if t_exit_last >= t_exit_ind:\r\n            \"\"\"\r\n            TODO\r\n            add handing here to record none closed open trades\r\n            \"\"\"\r\n            df_proc_time = f(st)\r\n            print(\"Processing DF took\", df_proc_time, \"\\n\" )\r\n            print('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++')\r\n            print('+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++')\r\n            f(st)\r\n            break\r\n        else:\r\n            t_exit_last = t_exit_ind\r\n\r\n\r\n```\r\n\r\n\r\n\r\nSample Dataframe being loaded: \r\n\r\n```\r\nhead ETH_BTC \r\n\tdate\tbuy\topen\tclose\tsell\thigh\tlow\r\n1\t2018-05-29 18:20:00+00:00\t0.0\t0.075762\t0.075844\t0.0\t0.075937\t0.075762\r\n2\t2018-05-29 18:25:00+00:00\t0.0\t0.075844\t0.07584\t0.0\t0.075964\t0.075747\r\n3\t2018-05-29 18:30:00+00:00\t0.0\t0.07584\t0.07583\t0.0\t0.075856\t0.075748\r\n4\t2018-05-29 18:35:00+00:00\t0.0\t0.075788\t0.076006\t0.0\t0.07602\t0.075779\r\n5\t2018-05-29 18:40:00+00:00\t0.0\t0.076006\t0.076001\t0.0\t0.076146\t0.075967\r\n6\t2018-05-29 18:45:00+00:00\t0.0\t0.076001\t0.076211\t0.0\t0.076211\t0.075886\r\n7\t2018-05-29 18:50:00+00:00\t0.0\t0.076213\t0.0758\t0.0\t0.076299\t0.075761\r\n8\t2018-05-29 18:55:00+00:00\t0.0\t0.075747\t0.075705\t0.0\t0.075847\t0.0757\r\n9\t2018-05-29 19:00:00+00:00\t0.0\t0.075705\t0.075947\t0.0\t0.075998\t0.075705\r\n\r\n```\r\n\r\nAnd my dir of test frames\r\n\r\n> (.env) DannyMBP:frames creslin$ ls\r\n> AMB_BTC\t\tBNB_BTC\t\tDNT_BTC\t\tEVX_BTC\t\tKMD_BTC\t\tMDA_BTC\t\tOAX_BTC\t\tREQ_BTC\t\tSTRAT_BTC\tXMR_BTC\r\n> ARK_BTC\t\tBNT_BTC\t\tENG_BTC\t\tFUN_BTC\t\tKNC_BTC\t\tMOD_BTC\t\tOMG_BTC\t\tSALT_BTC\tSUB_BTC\t\tXRP_BTC\r\n> AST_BTC\t\tBQX_BTC\t\tENJ_BTC\t\tGAS_BTC\t\tLINK_BTC\tMTH_BTC\t\tPOWR_BTC\tSNGLS_BTC\tTRX_BTC\t\tXVG_BTC\r\n> BAT_BTC\t\tBTG_BTC\t\tEOS_BTC\t\tHSR_BTC\t\tLRC_BTC\t\tMTL_BTC\t\tQTUM_BTC\tSNM_BTC\t\tVEN_BTC\t\tYOYOW_BTC\r\n> BCH_BTC\t\tDASH_BTC\tETC_BTC\t\tICN_BTC\t\tLTC_BTC\t\tNEO_BTC\t\tRCN_BTC\t\tSNT_BTC\t\tVIB_BTC\t\tZEC_BTC\r\n> BCPT_BTC\tDLT_BTC\t\tETH_BTC\t\tIOTA_BTC\tMCO_BTC\t\tNULS_BTC\tRDN_BTC\t\tSTORJ_BTC\tWTC_BTC\t\tZRX_BTC"},"comment":{"url":"https://api.github.com/repos/freqtrade/freqtrade/issues/comments/405337906","html_url":"https://github.com/freqtrade/freqtrade/issues/1025#issuecomment-405337906","issue_url":"https://api.github.com/repos/freqtrade/freqtrade/issues/1025","id":405337906,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTMzNzkwNg==","user":{"login":"jblestang","id":6736541,"node_id":"MDQ6VXNlcjY3MzY1NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/6736541?v=4","gravatar_id":"","url":"https://api.github.com/users/jblestang","html_url":"https://github.com/jblestang","followers_url":"https://api.github.com/users/jblestang/followers","following_url":"https://api.github.com/users/jblestang/following{/other_user}","gists_url":"https://api.github.com/users/jblestang/gists{/gist_id}","starred_url":"https://api.github.com/users/jblestang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jblestang/subscriptions","organizations_url":"https://api.github.com/users/jblestang/orgs","repos_url":"https://api.github.com/users/jblestang/repos","events_url":"https://api.github.com/users/jblestang/events{/privacy}","received_events_url":"https://api.github.com/users/jblestang/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T18:32:27Z","updated_at":"2018-07-16T18:32:27Z","author_association":"CONTRIBUTOR","body":"With the removal of the partial candle, we shall use open price for both buy & sell events. That is the closest we can get to the reality. Using the low of the candle could be a way to simulate the worst case scenario."}}
{"actor":{"display_login":"jblestang"}}