{"action":"created","issue":{"url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/15","repository_url":"https://api.github.com/repos/miyosuda/async_deep_reinforce","labels_url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/15/comments","events_url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/15/events","html_url":"https://github.com/miyosuda/async_deep_reinforce/issues/15","id":178089655,"number":15,"title":"Failing to fully replicate Pong with A3C-LSTM","user":{"login":"revilokeb","id":10588773,"avatar_url":"https://avatars.githubusercontent.com/u/10588773?v=3","gravatar_id":"","url":"https://api.github.com/users/revilokeb","html_url":"https://github.com/revilokeb","followers_url":"https://api.github.com/users/revilokeb/followers","following_url":"https://api.github.com/users/revilokeb/following{/other_user}","gists_url":"https://api.github.com/users/revilokeb/gists{/gist_id}","starred_url":"https://api.github.com/users/revilokeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/revilokeb/subscriptions","organizations_url":"https://api.github.com/users/revilokeb/orgs","repos_url":"https://api.github.com/users/revilokeb/repos","events_url":"https://api.github.com/users/revilokeb/events{/privacy}","received_events_url":"https://api.github.com/users/revilokeb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-09-20T15:14:32Z","updated_at":"2016-09-30T17:42:22Z","closed_at":null,"body":"@miyosuda I have been trying to replicate your very nice Pong A3C-LSTM chart (https://github.com/miyosuda/async_deep_reinforce/blob/master/docs/graph_24h_lstm.png). So far unfortunately I have not really succeeded. \r\n\r\nI have been using parameter settings in constants.py (setting USE_LSTM=True and USE_GPU=True). I have also been setting frame_skip=4 in ale.cfg (using master branch, tf r0.10, i7-5930K@3.5GHz - 8 threads, Nvidia Titan X).\r\n\r\n**My question**: should the above setting of parameters be sufficient to reproduce your A3C-LSTM chart?\r\n\r\nWhen doing the above my charts are looking as follows (I have done multiple runs, also simulating for more than 60M steps):\r\n![160920-a3c_lstm_pong_miyosuda_master5](https://cloud.githubusercontent.com/assets/10588773/18675350/61a8b068-7f52-11e6-99b2-be0c86440697.png)\r\nIn my case learning seems to be much slower and saturating at around -5 to 0 (better seen on my simulations running for more than 60M steps).\r\n\r\nComparing with http://arxiv.org/abs/1602.01783 their Figure 3 (Pong, A3C - 8 threads) it seems your result is faster in terms of number of steps to reach score 20: you require ~20-25M steps to reach score 20, whereas DeepMind on average seemd to need ~50-60M (but then theirs is an average value and from their paper I dont know if it has been A3C-FF or A3C-LSTM). \r\n\r\n**My second question**: Have you run A3C-FF / A3C-LSTM multiple times and were results similar? And do you have an explanation why A3C-FF is not reaching score 20? (I have also run A3C-FF which is looking similar to yours...)\r\n\r\nMany many thanks for the code!!"},"comment":{"url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/comments/250807068","html_url":"https://github.com/miyosuda/async_deep_reinforce/issues/15#issuecomment-250807068","issue_url":"https://api.github.com/repos/miyosuda/async_deep_reinforce/issues/15","id":250807068,"user":{"login":"joabim","id":6624587,"avatar_url":"https://avatars.githubusercontent.com/u/6624587?v=3","gravatar_id":"","url":"https://api.github.com/users/joabim","html_url":"https://github.com/joabim","followers_url":"https://api.github.com/users/joabim/followers","following_url":"https://api.github.com/users/joabim/following{/other_user}","gists_url":"https://api.github.com/users/joabim/gists{/gist_id}","starred_url":"https://api.github.com/users/joabim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joabim/subscriptions","organizations_url":"https://api.github.com/users/joabim/orgs","repos_url":"https://api.github.com/users/joabim/repos","events_url":"https://api.github.com/users/joabim/events{/privacy}","received_events_url":"https://api.github.com/users/joabim/received_events","type":"User","site_admin":false},"created_at":"2016-09-30T17:42:22Z","updated_at":"2016-09-30T17:42:22Z","body":"@danijar The reset function game_state.py handles the no-ops (max number defined in constants.py) at each reset"}}
{"actor":{"display_login":"joabim"}}