{"action":"created","issue":{"url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/30","repository_url":"https://api.github.com/repos/dennybritz/reinforcement-learning","labels_url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/30/comments","events_url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/30/events","html_url":"https://github.com/dennybritz/reinforcement-learning/issues/30","id":189459915,"number":30,"title":"DQN solution results peak at ~35 reward","user":{"login":"nerdoid","id":573430,"avatar_url":"https://avatars.githubusercontent.com/u/573430?v=3","gravatar_id":"","url":"https://api.github.com/users/nerdoid","html_url":"https://github.com/nerdoid","followers_url":"https://api.github.com/users/nerdoid/followers","following_url":"https://api.github.com/users/nerdoid/following{/other_user}","gists_url":"https://api.github.com/users/nerdoid/gists{/gist_id}","starred_url":"https://api.github.com/users/nerdoid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nerdoid/subscriptions","organizations_url":"https://api.github.com/users/nerdoid/orgs","repos_url":"https://api.github.com/users/nerdoid/repos","events_url":"https://api.github.com/users/nerdoid/events{/privacy}","received_events_url":"https://api.github.com/users/nerdoid/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-11-15T17:46:41Z","updated_at":"2016-11-15T23:28:09Z","closed_at":null,"body":"Hi Denny,\r\n\r\nThanks for this wonderful resource. It's been hugely helpful. Can you say what your results are when training the DQN solution? I've been unable to reproduce the results of the DeepMind paper. I'm using your DQN solution below, although I did try mine first :)\r\n\r\nWhile training, the TensorBoard \"episode_reward\" graph peaks at an average reward of ~35, and then tapers off. When I run the final checkpoint with a fully greedy policy (no epsilon), I get similar rewards.\r\n\r\nThe DeepMind paper cites rewards around 400.\r\n\r\nI have tried 1) implementing the error clipping from the paper, 2) changing the RMSProp arguments to (I think) reflect those from the paper, 3) changing the \"replay_memory_size\" and \"epsilon_decay_steps\" to the paper's 1,000,000, 4) enabling all six agent actions from the Open AI Gym. Still average reward peaks at 35.\r\n\r\nAny pointers would be greatly appreciated.\r\n\r\nScreenshot is of my still-running training session at episode 4878 with all above modifications to dqn.py:\r\n\r\n<img width=\"904\" alt=\"screen shot 2016-11-15 at 11 35 37 am\" src=\"https://cloud.githubusercontent.com/assets/573430/20316522/ce81f4c8-ab27-11e6-825d-720e0562ff45.png\">\r\n"},"comment":{"url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/comments/260803990","html_url":"https://github.com/dennybritz/reinforcement-learning/issues/30#issuecomment-260803990","issue_url":"https://api.github.com/repos/dennybritz/reinforcement-learning/issues/30","id":260803990,"user":{"login":"nerdoid","id":573430,"avatar_url":"https://avatars.githubusercontent.com/u/573430?v=3","gravatar_id":"","url":"https://api.github.com/users/nerdoid","html_url":"https://github.com/nerdoid","followers_url":"https://api.github.com/users/nerdoid/followers","following_url":"https://api.github.com/users/nerdoid/following{/other_user}","gists_url":"https://api.github.com/users/nerdoid/gists{/gist_id}","starred_url":"https://api.github.com/users/nerdoid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nerdoid/subscriptions","organizations_url":"https://api.github.com/users/nerdoid/orgs","repos_url":"https://api.github.com/users/nerdoid/repos","events_url":"https://api.github.com/users/nerdoid/events{/privacy}","received_events_url":"https://api.github.com/users/nerdoid/received_events","type":"User","site_admin":false},"created_at":"2016-11-15T23:28:09Z","updated_at":"2016-11-15T23:28:09Z","body":"Nice catch. That does seem like it would limit the network's expressiveness. Unless you already have something running, I'll go ahead and fire up an instance and give it a shot with ReLU on the hidden fully connected layer. Can also add biases to both fully connected layers (probably initialized to zero?)."}}
{"actor":{"display_login":"nerdoid"}}