{"action":"created","issue":{"url":"https://api.github.com/repos/chainer/chainerrl/issues/430","repository_url":"https://api.github.com/repos/chainer/chainerrl","labels_url":"https://api.github.com/repos/chainer/chainerrl/issues/430/labels{/name}","comments_url":"https://api.github.com/repos/chainer/chainerrl/issues/430/comments","events_url":"https://api.github.com/repos/chainer/chainerrl/issues/430/events","html_url":"https://github.com/chainer/chainerrl/pull/430","id":428396859,"node_id":"MDExOlB1bGxSZXF1ZXN0MjY2NzczODI1","number":430,"title":"Precompute log probability in PPO","user":{"login":"muupan","id":815970,"node_id":"MDQ6VXNlcjgxNTk3MA==","avatar_url":"https://avatars3.githubusercontent.com/u/815970?v=4","gravatar_id":"","url":"https://api.github.com/users/muupan","html_url":"https://github.com/muupan","followers_url":"https://api.github.com/users/muupan/followers","following_url":"https://api.github.com/users/muupan/following{/other_user}","gists_url":"https://api.github.com/users/muupan/gists{/gist_id}","starred_url":"https://api.github.com/users/muupan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/muupan/subscriptions","organizations_url":"https://api.github.com/users/muupan/orgs","repos_url":"https://api.github.com/users/muupan/repos","events_url":"https://api.github.com/users/muupan/events{/privacy}","received_events_url":"https://api.github.com/users/muupan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-04-02T19:16:08Z","updated_at":"2019-04-02T19:18:40Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/chainer/chainerrl/pulls/430","html_url":"https://github.com/chainer/chainerrl/pull/430","diff_url":"https://github.com/chainer/chainerrl/pull/430.diff","patch_url":"https://github.com/chainer/chainerrl/pull/430.patch"},"body":"Merge #427 before this PR.\r\n\r\nBefore this PR, for every minibatch (e.g. 64 transitions) in a batch (e.g. 2048 transitions) used for an iteration of PPO, both the current and old models are evaluated to compute the probability ratio. This PR instead pre-compute the outputs from the old model and evaluate only the current model for each minibatch. The reasons behind this PR are:\r\n- It is computationally more efficient since it can use a larger batch and only evaluate the old model once per PPO iteration.\r\n- Currently, `obs_normalizer` is updated after collecting a batch of transitions and before updating the policy. To correctly implement importance sampling, you need to use the policy used for collecting transitions, so it is better to compute the probabilities before updating `obs_normalizer` which can affect the policy. https://github.com/chainer/chainerrl/blob/master/chainerrl/agents/ppo.py#L225\r\n- It is what openai/baselines' ppo2 does.\r\n\r\nThis PR also adds `n_updates` to PPO's statistics to help debugging.\r\n\r\nTodos\r\n- [ ] check experimental results on mujoco"},"comment":{"url":"https://api.github.com/repos/chainer/chainerrl/issues/comments/479156660","html_url":"https://github.com/chainer/chainerrl/pull/430#issuecomment-479156660","issue_url":"https://api.github.com/repos/chainer/chainerrl/issues/430","id":479156660,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3OTE1NjY2MA==","user":{"login":"muupan","id":815970,"node_id":"MDQ6VXNlcjgxNTk3MA==","avatar_url":"https://avatars3.githubusercontent.com/u/815970?v=4","gravatar_id":"","url":"https://api.github.com/users/muupan","html_url":"https://github.com/muupan","followers_url":"https://api.github.com/users/muupan/followers","following_url":"https://api.github.com/users/muupan/following{/other_user}","gists_url":"https://api.github.com/users/muupan/gists{/gist_id}","starred_url":"https://api.github.com/users/muupan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/muupan/subscriptions","organizations_url":"https://api.github.com/users/muupan/orgs","repos_url":"https://api.github.com/users/muupan/repos","events_url":"https://api.github.com/users/muupan/events{/privacy}","received_events_url":"https://api.github.com/users/muupan/received_events","type":"User","site_admin":false},"created_at":"2019-04-02T19:18:40Z","updated_at":"2019-04-02T19:18:40Z","author_association":"MEMBER","body":"![image](https://user-images.githubusercontent.com/815970/55429812-4c643800-55c7-11e9-9ba1-9f518692d485.png)\r\n![image](https://user-images.githubusercontent.com/815970/55429830-5423dc80-55c7-11e9-96c7-dfd115b9c487.png)\r\n\r\nSeems like this PR (PPO ParallelLink Precompute) further improves PPO from #427 ."}}
{"actor":{"display_login":"muupan"}}