{"action":"created","issue":{"url":"https://api.github.com/repos/chainer/chainerrl/issues/283","repository_url":"https://api.github.com/repos/chainer/chainerrl","labels_url":"https://api.github.com/repos/chainer/chainerrl/issues/283/labels{/name}","comments_url":"https://api.github.com/repos/chainer/chainerrl/issues/283/comments","events_url":"https://api.github.com/repos/chainer/chainerrl/issues/283/events","html_url":"https://github.com/chainer/chainerrl/issues/283","id":337276411,"node_id":"MDU6SXNzdWUzMzcyNzY0MTE=","number":283,"title":"Doubt","user":{"login":"mirraaj","id":15613912,"node_id":"MDQ6VXNlcjE1NjEzOTEy","avatar_url":"https://avatars0.githubusercontent.com/u/15613912?v=4","gravatar_id":"","url":"https://api.github.com/users/mirraaj","html_url":"https://github.com/mirraaj","followers_url":"https://api.github.com/users/mirraaj/followers","following_url":"https://api.github.com/users/mirraaj/following{/other_user}","gists_url":"https://api.github.com/users/mirraaj/gists{/gist_id}","starred_url":"https://api.github.com/users/mirraaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirraaj/subscriptions","organizations_url":"https://api.github.com/users/mirraaj/orgs","repos_url":"https://api.github.com/users/mirraaj/repos","events_url":"https://api.github.com/users/mirraaj/events{/privacy}","received_events_url":"https://api.github.com/users/mirraaj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-07-01T09:13:25Z","updated_at":"2018-07-02T04:30:42Z","closed_at":null,"author_association":"NONE","body":"Could you please let me know why there is a negative sign? \r\n\r\nhttps://github.com/chainer/chainerrl/blob/e424a75c3c7560fb6d78594e70f52af9668788fb/chainerrl/agents/acer.py#L225"},"comment":{"url":"https://api.github.com/repos/chainer/chainerrl/issues/comments/401669616","html_url":"https://github.com/chainer/chainerrl/issues/283#issuecomment-401669616","issue_url":"https://api.github.com/repos/chainer/chainerrl/issues/283","id":401669616,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTY2OTYxNg==","user":{"login":"mirraaj","id":15613912,"node_id":"MDQ6VXNlcjE1NjEzOTEy","avatar_url":"https://avatars0.githubusercontent.com/u/15613912?v=4","gravatar_id":"","url":"https://api.github.com/users/mirraaj","html_url":"https://github.com/mirraaj","followers_url":"https://api.github.com/users/mirraaj/followers","following_url":"https://api.github.com/users/mirraaj/following{/other_user}","gists_url":"https://api.github.com/users/mirraaj/gists{/gist_id}","starred_url":"https://api.github.com/users/mirraaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirraaj/subscriptions","organizations_url":"https://api.github.com/users/mirraaj/orgs","repos_url":"https://api.github.com/users/mirraaj/repos","events_url":"https://api.github.com/users/mirraaj/events{/privacy}","received_events_url":"https://api.github.com/users/mirraaj/received_events","type":"User","site_admin":false},"created_at":"2018-07-02T04:30:42Z","updated_at":"2018-07-02T04:30:42Z","author_association":"NONE","body":"@muupan. During the policy gradient approach, we try to do θ += lr * ∇_(θ)(π(θ)) * A. In acer the gradient is being subtracted by derivative of trust region optimization (equation 12 (https://arxiv.org/pdf/1611.01224.pdf)). Since we have defined loss function to be a function of π(θ), we do loss = - single_step_loss, to allow SGD to do its work. However, if you look closely, for SGD we need to add the second term instead of subtracting(to make it compatible for SGD). Since you people have already taken care of that in the line 241, you do not need to put the (-kl) term.  https://github.com/chainer/chainerrl/blob/e424a75c3c7560fb6d78594e70f52af9668788fb/chainerrl/agents/acer.py#L241\r\n\r\nThe (+kl) term should be used for computing since we need the gradient of the parameters and that is independent of the optimizer. Let me know how you feel about this."}}
{"actor":{"display_login":"mirraaj"}}