{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/fairseq/issues/182","repository_url":"https://api.github.com/repos/pytorch/fairseq","labels_url":"https://api.github.com/repos/pytorch/fairseq/issues/182/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/fairseq/issues/182/comments","events_url":"https://api.github.com/repos/pytorch/fairseq/issues/182/events","html_url":"https://github.com/pytorch/fairseq/issues/182","id":333886056,"node_id":"MDU6SXNzdWUzMzM4ODYwNTY=","number":182,"title":"Inconsistent Perplexity Computation","user":{"login":"hmc-cs-mdrissi","id":16809055,"node_id":"MDQ6VXNlcjE2ODA5MDU1","avatar_url":"https://avatars0.githubusercontent.com/u/16809055?v=4","gravatar_id":"","url":"https://api.github.com/users/hmc-cs-mdrissi","html_url":"https://github.com/hmc-cs-mdrissi","followers_url":"https://api.github.com/users/hmc-cs-mdrissi/followers","following_url":"https://api.github.com/users/hmc-cs-mdrissi/following{/other_user}","gists_url":"https://api.github.com/users/hmc-cs-mdrissi/gists{/gist_id}","starred_url":"https://api.github.com/users/hmc-cs-mdrissi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmc-cs-mdrissi/subscriptions","organizations_url":"https://api.github.com/users/hmc-cs-mdrissi/orgs","repos_url":"https://api.github.com/users/hmc-cs-mdrissi/repos","events_url":"https://api.github.com/users/hmc-cs-mdrissi/events{/privacy}","received_events_url":"https://api.github.com/users/hmc-cs-mdrissi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-20T00:41:00Z","updated_at":"2018-06-20T00:54:52Z","closed_at":null,"author_association":"NONE","body":"In the train file, get_perplexity is computed by doing 2**nll. In eval_lm, perplexity is computed with e**nll. Assuming the nll in the two files are computed the same way, then one needs to fixed."},"comment":{"url":"https://api.github.com/repos/pytorch/fairseq/issues/comments/398590311","html_url":"https://github.com/pytorch/fairseq/issues/182#issuecomment-398590311","issue_url":"https://api.github.com/repos/pytorch/fairseq/issues/182","id":398590311,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODU5MDMxMQ==","user":{"login":"alexeib","id":3497946,"node_id":"MDQ6VXNlcjM0OTc5NDY=","avatar_url":"https://avatars3.githubusercontent.com/u/3497946?v=4","gravatar_id":"","url":"https://api.github.com/users/alexeib","html_url":"https://github.com/alexeib","followers_url":"https://api.github.com/users/alexeib/followers","following_url":"https://api.github.com/users/alexeib/following{/other_user}","gists_url":"https://api.github.com/users/alexeib/gists{/gist_id}","starred_url":"https://api.github.com/users/alexeib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexeib/subscriptions","organizations_url":"https://api.github.com/users/alexeib/orgs","repos_url":"https://api.github.com/users/alexeib/repos","events_url":"https://api.github.com/users/alexeib/events{/privacy}","received_events_url":"https://api.github.com/users/alexeib/received_events","type":"User","site_admin":false},"created_at":"2018-06-20T00:54:52Z","updated_at":"2018-06-20T00:54:52Z","author_association":"CONTRIBUTOR","body":"nll loss is not computed the same in the two files\r\nin the case of eval_lm, it is the average of scores which are log likelihoods with base e, in the other case it is base 2 (see for example adaptive_loss.py for example). so you will see that the losses are different between eval_lm and what you see during training, but the perplexities are the same"}}
{"actor":{"display_login":"alexeib"}}