{"action":"created","issue":{"url":"https://api.github.com/repos/awslabs/sockeye/issues/545","repository_url":"https://api.github.com/repos/awslabs/sockeye","labels_url":"https://api.github.com/repos/awslabs/sockeye/issues/545/labels{/name}","comments_url":"https://api.github.com/repos/awslabs/sockeye/issues/545/comments","events_url":"https://api.github.com/repos/awslabs/sockeye/issues/545/events","html_url":"https://github.com/awslabs/sockeye/issues/545","id":363754628,"node_id":"MDU6SXNzdWUzNjM3NTQ2Mjg=","number":545,"title":"Inconsistencies with scoring and inference","user":{"login":"mjpost","id":455256,"node_id":"MDQ6VXNlcjQ1NTI1Ng==","avatar_url":"https://avatars0.githubusercontent.com/u/455256?v=4","gravatar_id":"","url":"https://api.github.com/users/mjpost","html_url":"https://github.com/mjpost","followers_url":"https://api.github.com/users/mjpost/followers","following_url":"https://api.github.com/users/mjpost/following{/other_user}","gists_url":"https://api.github.com/users/mjpost/gists{/gist_id}","starred_url":"https://api.github.com/users/mjpost/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjpost/subscriptions","organizations_url":"https://api.github.com/users/mjpost/orgs","repos_url":"https://api.github.com/users/mjpost/repos","events_url":"https://api.github.com/users/mjpost/events{/privacy}","received_events_url":"https://api.github.com/users/mjpost/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-25T20:32:53Z","updated_at":"2018-09-26T07:14:12Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"There are a few inconsistencies related to inference that have come up in evaluating scoring (#538). However, these are corner cases that could arise in other settings, as well. The problem is:\r\n\r\n- Length normalization is only applied to completed hypotheses on the beam. However, if the beam hits the maximum sequence length before generating </s>, it will still return, with the unnormalized score.\r\n- This is a general problem for scoring: when retrieving a score from the Sockeye inference CLI (via `--output-type translation_with_score`), there is no way to know whether </s> was generated and therefore whether length normalization was applied (since </s> is stripped before returning to the user).\r\n- Sockeye's scores are therefore underspecified.\r\n\r\nThis is a problem for evaluating scoring. Scoring takes raw text and will therefore always append </s>, just as is done in training. I am running into this problem because sometimes the outputs haven't actually finished but have just hit the maximum output length and are unnormalized. This could be a problem more generally.\r\n\r\nI am not sure of the correct solution, but I propose this:\r\n\r\n- In inference, `--maximum-output-length` should refer to the hypothesis *excluding* </s>. The reasoning is that this flag is a user-facing feature, and users do not see the </s> since it is stripped off.\r\n- Beam search stops at the max output length. If there are unfinished hypotheses, however, the decoder should take one more step and force each the selection of </s> for all unfinished hypotheses. It will then apply length normalization, too.\r\n- Length normalization should be computed *including* </s>, since it is generated by the decoder.\r\n\r\nThis way, the user can be guaranteed that every hypothesis actually finished, and that all obtained scores are comparable.\r\n"},"comment":{"url":"https://api.github.com/repos/awslabs/sockeye/issues/comments/424610339","html_url":"https://github.com/awslabs/sockeye/issues/545#issuecomment-424610339","issue_url":"https://api.github.com/repos/awslabs/sockeye/issues/545","id":424610339,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDYxMDMzOQ==","user":{"login":"bricksdont","id":12611348,"node_id":"MDQ6VXNlcjEyNjExMzQ4","avatar_url":"https://avatars2.githubusercontent.com/u/12611348?v=4","gravatar_id":"","url":"https://api.github.com/users/bricksdont","html_url":"https://github.com/bricksdont","followers_url":"https://api.github.com/users/bricksdont/followers","following_url":"https://api.github.com/users/bricksdont/following{/other_user}","gists_url":"https://api.github.com/users/bricksdont/gists{/gist_id}","starred_url":"https://api.github.com/users/bricksdont/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bricksdont/subscriptions","organizations_url":"https://api.github.com/users/bricksdont/orgs","repos_url":"https://api.github.com/users/bricksdont/repos","events_url":"https://api.github.com/users/bricksdont/events{/privacy}","received_events_url":"https://api.github.com/users/bricksdont/received_events","type":"User","site_admin":false},"created_at":"2018-09-26T07:14:12Z","updated_at":"2018-09-26T07:14:12Z","author_association":"CONTRIBUTOR","body":"What was the rationale originally to not apply length normalization for unfinished hypotheses that hit maximum length?"}}
{"actor":{"display_login":"bricksdont"}}