{"action":"created","issue":{"url":"https://api.github.com/repos/microsoft/unilm/issues/5","repository_url":"https://api.github.com/repos/microsoft/unilm","labels_url":"https://api.github.com/repos/microsoft/unilm/issues/5/labels{/name}","comments_url":"https://api.github.com/repos/microsoft/unilm/issues/5/comments","events_url":"https://api.github.com/repos/microsoft/unilm/issues/5/events","html_url":"https://github.com/microsoft/unilm/issues/5","id":506421400,"node_id":"MDU6SXNzdWU1MDY0MjE0MDA=","number":5,"title":"❓ Question : Training - Evaluating discrepancy in Abstractive Summarization","user":{"login":"Colanim","id":43774355,"node_id":"MDQ6VXNlcjQzNzc0MzU1","avatar_url":"https://avatars2.githubusercontent.com/u/43774355?v=4","gravatar_id":"","url":"https://api.github.com/users/Colanim","html_url":"https://github.com/Colanim","followers_url":"https://api.github.com/users/Colanim/followers","following_url":"https://api.github.com/users/Colanim/following{/other_user}","gists_url":"https://api.github.com/users/Colanim/gists{/gist_id}","starred_url":"https://api.github.com/users/Colanim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Colanim/subscriptions","organizations_url":"https://api.github.com/users/Colanim/orgs","repos_url":"https://api.github.com/users/Colanim/repos","events_url":"https://api.github.com/users/Colanim/events{/privacy}","received_events_url":"https://api.github.com/users/Colanim/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2019-10-14T02:32:19Z","updated_at":"2019-10-14T04:23:34Z","closed_at":"2019-10-14T03:13:31Z","author_association":"NONE","body":"Thanks for open-sourcing the code !\r\n\r\nAfter reading your paper, I have a question about the finetuning procedure for Abstractive summarization (and more generally any Seq2Seq task).\r\n\r\n---\r\n\r\n![image](https://user-images.githubusercontent.com/43774355/66726279-566acb80-ee73-11e9-876a-423afc79a860.png)\r\n\r\nI understand this idea : Similarly to Bert and to UniLM **pretraining**, finetuning on Abstractive Summarization is masking some token and predicting it in order to learn a bidirectional representation of tokens.\r\n\r\nBut at inference time, since we don't have access to the whole summary (it is yet to be generated), we can only apply a left-to-right LM.\r\n\r\nIt seems a pretty big discrepancy between training and testing.\r\n\r\n---\r\n\r\nWhat I don't understand is that people already tried to use BERT (trained as a bidirectional encoder) as a left-to-right LM. But results were really low.\r\n\r\nAnd in your case, results are very high !\r\n\r\nSo my questions are :\r\n\r\n* Did I miss something ? Did I misunderstood and there is in fact no discrepancy ?\r\n\r\n* If I understood right, **why do you finetune Seq2Seq model using bidirectional LM, and not left-to-right LM** ?\r\n"},"comment":{"url":"https://api.github.com/repos/microsoft/unilm/issues/comments/541498871","html_url":"https://github.com/microsoft/unilm/issues/5#issuecomment-541498871","issue_url":"https://api.github.com/repos/microsoft/unilm/issues/5","id":541498871,"node_id":"MDEyOklzc3VlQ29tbWVudDU0MTQ5ODg3MQ==","user":{"login":"Colanim","id":43774355,"node_id":"MDQ6VXNlcjQzNzc0MzU1","avatar_url":"https://avatars2.githubusercontent.com/u/43774355?v=4","gravatar_id":"","url":"https://api.github.com/users/Colanim","html_url":"https://github.com/Colanim","followers_url":"https://api.github.com/users/Colanim/followers","following_url":"https://api.github.com/users/Colanim/following{/other_user}","gists_url":"https://api.github.com/users/Colanim/gists{/gist_id}","starred_url":"https://api.github.com/users/Colanim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Colanim/subscriptions","organizations_url":"https://api.github.com/users/Colanim/orgs","repos_url":"https://api.github.com/users/Colanim/repos","events_url":"https://api.github.com/users/Colanim/events{/privacy}","received_events_url":"https://api.github.com/users/Colanim/received_events","type":"User","site_admin":false},"created_at":"2019-10-14T04:23:34Z","updated_at":"2019-10-14T04:23:34Z","author_association":"NONE","body":"Ok so in the actual code there is only one loss, which is the generation loss for the decoder (so, **left-to-right LM** based on the summary).\r\n\r\nThank you very much for your answers ! "}}
{"actor":{"display_login":"Colanim"}}