{"action":"created","issue":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/348","repository_url":"https://api.github.com/repos/ModernMT/MMT","labels_url":"https://api.github.com/repos/ModernMT/MMT/issues/348/labels{/name}","comments_url":"https://api.github.com/repos/ModernMT/MMT/issues/348/comments","events_url":"https://api.github.com/repos/ModernMT/MMT/issues/348/events","html_url":"https://github.com/ModernMT/MMT/issues/348","id":296806271,"node_id":"MDU6SXNzdWUyOTY4MDYyNzE=","number":348,"title":"Idea : add extracted n-gram pairs to the neural training","user":{"login":"EtienneAb3d","id":25932245,"node_id":"MDQ6VXNlcjI1OTMyMjQ1","avatar_url":"https://avatars0.githubusercontent.com/u/25932245?v=4","gravatar_id":"","url":"https://api.github.com/users/EtienneAb3d","html_url":"https://github.com/EtienneAb3d","followers_url":"https://api.github.com/users/EtienneAb3d/followers","following_url":"https://api.github.com/users/EtienneAb3d/following{/other_user}","gists_url":"https://api.github.com/users/EtienneAb3d/gists{/gist_id}","starred_url":"https://api.github.com/users/EtienneAb3d/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EtienneAb3d/subscriptions","organizations_url":"https://api.github.com/users/EtienneAb3d/orgs","repos_url":"https://api.github.com/users/EtienneAb3d/repos","events_url":"https://api.github.com/users/EtienneAb3d/events{/privacy}","received_events_url":"https://api.github.com/users/EtienneAb3d/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2018-02-13T16:42:21Z","updated_at":"2018-06-04T07:15:31Z","closed_at":"2018-05-31T09:21:39Z","author_association":"NONE","body":"When looking at the neural translation log, it appears to me that, very often, for the live re-training, MMT is using large (or very large) sentence, where only a very small part is interesting, giving it a very low score. This seems to be often of poor effect, since the results rarely take it into account.\r\n\r\nI had the idea to extract chunks pairs from the training sentences, and add them to the training data. For 16M pairs of sentences (mainly MultiUN, Europarl, ..) I got 60M pairs of chunks (of course with many redundancies, and a lot of noise/errors). \r\n\r\n1) the model should learn directly how to translate these small part of text, all given alone, rather than having to find it by itself from very large sentences, without explicit information on the sub-alignments.\r\n\r\n2) knowing about how sub-parts should be translated, it could be easier for it to learn how to translate large sentences.\r\n\r\n3) perhaps this could bring with a better learning on the attention module. This, alone, can be a good point for the training and the final quality of the model.\r\n\r\n4) the noise/errors on chunks can even be useful : a way to tell the network that \"A\" should be translated in \"B\" whatever its context (noisy/damaged surroundings). I think, it's certainly nearly what is doing the drop-out parameter. It could play a regularisation role.\r\n\r\n5) for the live retraining, MMT will be able to select pertinent small pieces of texts, rather than large rich sentences. This could be much more efficient at the translation time.\r\n\r\nProblem 1: see #347\r\n\r\nProblem 2: as the chunks are quite noisy, it would be very useful that the live retraining use much more than only one sentence. How this can be set ?\r\n\r\nAs far as I understand, MMT is extracting sub-alignments, right ? Would it be possible to automatically add these sub-alignments to the neural training data, on the fly during the training ?"},"comment":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/comments/394256786","html_url":"https://github.com/ModernMT/MMT/issues/348#issuecomment-394256786","issue_url":"https://api.github.com/repos/ModernMT/MMT/issues/348","id":394256786,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDI1Njc4Ng==","user":{"login":"EtienneAb3d","id":25932245,"node_id":"MDQ6VXNlcjI1OTMyMjQ1","avatar_url":"https://avatars0.githubusercontent.com/u/25932245?v=4","gravatar_id":"","url":"https://api.github.com/users/EtienneAb3d","html_url":"https://github.com/EtienneAb3d","followers_url":"https://api.github.com/users/EtienneAb3d/followers","following_url":"https://api.github.com/users/EtienneAb3d/following{/other_user}","gists_url":"https://api.github.com/users/EtienneAb3d/gists{/gist_id}","starred_url":"https://api.github.com/users/EtienneAb3d/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EtienneAb3d/subscriptions","organizations_url":"https://api.github.com/users/EtienneAb3d/orgs","repos_url":"https://api.github.com/users/EtienneAb3d/repos","events_url":"https://api.github.com/users/EtienneAb3d/events{/privacy}","received_events_url":"https://api.github.com/users/EtienneAb3d/received_events","type":"User","site_admin":false},"created_at":"2018-06-04T07:15:31Z","updated_at":"2018-06-04T07:15:31Z","author_association":"NONE","body":"Why closing it !? It was an evolution suggestion, and an opened discussion. The only reason I see to close it, is to definitively show that you aren't interested. Since it's not the first time, I finally doubt you are interested in any suggestion that wasn't in your own road-map. Perhaps I should make my own work on my own side without loosing the time to share it with you."}}
{"actor":{"display_login":"EtienneAb3d"}}