{"action":"created","issue":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/362","repository_url":"https://api.github.com/repos/ModernMT/MMT","labels_url":"https://api.github.com/repos/ModernMT/MMT/issues/362/labels{/name}","comments_url":"https://api.github.com/repos/ModernMT/MMT/issues/362/comments","events_url":"https://api.github.com/repos/ModernMT/MMT/issues/362/events","html_url":"https://github.com/ModernMT/MMT/issues/362","id":313423101,"number":362,"title":"Protecting Substrings During Translation","user":{"login":"robertBrnnn","id":8713464,"avatar_url":"https://avatars1.githubusercontent.com/u/8713464?v=4","gravatar_id":"","url":"https://api.github.com/users/robertBrnnn","html_url":"https://github.com/robertBrnnn","followers_url":"https://api.github.com/users/robertBrnnn/followers","following_url":"https://api.github.com/users/robertBrnnn/following{/other_user}","gists_url":"https://api.github.com/users/robertBrnnn/gists{/gist_id}","starred_url":"https://api.github.com/users/robertBrnnn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robertBrnnn/subscriptions","organizations_url":"https://api.github.com/users/robertBrnnn/orgs","repos_url":"https://api.github.com/users/robertBrnnn/repos","events_url":"https://api.github.com/users/robertBrnnn/events{/privacy}","received_events_url":"https://api.github.com/users/robertBrnnn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-11T17:43:55Z","updated_at":"2018-04-17T11:10:23Z","closed_at":null,"author_association":"NONE","body":"Hi MMT team,\r\n\r\nIn issue https://github.com/ModernMT/MMT/issues/286 regarding protecting substrings that are part of the sentence i.e. using `<dnt>` tags alone won't work, the solution suggested is to use a single placeholder, with a tag.\r\n\r\nI've trained an en-es engine using content with a placeholder for protected strings say XXXX. When translating the string: \r\n`A window that displays metrics for the entire XXXX<dnt 3 /> is available to XXXX<dnt 2 /> administrators with the XXXX<dnt 1 /> permission.`\r\nThe translated text is:\r\n`Una ventana que muestra métricas para todo el XXXX<dnt 3 /> está disponible para<dnt 2 /> los administradores XXXX con<dnt 1 /> el permiso XXXX.`\r\n\r\nAs you can see this method doesn't work, and performs even worse when the source string contains other XML elements. \r\n\r\nThe solution I have been working involves a fixed set of placeholders to represent a piece of non translatable text i.e. XXXA, XXXB, XXXC etc.\r\nSo the source string:\r\n`A window that displays metrics for the entire XXXXA is available to XXXXB administrators with the XXXXC permission.`\r\ngets correctly translated to:\r\n`Una ventana que muestra métricas para toda XXXXA está disponible para los administradores XXXXB con el permiso XXXXC.`\r\n\r\nThis method however, doesn't work with en-zh or en-ja as the placeholders get translated, tokenized, or their case is changed i.e \r\n`XXXXA` becomes `XXXxa` or `XXX Xa` or `Xxxxa` and sometimes `XXXXC`  etc. \r\n\r\nThis make it's quite difficult/impossible to replace the placeholders. I should add however, the translated sentence is linguistically correct, the only problem is the modification of the placeholders.\r\n\r\nFor reference I tested this same approach against a MMT 2.3 engine, which was trained without these placeholders, and it handled them correctly, there was a negligible amount of placeholders that were modified. \r\n\r\nDo you have any ideas why the issue is worse with MMT 2.4? \r\n\r\nThanks,\r\nR."},"comment":{"url":"https://api.github.com/repos/ModernMT/MMT/issues/comments/381951677","html_url":"https://github.com/ModernMT/MMT/issues/362#issuecomment-381951677","issue_url":"https://api.github.com/repos/ModernMT/MMT/issues/362","id":381951677,"user":{"login":"davidecaroselli","id":1674891,"avatar_url":"https://avatars0.githubusercontent.com/u/1674891?v=4","gravatar_id":"","url":"https://api.github.com/users/davidecaroselli","html_url":"https://github.com/davidecaroselli","followers_url":"https://api.github.com/users/davidecaroselli/followers","following_url":"https://api.github.com/users/davidecaroselli/following{/other_user}","gists_url":"https://api.github.com/users/davidecaroselli/gists{/gist_id}","starred_url":"https://api.github.com/users/davidecaroselli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidecaroselli/subscriptions","organizations_url":"https://api.github.com/users/davidecaroselli/orgs","repos_url":"https://api.github.com/users/davidecaroselli/repos","events_url":"https://api.github.com/users/davidecaroselli/events{/privacy}","received_events_url":"https://api.github.com/users/davidecaroselli/received_events","type":"User","site_admin":false},"created_at":"2018-04-17T11:10:23Z","updated_at":"2018-04-17T11:10:23Z","author_association":"OWNER","body":"Hi @robertBrnnn \r\n\r\nlet's make a test. Engines EN <> ES are backward compatible from 2.4 and 2.3. Can you please run the following test: translate the **exact source** file with the **exact same** engine (don't perform any re-training of any sort, use exactly the same engine folder not only the decoder).\r\n\r\nDo the 2 files differ? If yes, can you please provide me the output of the two system on a single line with parameter `verbose=1`? I would like to inspect if the alignment information is the same."}}
{"actor":{"display_login":"davidecaroselli"}}