{"action":"created","issue":{"url":"https://api.github.com/repos/translate/pootle/issues/3971","labels_url":"https://api.github.com/repos/translate/pootle/issues/3971/labels{/name}","comments_url":"https://api.github.com/repos/translate/pootle/issues/3971/comments","events_url":"https://api.github.com/repos/translate/pootle/issues/3971/events","html_url":"https://github.com/translate/pootle/issues/3971","id":100721493,"number":3971,"title":"Local TM results are a large edit distance from the source text","user":{"login":"dwaynebailey","id":647438,"avatar_url":"https://avatars.githubusercontent.com/u/647438?v=3","gravatar_id":"","url":"https://api.github.com/users/dwaynebailey","html_url":"https://github.com/dwaynebailey","followers_url":"https://api.github.com/users/dwaynebailey/followers","following_url":"https://api.github.com/users/dwaynebailey/following{/other_user}","gists_url":"https://api.github.com/users/dwaynebailey/gists{/gist_id}","starred_url":"https://api.github.com/users/dwaynebailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwaynebailey/subscriptions","organizations_url":"https://api.github.com/users/dwaynebailey/orgs","repos_url":"https://api.github.com/users/dwaynebailey/repos","events_url":"https://api.github.com/users/dwaynebailey/events{/privacy}","received_events_url":"https://api.github.com/users/dwaynebailey/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/translate/pootle/labels/translation-memory","name":"translation-memory","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/translate/pootle/milestones/21","html_url":"https://github.com/translate/pootle/milestones/2.8.0","labels_url":"https://api.github.com/repos/translate/pootle/milestones/21/labels","id":1321676,"number":21,"title":"2.8.0","description":"","creator":{"login":"dwaynebailey","id":647438,"avatar_url":"https://avatars.githubusercontent.com/u/647438?v=3","gravatar_id":"","url":"https://api.github.com/users/dwaynebailey","html_url":"https://github.com/dwaynebailey","followers_url":"https://api.github.com/users/dwaynebailey/followers","following_url":"https://api.github.com/users/dwaynebailey/following{/other_user}","gists_url":"https://api.github.com/users/dwaynebailey/gists{/gist_id}","starred_url":"https://api.github.com/users/dwaynebailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwaynebailey/subscriptions","organizations_url":"https://api.github.com/users/dwaynebailey/orgs","repos_url":"https://api.github.com/users/dwaynebailey/repos","events_url":"https://api.github.com/users/dwaynebailey/events{/privacy}","received_events_url":"https://api.github.com/users/dwaynebailey/received_events","type":"User","site_admin":false},"open_issues":45,"closed_issues":0,"state":"open","created_at":"2015-09-24T08:33:48Z","updated_at":"2015-12-09T13:47:52Z","due_on":"2016-03-31T23:00:00Z","closed_at":null},"comments":1,"created_at":"2015-08-13T09:14:48Z","updated_at":"2016-02-01T16:01:54Z","closed_at":null,"body":"Current Local TM is giving matches based on Elasticsearches Levenshtein distance.  The matches do have similarity but in quite a few cases the distance is so great that its not a useful suggestion.  There are cases where substrings within the match prove useful (and indicate the likely reason the match was found).\r\n\r\nIt does seems that ES's idea of Levenshtein is different from what we are used to.  So we either need delve deaper to find some setting to alter or apply our own Levenshtein to the matches proposed."},"comment":{"url":"https://api.github.com/repos/translate/pootle/issues/comments/178043055","html_url":"https://github.com/translate/pootle/issues/3971#issuecomment-178043055","issue_url":"https://api.github.com/repos/translate/pootle/issues/3971","id":178043055,"user":{"login":"julen","id":16768,"avatar_url":"https://avatars.githubusercontent.com/u/16768?v=3","gravatar_id":"","url":"https://api.github.com/users/julen","html_url":"https://github.com/julen","followers_url":"https://api.github.com/users/julen/followers","following_url":"https://api.github.com/users/julen/following{/other_user}","gists_url":"https://api.github.com/users/julen/gists{/gist_id}","starred_url":"https://api.github.com/users/julen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/julen/subscriptions","organizations_url":"https://api.github.com/users/julen/orgs","repos_url":"https://api.github.com/users/julen/repos","events_url":"https://api.github.com/users/julen/events{/privacy}","received_events_url":"https://api.github.com/users/julen/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T16:01:54Z","updated_at":"2016-02-01T16:01:54Z","body":"From my understanding, since ad7439c there's a sort of _regression_ in this aspect: result sets were previously sliced [based on their relevance score](https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-intro.html#relevance-intro), and therefore users see matches that can be close as well as really far from the source text.\r\n\r\nNotice [how relevance score was being considered](https://github.com/translate/pootle/commit/173f201) in the past. Also note that [minimum score filtering can also be done when querying](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/search-request-min-score.html). Nevertheless I'm not quite sure whether we should [rely on score when doing fuzzy matching or not](https://www.elastic.co/guide/en/elasticsearch/guide/current/fuzzy-scoring.html).\r\n\r\nAlso I admit that these might smell like _hacky_ ways to slice the obtained results, instead of making use of the power of the search syntax to have more accurate results.  \r\nFor instance if I add `\"operator\": \"and\"` in the `\"match\": \"source\"` section, it feels like I can remove the results that are very far from the source text; some other times texts that could be close enough to the source text go missing though.\r\n\r\nSo while I don't have a definite answer on how to proceed here, what is clear to me is that `MIN_SCORE`, with its initial semantics, is not meant to be used as a parameter to `fuzziness`, and it's misleading to say the least.\r\n\r\nAdded to this last point, note that using `fuzziness` values in the `[0.0 .. 1.0]` range [are not supported since version 2.0](https://github.com/elastic/elasticsearch/issues/10638), and here Pootle should probably always use [`AUTO`, as suggested by the docs](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/common-options.html#_string_fields)."}}