{"action":"created","issue":{"url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/20","repository_url":"https://api.github.com/repos/wikimedia/search-highlighter","labels_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/20/comments","events_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/20/events","html_url":"https://github.com/wikimedia/search-highlighter/issues/20","id":185134626,"number":20,"title":"Highlighting Not Honoring Slop","user":{"login":"mkrauklis","id":711393,"avatar_url":"https://avatars.githubusercontent.com/u/711393?v=3","gravatar_id":"","url":"https://api.github.com/users/mkrauklis","html_url":"https://github.com/mkrauklis","followers_url":"https://api.github.com/users/mkrauklis/followers","following_url":"https://api.github.com/users/mkrauklis/following{/other_user}","gists_url":"https://api.github.com/users/mkrauklis/gists{/gist_id}","starred_url":"https://api.github.com/users/mkrauklis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkrauklis/subscriptions","organizations_url":"https://api.github.com/users/mkrauklis/orgs","repos_url":"https://api.github.com/users/mkrauklis/repos","events_url":"https://api.github.com/users/mkrauklis/events{/privacy}","received_events_url":"https://api.github.com/users/mkrauklis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-10-25T14:30:02Z","updated_at":"2016-10-31T18:50:29Z","closed_at":null,"body":"I should first note this exact bug exists with the Plain Highlighter in 1.7.x-2.4.x and is supposedly fixed in 5.x as reported here: https://github.com/elastic/elasticsearch/issues/18246\n\nThe root of the problem is that although the we only get hits on documents that meet the slop criteria for a span_near query, the highlighter highlights all instances of the matched terms regardless of proximity.\n\nFor example, the following request:\n\n``` json\nPOST /idx/_search\n{\n   \"highlight\": {\n      \"pre_tags\": [\n         \"|~|\"\n      ],\n      \"post_tags\": [\n         \"|^|\"\n      ],\n      \"order\": \"score\",\n      \"fields\": {\n         \"fieldx\": {\n            \"fragment_size\": 1000,\n            \"number_of_fragments\": 3,\n            \"type\": \"experimental\",\n            \"matched_fields\": [\n               \"fieldx\",\n               \"fieldx.exact\"\n            ]\n         }\n      }\n   },\n   \"query\": {\n      \"span_near\": {\n         \"clauses\": [\n            {\n               \"span_term\": {\n                  \"fieldx.exact\": {\n                     \"value\": \"term1\"\n                  }\n               }\n            },\n            {\n               \"span_term\": {\n                  \"fieldx.exact\": {\n                     \"value\": \"term2\"\n                  }\n               }\n            }\n         ],\n         \"slop\": 2,\n         \"in_order\": false\n      }\n   }\n}\n```\n\nReturns the following highlight:\n\n``` json\n...\n\"fieldx\": [\"Shouldn't hit |~|term1|^| because it's not close enough to |~|term2|^|. However, the distance between the second instances of |~|term1|^| and |~|term2|^| are close enough and should get highlighted.\"]\n...\n```\n\nIn this example the highlights for the first instance of the search terms are incorrect.\n\nWe've traced this into the PostingsHitEnum's iteration through the PostingsEnum which is returning the offsets for the incorrect term offsets.\n"},"comment":{"url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/comments/257384966","html_url":"https://github.com/wikimedia/search-highlighter/issues/20#issuecomment-257384966","issue_url":"https://api.github.com/repos/wikimedia/search-highlighter/issues/20","id":257384966,"user":{"login":"mkrauklis","id":711393,"avatar_url":"https://avatars.githubusercontent.com/u/711393?v=3","gravatar_id":"","url":"https://api.github.com/users/mkrauklis","html_url":"https://github.com/mkrauklis","followers_url":"https://api.github.com/users/mkrauklis/followers","following_url":"https://api.github.com/users/mkrauklis/following{/other_user}","gists_url":"https://api.github.com/users/mkrauklis/gists{/gist_id}","starred_url":"https://api.github.com/users/mkrauklis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkrauklis/subscriptions","organizations_url":"https://api.github.com/users/mkrauklis/orgs","repos_url":"https://api.github.com/users/mkrauklis/repos","events_url":"https://api.github.com/users/mkrauklis/events{/privacy}","received_events_url":"https://api.github.com/users/mkrauklis/received_events","type":"User","site_admin":false},"created_at":"2016-10-31T18:50:29Z","updated_at":"2016-10-31T18:50:29Z","body":"Alrighty, I think I figured it out. I was missing that in QueryFlattener we were passing a 0 for the weight of terms that were part of a phrase.\r\n\r\nI think I have a partial solution implemented, piggybacking on a lot of the concepts for phrase handling. So far I seem to have it working for SpanTerm and SpanNear queries and would like to run the concept past you experts before I spend time implementing the rest.\r\n\r\nThe high level solution is to, while flattening the query, build up a list of root-level `SpanValidator` trees. Then, when wrapping (`BasicQueryWeigher.wrap`), for each root-level validator I wrap the current HitEnum with a `SpanHitEnumWrapper`.\r\n\r\nThe `SpanHitEnumWrapper` tries to validate the current hit. If there is a partial match (e.g. we have a hit on one of the `SpanTerms` in a list of span near clauses) it looks forward (`wrapped.next()`) until either there is a match or there is no match. For example, if the positions of the current hits are outside the bounds allowed by slop. When validating each acceptable hit is stored by that validator node.\r\n\r\nLastly, when `SpanHitEnumWrapper.queryWeight()` is called it checks with its validator to see if the given source/position combination was recorded as a successful hit during validation:\r\n\r\n```java\r\n\t@Override\r\n\tpublic float queryWeight() {\r\n\t\tfloat queryWeight = toReplay.queryWeight();\r\n\r\n\t\tif (queryWeight <= 0) {\r\n\t\t\tif (spanValidator.isSuccessful()) {\r\n\t\t\t\tif (spanValidator.isSourceAtPositionValid(source(), position())) {\r\n\t\t\t\t\tqueryWeight = 1;\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t}\r\n\r\n\t\treturn queryWeight;\r\n\t}\r\n```\r\n\r\nA little more detail:\r\n\r\nI added functions to QueryFlattener.Callback to mark the start and end of the span types:\r\n```java\r\n\t\tvoid startSpanTermQuery(SpanTermQuery query);\r\n\t\tvoid endSpanTermQuery(SpanTermQuery query, int source);\r\n\t\tvoid startSpanNearQuery(SpanNearQuery query);\r\n\t\tvoid endSpanNearQuery(SpanNearQuery query);\r\n```\r\n\r\nThese call back into `BasicQueryWeigher` which maintains a stack of `List` of `SpanValidator` instances. When the bottom of the stack is popped off this validator is added to the list of root-level validators for the given field.\r\n\r\nConcerns:\r\n* It seems to be following the pattern established by phrases but with this solution we're throwing away queryWeight/boost for nested span queries. This seems like it could be more problematic than it was for phrases.\r\n"}}
{"actor":{"display_login":"mkrauklis"}}