{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15800","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15800/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15800/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15800/events","html_url":"https://github.com/apache/spark/pull/15800","id":187835881,"number":15800,"title":"[SPARK-18334] MinHash should use binary hash distance","user":{"login":"Yunni","id":6089987,"avatar_url":"https://avatars.githubusercontent.com/u/6089987?v=3","gravatar_id":"","url":"https://api.github.com/users/Yunni","html_url":"https://github.com/Yunni","followers_url":"https://api.github.com/users/Yunni/followers","following_url":"https://api.github.com/users/Yunni/following{/other_user}","gists_url":"https://api.github.com/users/Yunni/gists{/gist_id}","starred_url":"https://api.github.com/users/Yunni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yunni/subscriptions","organizations_url":"https://api.github.com/users/Yunni/orgs","repos_url":"https://api.github.com/users/Yunni/repos","events_url":"https://api.github.com/users/Yunni/events{/privacy}","received_events_url":"https://api.github.com/users/Yunni/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2016-11-07T21:36:40Z","updated_at":"2016-11-10T18:06:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15800","html_url":"https://github.com/apache/spark/pull/15800","diff_url":"https://github.com/apache/spark/pull/15800.diff","patch_url":"https://github.com/apache/spark/pull/15800.patch"},"body":"## What changes were proposed in this pull request?\r\nMinHash currently is using the same `hashDistance` function as RandomProjection. This does not make sense for MinHash because the Jaccard distance of two sets is not relevant to the absolute distance of their hash buckets indices.\r\nThis bug could affect accuracy of multi probing NN search for MinHash.\r\n\r\nMinHash hash distance should just be binary since there is no distance on the buckets.\r\n\r\n## How was this patch tested?\r\nAn incorrect unit test was also introduced, and it's fixed in this PR.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259763068","html_url":"https://github.com/apache/spark/pull/15800#issuecomment-259763068","issue_url":"https://api.github.com/repos/apache/spark/issues/15800","id":259763068,"user":{"login":"Yunni","id":6089987,"avatar_url":"https://avatars.githubusercontent.com/u/6089987?v=3","gravatar_id":"","url":"https://api.github.com/users/Yunni","html_url":"https://github.com/Yunni","followers_url":"https://api.github.com/users/Yunni/followers","following_url":"https://api.github.com/users/Yunni/following{/other_user}","gists_url":"https://api.github.com/users/Yunni/gists{/gist_id}","starred_url":"https://api.github.com/users/Yunni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yunni/subscriptions","organizations_url":"https://api.github.com/users/Yunni/orgs","repos_url":"https://api.github.com/users/Yunni/repos","events_url":"https://api.github.com/users/Yunni/events{/privacy}","received_events_url":"https://api.github.com/users/Yunni/received_events","type":"User","site_admin":false},"created_at":"2016-11-10T18:06:35Z","updated_at":"2016-11-10T18:06:35Z","body":"> If a query point vector q hashes to some MinHash Vector [5.0, 22.0, 13.0] the best candidates will be ones that hash to that same vector.\r\n\r\nMy second half is suggesting: If a query point vector q hashes to MinHash Array `[[5.0, 10.0], [22.0, 7.0], [13.0, 25.0]]`, the buckets to search will be ones that hash to any of the MinHash vectors like\r\n- `[[5.0, 10.0], [39.0, 7.0], [18.0, 99.0]]`\r\n- `[[100.0, 77.0], [22.0, 7.0], [13.0, 2.0]]`\r\n- `[[78.0, 40.0], [96.0, 55.0], [13.0, 25.0]]`\r\n- ......\r\n\r\nTo increase chance to find the exact k nearest neighbor."}}
{"actor":{"display_login":"Yunni"}}