{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15800","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15800/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15800/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15800/events","html_url":"https://github.com/apache/spark/pull/15800","id":187835881,"number":15800,"title":"[SPARK-18334] MinHash should use binary hash distance","user":{"login":"Yunni","id":6089987,"avatar_url":"https://avatars.githubusercontent.com/u/6089987?v=3","gravatar_id":"","url":"https://api.github.com/users/Yunni","html_url":"https://github.com/Yunni","followers_url":"https://api.github.com/users/Yunni/followers","following_url":"https://api.github.com/users/Yunni/following{/other_user}","gists_url":"https://api.github.com/users/Yunni/gists{/gist_id}","starred_url":"https://api.github.com/users/Yunni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yunni/subscriptions","organizations_url":"https://api.github.com/users/Yunni/orgs","repos_url":"https://api.github.com/users/Yunni/repos","events_url":"https://api.github.com/users/Yunni/events{/privacy}","received_events_url":"https://api.github.com/users/Yunni/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-07T21:36:40Z","updated_at":"2016-11-08T16:25:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15800","html_url":"https://github.com/apache/spark/pull/15800","diff_url":"https://github.com/apache/spark/pull/15800.diff","patch_url":"https://github.com/apache/spark/pull/15800.patch"},"body":"## What changes were proposed in this pull request?\r\nMinHash currently is using the same `hashDistance` function as RandomProjection. This does not make sense for MinHash because the Jaccard distance of two sets is not relevant to the absolute distance of their hash buckets indices.\r\nThis bug could affect accuracy of multi probing NN search for MinHash.\r\n\r\nMinHash hash distance should just be binary since there is no distance on the buckets.\r\n\r\n## How was this patch tested?\r\nAn incorrect unit test was also introduced, and it's fixed in this PR.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259184437","html_url":"https://github.com/apache/spark/pull/15800#issuecomment-259184437","issue_url":"https://api.github.com/repos/apache/spark/issues/15800","id":259184437,"user":{"login":"sethah","id":7275795,"avatar_url":"https://avatars.githubusercontent.com/u/7275795?v=3","gravatar_id":"","url":"https://api.github.com/users/sethah","html_url":"https://github.com/sethah","followers_url":"https://api.github.com/users/sethah/followers","following_url":"https://api.github.com/users/sethah/following{/other_user}","gists_url":"https://api.github.com/users/sethah/gists{/gist_id}","starred_url":"https://api.github.com/users/sethah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethah/subscriptions","organizations_url":"https://api.github.com/users/sethah/orgs","repos_url":"https://api.github.com/users/sethah/repos","events_url":"https://api.github.com/users/sethah/events{/privacy}","received_events_url":"https://api.github.com/users/sethah/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T16:25:57Z","updated_at":"2016-11-08T16:25:57Z","body":"Using this as hashing distance for near-neighbor search doesn't make sense to me. If there aren't enough candidates where the distance is zero, we'll select some candidates who have distance one. But these are just random candidates since distance of one doesn't correspond to being similar at all, if my understanding is correct. Does minhash really fit the abstraction of multi-probing? I notice that they only use hyperplane projection method in [this paper](http://www.cs.princeton.edu/cass/papers/mplsh_vldb07.pdf)."}}
{"actor":{"display_login":"sethah"}}