{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15148","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15148/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15148/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15148/events","html_url":"https://github.com/apache/spark/pull/15148","id":177712968,"number":15148,"title":"[SPARK-5992][ML] Locality Sensitive Hashing","user":{"login":"Yunni","id":6089987,"avatar_url":"https://avatars.githubusercontent.com/u/6089987?v=3","gravatar_id":"","url":"https://api.github.com/users/Yunni","html_url":"https://github.com/Yunni","followers_url":"https://api.github.com/users/Yunni/followers","following_url":"https://api.github.com/users/Yunni/following{/other_user}","gists_url":"https://api.github.com/users/Yunni/gists{/gist_id}","starred_url":"https://api.github.com/users/Yunni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yunni/subscriptions","organizations_url":"https://api.github.com/users/Yunni/orgs","repos_url":"https://api.github.com/users/Yunni/repos","events_url":"https://api.github.com/users/Yunni/events{/privacy}","received_events_url":"https://api.github.com/users/Yunni/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":82,"created_at":"2016-09-19T06:38:23Z","updated_at":"2016-11-10T00:11:33Z","closed_at":"2016-10-28T22:00:15Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15148","html_url":"https://github.com/apache/spark/pull/15148","diff_url":"https://github.com/apache/spark/pull/15148.diff","patch_url":"https://github.com/apache/spark/pull/15148.patch"},"body":"## What changes were proposed in this pull request?\n\nImplement Locality Sensitive Hashing along with approximate nearest neighbors and approximate similarity join based on the [design doc](https://docs.google.com/document/d/1D15DTDMF_UWTTyWqXfG7y76iZalky4QmifUYQ6lH5GM/edit).\n\nDetailed changes are as follows:\n(1) Implement abstract LSH, LSHModel classes as Estimator-Model\n(2) Implement approxNearestNeighbors and approxSimilarityJoin in the abstract LSHModel\n(3) Implement Random Projection as LSH subclass for Euclidean distance, Min Hash for Jaccard Distance\n(4) Implement unit test utility methods including checkLshProperty, checkNearestNeighbor and checkSimilarityJoin\n\nThings that will be implemented in a follow-up PR:\n- Bit Sampling for Hamming Distance, SignRandomProjection for Cosine Distance\n- PySpark Integration for the scala classes and methods.\n## How was this patch tested?\n\nUnit test is implemented for all the implemented classes and algorithms. A scalability test on Uber's dataset was performed internally.\n\nTested the methods on [WEX dataset](https://aws.amazon.com/items/2345) from AWS, with the steps and results [here](https://docs.google.com/document/d/19BXg-67U83NVB3M0I84HVBVg3baAVaESD_mrg_-vLro/edit).\n## References\n\nGionis, Aristides, Piotr Indyk, and Rajeev Motwani. \"Similarity search in high dimensions via hashing.\" VLDB 7 Sep. 1999: 518-529.\nWang, Jingdong et al. \"Hashing for similarity search: A survey.\" arXiv preprint arXiv:1408.2927 (2014).\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259563972","html_url":"https://github.com/apache/spark/pull/15148#issuecomment-259563972","issue_url":"https://api.github.com/repos/apache/spark/issues/15148","id":259563972,"user":{"login":"Yunni","id":6089987,"avatar_url":"https://avatars.githubusercontent.com/u/6089987?v=3","gravatar_id":"","url":"https://api.github.com/users/Yunni","html_url":"https://github.com/Yunni","followers_url":"https://api.github.com/users/Yunni/followers","following_url":"https://api.github.com/users/Yunni/following{/other_user}","gists_url":"https://api.github.com/users/Yunni/gists{/gist_id}","starred_url":"https://api.github.com/users/Yunni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yunni/subscriptions","organizations_url":"https://api.github.com/users/Yunni/orgs","repos_url":"https://api.github.com/users/Yunni/repos","events_url":"https://api.github.com/users/Yunni/events{/privacy}","received_events_url":"https://api.github.com/users/Yunni/received_events","type":"User","site_admin":false},"created_at":"2016-11-10T00:11:33Z","updated_at":"2016-11-10T00:11:33Z","body":"Thanks for the discussion, everyone! I will take a look at the JIRA."}}
{"actor":{"display_login":"Yunni"}}