{"push_id":976838897,"size":4,"distinct_size":1,"ref":"refs/heads/master","head":"989c53767f42b3b51f0289814acb13c87b3e0f57","before":"2a30159ee53b1bb6f79aa0c161948b856765dca1","commits":[{"sha":"74b75c80dd0e8408f9e7a1496ce758499474f33d","author":{"email":"0821cf763cd5aa19e3ea8599841d6cc9a772d397@gmail.com","name":"Karl Higley"},"message":"Include points in the hash tables (instead of just ids)\n\nThis eliminates two joins, which reduces the need to persist\nintermediate RDDs in memory/on disk. It does increase the memory\nrequired to store the hash tables, but since it ends up materializing\nthe same set of candidate pairs, the memory required by the distance\ncomputation doesn't change much.","distinct":false,"url":"https://api.github.com/repos/karlhigley/spark-neighbors/commits/74b75c80dd0e8408f9e7a1496ce758499474f33d"},{"sha":"19021a2ef44d2f6cde6c0520bac4df0760bb791d","author":{"email":"0821cf763cd5aa19e3ea8599841d6cc9a772d397@gmail.com","name":"Karl Higley"},"message":"Persist the hash tables in memory/on disk\n\nAfter including the points in the hash tables, less joins are required,\nwhich makes persistence less of an issue. The candidate pairs, for\nexample, are no longer persisted, because there's no join and therefore\nno need to avoid recomputing the RDD for each half of the join.\n\nThis change removes persistence from CandidateStrategy and (mostly) from\nANNModel. The hash tables are now persisted while building the model,\nwhich allows the removal of persistenceLevel as a model member.","distinct":false,"url":"https://api.github.com/repos/karlhigley/spark-neighbors/commits/19021a2ef44d2f6cde6c0520bac4df0760bb791d"},{"sha":"b75fc0e2402276282fb07d73a86c9f830160bd4b","author":{"email":"0821cf763cd5aa19e3ea8599841d6cc9a772d397@gmail.com","name":"Karl Higley"},"message":"Repartition candidates after identifying them from hash tables\n\nWith the points included in the hash tables, the size of the tables is\nnow larger, which increases the memory pressure on executors. Since the\ndensity of points is unlikely to be uniform and the LSH functions\ninvolve some randomness, the identified candidate pairs will suffer from\npartition skew. That makes it important to repartition after identifying\ncandidate pairs, both to reduce the memory pressure and to speed up\nsubsequent steps (e.g. distance computation).","distinct":false,"url":"https://api.github.com/repos/karlhigley/spark-neighbors/commits/b75fc0e2402276282fb07d73a86c9f830160bd4b"},{"sha":"989c53767f42b3b51f0289814acb13c87b3e0f57","author":{"email":"0821cf763cd5aa19e3ea8599841d6cc9a772d397@gmail.com","name":"Karl Higley"},"message":"Merge pull request #13 from karlhigley/fix/candidates\n\nSimplify candidate generation and persistence","distinct":true,"url":"https://api.github.com/repos/karlhigley/spark-neighbors/commits/989c53767f42b3b51f0289814acb13c87b3e0f57"}]}