{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15047","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15047/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15047/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15047/events","html_url":"https://github.com/apache/spark/pull/15047","id":176198607,"number":15047,"title":"[SPARK-17495] [SQL] Add Hash capability semantically equivalent to Hive's","user":{"login":"tejasapatil","id":3230057,"avatar_url":"https://avatars.githubusercontent.com/u/3230057?v=3","gravatar_id":"","url":"https://api.github.com/users/tejasapatil","html_url":"https://github.com/tejasapatil","followers_url":"https://api.github.com/users/tejasapatil/followers","following_url":"https://api.github.com/users/tejasapatil/following{/other_user}","gists_url":"https://api.github.com/users/tejasapatil/gists{/gist_id}","starred_url":"https://api.github.com/users/tejasapatil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tejasapatil/subscriptions","organizations_url":"https://api.github.com/users/tejasapatil/orgs","repos_url":"https://api.github.com/users/tejasapatil/repos","events_url":"https://api.github.com/users/tejasapatil/events{/privacy}","received_events_url":"https://api.github.com/users/tejasapatil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-10T19:40:29Z","updated_at":"2016-09-10T21:09:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15047","html_url":"https://github.com/apache/spark/pull/15047","diff_url":"https://github.com/apache/spark/pull/15047.diff","patch_url":"https://github.com/apache/spark/pull/15047.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nJira : https://issues.apache.org/jira/browse/SPARK-17495\r\n\r\nSpark internally uses Murmur3Hash for partitioning. This is different from the one used by Hive. For queries which use bucketing this leads to different results if one tries the same query on both engines. For us, we want users to have backward compatibility to that one can switch parts of applications across the engines without observing regressions.\r\n\r\nThis PR includes `HiveHash`, `HiveHashFunction`, `HiveHasher` which mimics Hive's hashing at https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L638\r\n\r\nI am intentionally not introducing any usages of this hash function in rest of the code to keep this PR small. My eventual goal is to have Hive bucketing support in Spark. Once this PR gets in, I will make hash function pluggable in relevant areas (eg. `HashPartitioning`'s `partitionIdExpression` has Murmur3 hardcoded : https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala#L265)\r\n\r\n## How was this patch tested?\r\n\r\nAdded `HiveHashSuite`"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/246140528","html_url":"https://github.com/apache/spark/pull/15047#issuecomment-246140528","issue_url":"https://api.github.com/repos/apache/spark/issues/15047","id":246140528,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-09-10T21:09:19Z","updated_at":"2016-09-10T21:09:19Z","body":"**[Test build #65214 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65214/consoleFull)** for PR 15047 at commit [`c898f5a`](https://github.com/apache/spark/commit/c898f5af10ead29416fec7fee49de5c37a7f48cb).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `case class HiveHash(children: Seq[Expression], seed: Int) extends HashExpression[Int] `"}}
{"actor":{"display_login":"SparkQA"}}