{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/86394739","pull_request_review_id":7056442,"id":86394739,"diff_hunk":"@@ -247,15 +247,16 @@ abstract class HashExpression[E] extends Expression {\n \n   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {\n     ev.isNull = \"false\"\n-    val childrenHash = children.map { child =>\n+    val childrenHash = ctx.splitExpressions(ctx.INPUT_ROW, children.map { child =>","path":"sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala","position":5,"original_position":5,"commit_id":"9346117b53fdefbcbc3391c8e05de066abb80811","original_commit_id":"9346117b53fdefbcbc3391c8e05de066abb80811","user":{"login":"hvanhovell","id":9616802,"avatar_url":"https://avatars.githubusercontent.com/u/9616802?v=3","gravatar_id":"","url":"https://api.github.com/users/hvanhovell","html_url":"https://github.com/hvanhovell","followers_url":"https://api.github.com/users/hvanhovell/followers","following_url":"https://api.github.com/users/hvanhovell/following{/other_user}","gists_url":"https://api.github.com/users/hvanhovell/gists{/gist_id}","starred_url":"https://api.github.com/users/hvanhovell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hvanhovell/subscriptions","organizations_url":"https://api.github.com/users/hvanhovell/orgs","repos_url":"https://api.github.com/users/hvanhovell/repos","events_url":"https://api.github.com/users/hvanhovell/events{/privacy}","received_events_url":"https://api.github.com/users/hvanhovell/received_events","type":"User","site_admin":false},"body":"I think you also need to add this to `HiveHash`.","created_at":"2016-11-03T17:09:27Z","updated_at":"2016-11-03T17:09:27Z","html_url":"https://github.com/apache/spark/pull/15745#discussion_r86394739","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/15745","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/86394739"},"html":{"href":"https://github.com/apache/spark/pull/15745#discussion_r86394739"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/15745"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15745","id":92089755,"html_url":"https://github.com/apache/spark/pull/15745","diff_url":"https://github.com/apache/spark/pull/15745.diff","patch_url":"https://github.com/apache/spark/pull/15745.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/15745","number":15745,"state":"open","locked":false,"title":"[SPARK-18207][SQL] Fix a compilation error due to HashExpression.doGenCode","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR avoids a compilation error due to more than 64KB Java byte code size. This error occur since  generate java code for computing a hash value for a row is too big. This PR fixes this compilation error by splitting a big code chunk into multiple methods by calling `CodegenContext.splitExpression` at `HashExpression.doGenCode`\r\n\r\nThe test case requires a calculation of hash code for a row that includes 1000 String fields. `HashExpression.doGenCode` generate a lot of Java code for this computation into one function. As a result, the size of the corresponding Java bytecode is more than 64 KB.\r\n\r\nGenerated code without this PR\r\n````java\r\n/* 027 */   public UnsafeRow apply(InternalRow i) {\r\n/* 028 */     boolean isNull = false;\r\n/* 029 */\r\n/* 030 */     int value1 = 42;\r\n/* 031 */\r\n/* 032 */     boolean isNull2 = i.isNullAt(0);\r\n/* 033 */     UTF8String value2 = isNull2 ? null : (i.getUTF8String(0));\r\n/* 034 */     if (!isNull2) {\r\n/* 035 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value2.getBaseObject(), value2.getBaseOffset(), value2.numBytes(), value1);\r\n/* 036 */     }\r\n/* 037 */\r\n/* 038 */\r\n/* 039 */     boolean isNull3 = i.isNullAt(1);\r\n/* 040 */     UTF8String value3 = isNull3 ? null : (i.getUTF8String(1));\r\n/* 041 */     if (!isNull3) {\r\n/* 042 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value3.getBaseObject(), value3.getBaseOffset(), value3.numBytes(), value1);\r\n/* 043 */     }\r\n/* 044 */\r\n/* 045 */\r\n...\r\n/* 7024 */\r\n/* 7025 */     boolean isNull1001 = i.isNullAt(999);\r\n/* 7026 */     UTF8String value1001 = isNull1001 ? null : (i.getUTF8String(999));\r\n/* 7027 */     if (!isNull1001) {\r\n/* 7028 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value1001.getBaseObject(), value1001.getBaseOffset(), value1001.numBytes(), value1);\r\n/* 7029 */     }\r\n/* 7030 */\r\n/* 7031 */\r\n/* 7032 */     boolean isNull1002 = i.isNullAt(1000);\r\n/* 7033 */     UTF8String value1002 = isNull1002 ? null : (i.getUTF8String(1000));\r\n/* 7034 */     if (!isNull1002) {\r\n/* 7035 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value1002.getBaseObject(), value1002.getBaseOffset(), value1002.numBytes(), value1);\r\n/* 7036 */     }\r\n````\r\n\r\nGenerated code with this PR\r\n````java\r\n/* 3807 */   private void apply_249(InternalRow i) {\r\n/* 3808 */\r\n/* 3809 */     boolean isNull998 = i.isNullAt(996);\r\n/* 3810 */     UTF8String value998 = isNull998 ? null : (i.getUTF8String(996));\r\n/* 3811 */     if (!isNull998) {\r\n/* 3812 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value998.getBaseObject(), value998.getBaseOffset(), value998.numBytes(), value1);\r\n/* 3813 */     }\r\n/* 3814 */\r\n/* 3815 */     boolean isNull999 = i.isNullAt(997);\r\n/* 3816 */     UTF8String value999 = isNull999 ? null : (i.getUTF8String(997));\r\n/* 3817 */     if (!isNull999) {\r\n/* 3818 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value999.getBaseObject(), value999.getBaseOffset(), value999.numBytes(), value1);\r\n/* 3819 */     }\r\n/* 3820 */\r\n/* 3821 */     boolean isNull1000 = i.isNullAt(998);\r\n/* 3822 */     UTF8String value1000 = isNull1000 ? null : (i.getUTF8String(998));\r\n/* 3823 */     if (!isNull1000) {\r\n/* 3824 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value1000.getBaseObject(), value1000.getBaseOffset(), value1000.numBytes(), value1);\r\n/* 3825 */     }\r\n/* 3826 */\r\n/* 3827 */     boolean isNull1001 = i.isNullAt(999);\r\n/* 3828 */     UTF8String value1001 = isNull1001 ? null : (i.getUTF8String(999));\r\n/* 3829 */     if (!isNull1001) {\r\n/* 3830 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value1001.getBaseObject(), value1001.getBaseOffset(), value1001.numBytes(), value1);\r\n/* 3831 */     }\r\n/* 3832 */\r\n/* 3833 */   }\r\n/* 3834 */\r\n...\r\n/* 4532 */   private void apply_0(InternalRow i) {\r\n/* 4533 */\r\n/* 4534 */     boolean isNull2 = i.isNullAt(0);\r\n/* 4535 */     UTF8String value2 = isNull2 ? null : (i.getUTF8String(0));\r\n/* 4536 */     if (!isNull2) {\r\n/* 4537 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value2.getBaseObject(), value2.getBaseOffset(), value2.numBytes(), value1);\r\n/* 4538 */     }\r\n/* 4539 */\r\n/* 4540 */     boolean isNull3 = i.isNullAt(1);\r\n/* 4541 */     UTF8String value3 = isNull3 ? null : (i.getUTF8String(1));\r\n/* 4542 */     if (!isNull3) {\r\n/* 4543 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value3.getBaseObject(), value3.getBaseOffset(), value3.numBytes(), value1);\r\n/* 4544 */     }\r\n/* 4545 */\r\n/* 4546 */     boolean isNull4 = i.isNullAt(2);\r\n/* 4547 */     UTF8String value4 = isNull4 ? null : (i.getUTF8String(2));\r\n/* 4548 */     if (!isNull4) {\r\n/* 4549 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value4.getBaseObject(), value4.getBaseOffset(), value4.numBytes(), value1);\r\n/* 4550 */     }\r\n/* 4551 */\r\n/* 4552 */     boolean isNull5 = i.isNullAt(3);\r\n/* 4553 */     UTF8String value5 = isNull5 ? null : (i.getUTF8String(3));\r\n/* 4554 */     if (!isNull5) {\r\n/* 4555 */       value1 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(value5.getBaseObject(), value5.getBaseOffset(), value5.numBytes(), value1);\r\n/* 4556 */     }\r\n/* 4557 */\r\n/* 4558 */   }\r\n...\r\n/* 7344 */   public UnsafeRow apply(InternalRow i) {\r\n/* 7345 */     boolean isNull = false;\r\n/* 7346 */\r\n/* 7347 */     value1 = 42;\r\n/* 7348 */     apply_0(i);\r\n/* 7349 */     apply_1(i);\r\n...\r\n/* 7596 */     apply_248(i);\r\n/* 7597 */     apply_249(i);\r\n/* 7598 */     apply_250(i);\r\n/* 7599 */     apply_251(i);\r\n...\r\n````\r\n\r\n\r\n## How was this patch tested?\r\n\r\nAdd a new test in `DataFrameSuite`\r\n","created_at":"2016-11-03T02:46:19Z","updated_at":"2016-11-03T17:09:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ea8d46c7c2ecb98cfb2651315f082652ba17c060","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/15745/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/15745/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/15745/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/9346117b53fdefbcbc3391c8e05de066abb80811","head":{"label":"kiszk:SPARK-18207","ref":"SPARK-18207","sha":"9346117b53fdefbcbc3391c8e05de066abb80811","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"repo":{"id":45008789,"name":"spark","full_name":"kiszk/spark","owner":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kiszk/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/kiszk/spark","forks_url":"https://api.github.com/repos/kiszk/spark/forks","keys_url":"https://api.github.com/repos/kiszk/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kiszk/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kiszk/spark/teams","hooks_url":"https://api.github.com/repos/kiszk/spark/hooks","issue_events_url":"https://api.github.com/repos/kiszk/spark/issues/events{/number}","events_url":"https://api.github.com/repos/kiszk/spark/events","assignees_url":"https://api.github.com/repos/kiszk/spark/assignees{/user}","branches_url":"https://api.github.com/repos/kiszk/spark/branches{/branch}","tags_url":"https://api.github.com/repos/kiszk/spark/tags","blobs_url":"https://api.github.com/repos/kiszk/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kiszk/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kiszk/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/kiszk/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kiszk/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/kiszk/spark/languages","stargazers_url":"https://api.github.com/repos/kiszk/spark/stargazers","contributors_url":"https://api.github.com/repos/kiszk/spark/contributors","subscribers_url":"https://api.github.com/repos/kiszk/spark/subscribers","subscription_url":"https://api.github.com/repos/kiszk/spark/subscription","commits_url":"https://api.github.com/repos/kiszk/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/kiszk/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/kiszk/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/kiszk/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/kiszk/spark/contents/{+path}","compare_url":"https://api.github.com/repos/kiszk/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kiszk/spark/merges","archive_url":"https://api.github.com/repos/kiszk/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kiszk/spark/downloads","issues_url":"https://api.github.com/repos/kiszk/spark/issues{/number}","pulls_url":"https://api.github.com/repos/kiszk/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/kiszk/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/kiszk/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kiszk/spark/labels{/name}","releases_url":"https://api.github.com/repos/kiszk/spark/releases{/id}","deployments_url":"https://api.github.com/repos/kiszk/spark/deployments","created_at":"2015-10-27T01:25:07Z","updated_at":"2016-06-12T04:05:26Z","pushed_at":"2016-11-03T09:37:41Z","git_url":"git://github.com/kiszk/spark.git","ssh_url":"git@github.com:kiszk/spark.git","clone_url":"https://github.com/kiszk/spark.git","svn_url":"https://github.com/kiszk/spark","homepage":null,"size":243160,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"3a1bc6f4780f8384c1211b1335e7394a4a28377e","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-11-03T15:35:02Z","pushed_at":"2016-11-03T16:54:30Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":606032,"stargazers_count":10609,"watchers_count":10609,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9889,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":460,"forks":9889,"open_issues":460,"watchers":10609,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/15745"},"html":{"href":"https://github.com/apache/spark/pull/15745"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/15745"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/15745/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/15745/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/15745/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/9346117b53fdefbcbc3391c8e05de066abb80811"}}}}
{"actor":{"display_login":"hvanhovell"}}