{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/183253226","pull_request_review_id":114216457,"id":183253226,"diff_hunk":"@@ -883,3 +884,140 @@ case class Concat(children: Seq[Expression]) extends Expression {\n \n   override def sql: String = s\"concat(${children.map(_.sql).mkString(\", \")})\"\n }\n+\n+/**\n+ * Returns the maximum value in the array.\n+ */\n+// scalastyle:off line.size.limit\n+@ExpressionDescription(\n+  usage = \"_FUNC_(array[, indexFirst]) - Transforms the input array by encapsulating elements into pairs with indexes indicating the order.\",","path":"sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala","position":18,"original_position":18,"commit_id":"06348c3cf53d3631fd620b9e4d46b408a30b341e","original_commit_id":"06348c3cf53d3631fd620b9e4d46b408a30b341e","user":{"login":"mn-mikke","id":3674621,"avatar_url":"https://avatars3.githubusercontent.com/u/3674621?v=4","gravatar_id":"","url":"https://api.github.com/users/mn-mikke","html_url":"https://github.com/mn-mikke","followers_url":"https://api.github.com/users/mn-mikke/followers","following_url":"https://api.github.com/users/mn-mikke/following{/other_user}","gists_url":"https://api.github.com/users/mn-mikke/gists{/gist_id}","starred_url":"https://api.github.com/users/mn-mikke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mn-mikke/subscriptions","organizations_url":"https://api.github.com/users/mn-mikke/orgs","repos_url":"https://api.github.com/users/mn-mikke/repos","events_url":"https://api.github.com/users/mn-mikke/events{/privacy}","received_events_url":"https://api.github.com/users/mn-mikke/received_events","type":"User","site_admin":false},"body":"That's really good question! The newly added functions `element_at` and `array_position` are 1-based. But on the other handed, the `getItem` from the `Column` class is 0-based. What about adding one extra parameter and let users decide whether the array will indexed from 0 or 1. ","created_at":"2018-04-22T22:01:26Z","updated_at":"2018-04-22T22:01:26Z","html_url":"https://github.com/apache/spark/pull/21121#discussion_r183253226","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/21121","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/183253226"},"html":{"href":"https://github.com/apache/spark/pull/21121#discussion_r183253226"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/21121"}},"in_reply_to_id":183236405},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21121","id":183221358,"html_url":"https://github.com/apache/spark/pull/21121","diff_url":"https://github.com/apache/spark/pull/21121.diff","patch_url":"https://github.com/apache/spark/pull/21121.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/21121","number":21121,"state":"open","locked":false,"title":"[SPARK-24042][SQL] Collection function: zip_with_index","user":{"login":"mn-mikke","id":3674621,"avatar_url":"https://avatars3.githubusercontent.com/u/3674621?v=4","gravatar_id":"","url":"https://api.github.com/users/mn-mikke","html_url":"https://github.com/mn-mikke","followers_url":"https://api.github.com/users/mn-mikke/followers","following_url":"https://api.github.com/users/mn-mikke/following{/other_user}","gists_url":"https://api.github.com/users/mn-mikke/gists{/gist_id}","starred_url":"https://api.github.com/users/mn-mikke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mn-mikke/subscriptions","organizations_url":"https://api.github.com/users/mn-mikke/orgs","repos_url":"https://api.github.com/users/mn-mikke/repos","events_url":"https://api.github.com/users/mn-mikke/events{/privacy}","received_events_url":"https://api.github.com/users/mn-mikke/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nAdding function zip_with_index(array[, indexFirst]) that transforms the input array by encapsulating elements into pairs with indexes indicating the order.\r\n\r\n```\r\nzip_with_index(array(\"d\", \"a\", null, \"b\")) => [(\"d\",0),(\"a\",1),(null,2),(\"b\",3)]\r\nzip_with_index(array(\"d\", \"a\", null, \"b\"), true) => [(0,\"d\"),(1,\"a\"),(2,null),(3,\"b\")]\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nNew tests added into:\r\n- CollectionExpressionSuite\r\n- DataFrameFunctionsSuite\r\n\r\n## Codegen examples\r\n### Primitive type\r\n```\r\nval df = Seq(\r\n  Seq(1, 3, 4, 2),\r\n  null\r\n).toDF(\"i\")\r\ndf.filter($\"i\".isNotNull || $\"i\".isNull).select(zip_with_index($\"i\")).debugCodegen\r\n```\r\nResult:\r\n```\r\n/* 033 */         boolean inputadapter_isNull_0 = inputadapter_row_0.isNullAt(0);\r\n/* 034 */         ArrayData inputadapter_value_0 = inputadapter_isNull_0 ?\r\n/* 035 */         null : (inputadapter_row_0.getArray(0));\r\n/* 036 */\r\n/* 037 */         boolean filter_value_0 = true;\r\n/* 038 */\r\n/* 039 */         if (!(!inputadapter_isNull_0)) {\r\n/* 040 */           filter_value_0 = inputadapter_isNull_0;\r\n/* 041 */         }\r\n/* 042 */         if (!filter_value_0) continue;\r\n/* 043 */\r\n/* 044 */         ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);\r\n/* 045 */\r\n/* 046 */         boolean project_isNull_0 = inputadapter_isNull_0;\r\n/* 047 */         ArrayData project_value_0 = null;\r\n/* 048 */\r\n/* 049 */         if (!inputadapter_isNull_0) {\r\n/* 050 */           final int project_numElements_0 = inputadapter_value_0.numElements();\r\n/* 051 */           final int project_structSize_0 = 24;\r\n/* 052 */           final long project_byteArraySize_0 = UnsafeArrayData.calculateSizeOfUnderlyingByteArray(project_numElements_0, 8 + project_structSize_0);\r\n/* 053 */           final int project_structsOffset_0 = UnsafeArrayData.calculateHeaderPortionInBytes(project_numElements_0) + project_numElements_0 * 8;\r\n/* 054 */           if (project_byteArraySize_0 > 2147483632) {\r\n/* 055 */             throw new RuntimeException(\"Unsuccessful try to zip array with index due to exceeding\" +\r\n/* 056 */               \" the limit 2147483632 bytes for UnsafeArrayData. \" + project_byteArraySize_0 +\r\n/* 057 */               \" bytes of data are required for performing the operation with the given array.\");\r\n/* 058 */           }\r\n/* 059 */           final byte[] project_byteArray_0 = new byte[(int)project_byteArraySize_0];\r\n/* 060 */           UnsafeArrayData project_unsafeArrayData_0 = new UnsafeArrayData();\r\n/* 061 */           Platform.putLong(project_byteArray_0, 16, project_numElements_0);\r\n/* 062 */           project_unsafeArrayData_0.pointTo(project_byteArray_0, 16, (int)project_byteArraySize_0);\r\n/* 063 */           UnsafeRow project_unsafeRow_0 = new UnsafeRow(2);\r\n/* 064 */           for (int z = 0; z < project_numElements_0; z++) {\r\n/* 065 */             long offset = project_structsOffset_0 + z * project_structSize_0;\r\n/* 066 */             project_unsafeArrayData_0.setLong(z, (offset << 32) + project_structSize_0);\r\n/* 067 */             project_unsafeRow_0.pointTo(project_byteArray_0, 16 + offset, project_structSize_0);\r\n/* 068 */             if (inputadapter_value_0.isNullAt(z)) {\r\n/* 069 */               project_unsafeRow_0.setNullAt(0);\r\n/* 070 */             } else {\r\n/* 071 */               project_unsafeRow_0.setInt(\r\n/* 072 */                 0,\r\n/* 073 */                 inputadapter_value_0.getInt(z)\r\n/* 074 */               );\r\n/* 075 */             }\r\n/* 076 */             project_unsafeRow_0.setInt(1, z);\r\n/* 077 */           }\r\n/* 078 */           project_value_0 = project_unsafeArrayData_0;\r\n/* 079 */\r\n/* 080 */         }\r\n```\r\n### Non-primitive type\r\n```\r\nval df = Seq(\r\n  Seq(\"d\", \"a\", \"f\", \"g\"),\r\n  null\r\n).toDF(\"s\")\r\ndf.filter($\"s\".isNotNull || $\"s\".isNull).select(zip_with_index($\"s\")).debugCodegen\r\n```\r\nResult:\r\n```/* 033 */         boolean inputadapter_isNull_0 = inputadapter_row_0.isNullAt(0);\r\n/* 034 */         ArrayData inputadapter_value_0 = inputadapter_isNull_0 ?\r\n/* 035 */         null : (inputadapter_row_0.getArray(0));\r\n/* 036 */\r\n/* 037 */         boolean filter_value_0 = true;\r\n/* 038 */\r\n/* 039 */         if (!(!inputadapter_isNull_0)) {\r\n/* 040 */           filter_value_0 = inputadapter_isNull_0;\r\n/* 041 */         }\r\n/* 042 */         if (!filter_value_0) continue;\r\n/* 043 */\r\n/* 044 */         ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);\r\n/* 045 */\r\n/* 046 */         boolean project_isNull_0 = inputadapter_isNull_0;\r\n/* 047 */         ArrayData project_value_0 = null;\r\n/* 048 */\r\n/* 049 */         if (!inputadapter_isNull_0) {\r\n/* 050 */           final int project_numElements_0 = inputadapter_value_0.numElements();\r\n/* 051 */           final Object[] project_internalRowArray_0 = new Object[project_numElements_0];\r\n/* 052 */           for (int z = 0; z < project_numElements_0; z++) {\r\n/* 053 */             project_internalRowArray_0[z] = new org.apache.spark.sql.catalyst.expressions.GenericInternalRow(new Object[]{inputadapter_value_0.getUTF8String(z), z});\r\n/* 054 */           }\r\n/* 055 */           project_value_0 = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_internalRowArray_0);\r\n/* 056 */\r\n/* 057 */         }\r\n```","created_at":"2018-04-21T12:32:20Z","updated_at":"2018-04-22T22:01:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0d72000e72fdc1fe495a2916584ef1faa61aa37c","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/21121/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/21121/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/21121/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/06348c3cf53d3631fd620b9e4d46b408a30b341e","head":{"label":"AbsaOSS:feature/array-api-zip_with_index-to-master","ref":"feature/array-api-zip_with_index-to-master","sha":"06348c3cf53d3631fd620b9e4d46b408a30b341e","user":{"login":"AbsaOSS","id":18133719,"avatar_url":"https://avatars0.githubusercontent.com/u/18133719?v=4","gravatar_id":"","url":"https://api.github.com/users/AbsaOSS","html_url":"https://github.com/AbsaOSS","followers_url":"https://api.github.com/users/AbsaOSS/followers","following_url":"https://api.github.com/users/AbsaOSS/following{/other_user}","gists_url":"https://api.github.com/users/AbsaOSS/gists{/gist_id}","starred_url":"https://api.github.com/users/AbsaOSS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbsaOSS/subscriptions","organizations_url":"https://api.github.com/users/AbsaOSS/orgs","repos_url":"https://api.github.com/users/AbsaOSS/repos","events_url":"https://api.github.com/users/AbsaOSS/events{/privacy}","received_events_url":"https://api.github.com/users/AbsaOSS/received_events","type":"Organization","site_admin":false},"repo":{"id":122981537,"name":"spark","full_name":"AbsaOSS/spark","owner":{"login":"AbsaOSS","id":18133719,"avatar_url":"https://avatars0.githubusercontent.com/u/18133719?v=4","gravatar_id":"","url":"https://api.github.com/users/AbsaOSS","html_url":"https://github.com/AbsaOSS","followers_url":"https://api.github.com/users/AbsaOSS/followers","following_url":"https://api.github.com/users/AbsaOSS/following{/other_user}","gists_url":"https://api.github.com/users/AbsaOSS/gists{/gist_id}","starred_url":"https://api.github.com/users/AbsaOSS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbsaOSS/subscriptions","organizations_url":"https://api.github.com/users/AbsaOSS/orgs","repos_url":"https://api.github.com/users/AbsaOSS/repos","events_url":"https://api.github.com/users/AbsaOSS/events{/privacy}","received_events_url":"https://api.github.com/users/AbsaOSS/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/AbsaOSS/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/AbsaOSS/spark","forks_url":"https://api.github.com/repos/AbsaOSS/spark/forks","keys_url":"https://api.github.com/repos/AbsaOSS/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/AbsaOSS/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/AbsaOSS/spark/teams","hooks_url":"https://api.github.com/repos/AbsaOSS/spark/hooks","issue_events_url":"https://api.github.com/repos/AbsaOSS/spark/issues/events{/number}","events_url":"https://api.github.com/repos/AbsaOSS/spark/events","assignees_url":"https://api.github.com/repos/AbsaOSS/spark/assignees{/user}","branches_url":"https://api.github.com/repos/AbsaOSS/spark/branches{/branch}","tags_url":"https://api.github.com/repos/AbsaOSS/spark/tags","blobs_url":"https://api.github.com/repos/AbsaOSS/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/AbsaOSS/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/AbsaOSS/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/AbsaOSS/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/AbsaOSS/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/AbsaOSS/spark/languages","stargazers_url":"https://api.github.com/repos/AbsaOSS/spark/stargazers","contributors_url":"https://api.github.com/repos/AbsaOSS/spark/contributors","subscribers_url":"https://api.github.com/repos/AbsaOSS/spark/subscribers","subscription_url":"https://api.github.com/repos/AbsaOSS/spark/subscription","commits_url":"https://api.github.com/repos/AbsaOSS/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/AbsaOSS/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/AbsaOSS/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/AbsaOSS/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/AbsaOSS/spark/contents/{+path}","compare_url":"https://api.github.com/repos/AbsaOSS/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/AbsaOSS/spark/merges","archive_url":"https://api.github.com/repos/AbsaOSS/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/AbsaOSS/spark/downloads","issues_url":"https://api.github.com/repos/AbsaOSS/spark/issues{/number}","pulls_url":"https://api.github.com/repos/AbsaOSS/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/AbsaOSS/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/AbsaOSS/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/AbsaOSS/spark/labels{/name}","releases_url":"https://api.github.com/repos/AbsaOSS/spark/releases{/id}","deployments_url":"https://api.github.com/repos/AbsaOSS/spark/deployments","created_at":"2018-02-26T14:30:33Z","updated_at":"2018-04-17T14:39:48Z","pushed_at":"2018-04-22T08:25:29Z","git_url":"git://github.com/AbsaOSS/spark.git","ssh_url":"git@github.com:AbsaOSS/spark.git","clone_url":"https://github.com/AbsaOSS/spark.git","svn_url":"https://github.com/AbsaOSS/spark","homepage":null,"size":268176,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"1d758dc73b54e802fdc92be204185fe7414e6553","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-04-22T18:42:42Z","pushed_at":"2018-04-22T21:40:20Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":268180,"stargazers_count":17030,"watchers_count":17030,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":15599,"mirror_url":null,"archived":false,"open_issues_count":514,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":15599,"open_issues":514,"watchers":17030,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/21121"},"html":{"href":"https://github.com/apache/spark/pull/21121"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/21121"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/21121/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/21121/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/21121/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/06348c3cf53d3631fd620b9e4d46b408a30b341e"}},"author_association":"NONE"}}
{"actor":{"display_login":"mn-mikke"}}