{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21282","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21282/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21282/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21282/events","html_url":"https://github.com/apache/spark/pull/21282","id":321660890,"number":21282,"title":"[SPARK-23934][SQL] Adding map_from_entries function","user":{"login":"mn-mikke","id":3674621,"avatar_url":"https://avatars3.githubusercontent.com/u/3674621?v=4","gravatar_id":"","url":"https://api.github.com/users/mn-mikke","html_url":"https://github.com/mn-mikke","followers_url":"https://api.github.com/users/mn-mikke/followers","following_url":"https://api.github.com/users/mn-mikke/following{/other_user}","gists_url":"https://api.github.com/users/mn-mikke/gists{/gist_id}","starred_url":"https://api.github.com/users/mn-mikke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mn-mikke/subscriptions","organizations_url":"https://api.github.com/users/mn-mikke/orgs","repos_url":"https://api.github.com/users/mn-mikke/repos","events_url":"https://api.github.com/users/mn-mikke/events{/privacy}","received_events_url":"https://api.github.com/users/mn-mikke/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2018-05-09T17:36:05Z","updated_at":"2018-05-18T06:25:52Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21282","html_url":"https://github.com/apache/spark/pull/21282","diff_url":"https://github.com/apache/spark/pull/21282.diff","patch_url":"https://github.com/apache/spark/pull/21282.patch"},"body":"## What changes were proposed in this pull request?\r\nThe PR adds the `map_from_entries` function that returns a map created from the given array of entries.\r\n\r\n## How was this patch tested?\r\nNew tests added into:\r\n- `CollectionExpressionSuite`\r\n- `DataFrameFunctionSuite`\r\n\r\n## Note\r\nThe proposed behavior doesn't allow duplicated keys in contrast with `map` function.\r\n\r\n## CodeGen Examples\r\n### Primitive-type Keys and Values\r\n```\r\nval idf = Seq(\r\n  Seq((1, 10), (2, 20), (3, 10)),\r\n  Seq((1, 10), null, (2, 20))\r\n).toDF(\"a\")\r\nidf.filter('a.isNotNull).select(map_from_entries('a)).debugCodegen\r\n```\r\nResult:\r\n```\r\n/* 042 */         boolean project_isNull_0 = false;\r\n/* 043 */\r\n/* 044 */         MapData project_value_0 = null;\r\n/* 045 */\r\n/* 046 */         final int project_length_0 = inputadapter_value_0.numElements();\r\n/* 047 */\r\n/* 048 */         int project_numEntries_0 = 0;\r\n/* 049 */         for (int project_idx_4 = 0; project_idx_4 < project_length_0; project_idx_4++) {\r\n/* 050 */           if (!inputadapter_value_0.isNullAt(project_idx_4)) project_numEntries_0++;\r\n/* 051 */         }\r\n/* 052 */\r\n/* 053 */         final org.apache.spark.util.collection.OpenHashSet$mcI$sp project_keySet_0 = new org.apache.spark.util.collection.OpenHashSet$mcI$sp(scala.reflect.ClassTag$.MODULE$.Int());\r\n/* 054 */\r\n/* 055 */         final long project_keySectionSize_0 = UnsafeArrayData.calculateSizeOfUnderlyingByteArray(project_numEntries_0, 4);\r\n/* 056 */         final long project_valueSectionSize_0 = UnsafeArrayData.calculateSizeOfUnderlyingByteArray(project_numEntries_0, 4);\r\n/* 057 */         final long project_byteArraySize_0 = 8 + project_keySectionSize_0 + project_valueSectionSize_0;\r\n/* 058 */         if (project_byteArraySize_0 > 2147483632) {\r\n/* 059 */           final Object[] project_keys_0 = new Object[project_numEntries_0];\r\n/* 060 */           final Object[] project_values_0 = new Object[project_numEntries_0];\r\n/* 061 */\r\n/* 062 */           for (int project_idx_2 = 0, project_idx_3 = 0; project_idx_2 < project_length_0; project_idx_2++) {\r\n/* 063 */             if (inputadapter_value_0.isNullAt(project_idx_2)) continue;\r\n/* 064 */             InternalRow project_entry_1 = inputadapter_value_0.getStruct(project_idx_2, 2);\r\n/* 065 */\r\n/* 066 */             int project_key_1 = project_entry_1.getInt(0);\r\n/* 067 */             if (project_keySet_0.contains(project_key_1)) {\r\n/* 068 */               throw new RuntimeException(\r\n/* 069 */                 \"The first field from a struct (key) can't produce duplicates.\");\r\n/* 070 */             }\r\n/* 071 */             project_keySet_0.add(project_key_1);\r\n/* 072 */             project_keys_0[project_idx_3] = project_key_1;\r\n/* 073 */             project_values_0[project_idx_3] = project_entry_1.getInt(1);\r\n/* 074 */             project_idx_3++;\r\n/* 075 */           }\r\n/* 076 */\r\n/* 077 */           project_value_0 = org.apache.spark.sql.catalyst.util.ArrayBasedMapData.apply(project_keys_0, project_values_0);\r\n/* 078 */\r\n/* 079 */         } else {\r\n/* 080 */           final byte[] project_byteArray_0 = new byte[(int)project_byteArraySize_0];\r\n/* 081 */           UnsafeMapData project_unsafeMapData_0 = new UnsafeMapData();\r\n/* 082 */           Platform.putLong(project_byteArray_0, 16, project_keySectionSize_0);\r\n/* 083 */           Platform.putLong(project_byteArray_0, 24, project_numEntries_0);\r\n/* 084 */           Platform.putLong(project_byteArray_0, 24 + project_keySectionSize_0, project_numEntries_0);\r\n/* 085 */           project_unsafeMapData_0.pointTo(project_byteArray_0, 16, (int)project_byteArraySize_0);\r\n/* 086 */           ArrayData project_keyArrayData_0 = project_unsafeMapData_0.keyArray();\r\n/* 087 */           ArrayData project_valueArrayData_0 = project_unsafeMapData_0.valueArray();\r\n/* 088 */\r\n/* 089 */           for (int project_idx_0 = 0, project_idx_1 = 0; project_idx_0 < project_length_0; project_idx_0++) {\r\n/* 090 */             if (inputadapter_value_0.isNullAt(project_idx_0)) continue;\r\n/* 091 */             InternalRow project_entry_0 = inputadapter_value_0.getStruct(project_idx_0, 2);\r\n/* 092 */\r\n/* 093 */             int project_key_0 = project_entry_0.getInt(0);\r\n/* 094 */             if (project_keySet_0.contains(project_key_0)) {\r\n/* 095 */               throw new RuntimeException(\r\n/* 096 */                 \"The first field from a struct (key) can't produce duplicates.\");\r\n/* 097 */             }\r\n/* 098 */             project_keySet_0.add(project_key_0);\r\n/* 099 */             project_keyArrayData_0.setInt(project_idx_1, project_key_0);\r\n/* 100 */             project_valueArrayData_0.setInt(project_idx_1, project_entry_0.getInt(1));\r\n/* 101 */             project_idx_1++;\r\n/* 102 */           }\r\n/* 103 */\r\n/* 104 */           project_value_0 = project_unsafeMapData_0;\r\n/* 105 */         }\r\n```\r\n### Non-primitive-type Keys and Values\r\n```\r\nval sdf = Seq(\r\n  Seq((\"a\", null), (\"b\", \"bb\"), (\"c\", \"aa\")),\r\n  Seq((\"a\", \"aa\"), null, (null, \"bb\"))\r\n).toDF(\"a\")\r\nsdf.filter('a.isNotNull).select(map_from_entries('a)).debugCodegen\r\n```\r\nResult:\r\n```\r\n/* 042 */         boolean project_isNull_0 = false;\r\n/* 043 */\r\n/* 044 */         MapData project_value_0 = null;\r\n/* 045 */\r\n/* 046 */         final int project_length_0 = inputadapter_value_0.numElements();\r\n/* 047 */\r\n/* 048 */         int project_numEntries_0 = 0;\r\n/* 049 */         for (int project_idx_2 = 0; project_idx_2 < project_length_0; project_idx_2++) {\r\n/* 050 */           if (!inputadapter_value_0.isNullAt(project_idx_2)) project_numEntries_0++;\r\n/* 051 */         }\r\n/* 052 */\r\n/* 053 */         final org.apache.spark.util.collection.OpenHashSet project_keySet_0 = new org.apache.spark.util.collection.OpenHashSet(scala.reflect.ClassTag$.MODULE$.Object());\r\n/* 054 */\r\n/* 055 */         final Object[] project_keys_0 = new Object[project_numEntries_0];\r\n/* 056 */         final Object[] project_values_0 = new Object[project_numEntries_0];\r\n/* 057 */\r\n/* 058 */         for (int project_idx_0 = 0, project_idx_1 = 0; project_idx_0 < project_length_0; project_idx_0++) {\r\n/* 059 */           if (inputadapter_value_0.isNullAt(project_idx_0)) continue;\r\n/* 060 */           InternalRow project_entry_0 = inputadapter_value_0.getStruct(project_idx_0, 2);\r\n/* 061 */\r\n/* 062 */           if (project_entry_0.isNullAt(0)) {\r\n/* 063 */             throw new RuntimeException(\"The first field from a struct (key) can't be null.\");\r\n/* 064 */           }\r\n/* 065 */\r\n/* 066 */           UTF8String project_key_0 = project_entry_0.getUTF8String(0);\r\n/* 067 */           if (project_keySet_0.contains(project_key_0)) {\r\n/* 068 */             throw new RuntimeException(\r\n/* 069 */               \"The first field from a struct (key) can't produce duplicates.\");\r\n/* 070 */           }\r\n/* 071 */           project_keySet_0.add(project_key_0);\r\n/* 072 */           project_keys_0[project_idx_1] = project_key_0;\r\n/* 073 */           project_values_0[project_idx_1] = project_entry_0.getUTF8String(1);\r\n/* 074 */           project_idx_1++;\r\n/* 075 */         }\r\n/* 076 */\r\n/* 077 */         project_value_0 = org.apache.spark.sql.catalyst.util.ArrayBasedMapData.apply(project_keys_0, project_values_0);\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/390107354","html_url":"https://github.com/apache/spark/pull/21282#issuecomment-390107354","issue_url":"https://api.github.com/repos/apache/spark/issues/21282","id":390107354,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-05-18T06:25:52Z","updated_at":"2018-05-18T06:25:52Z","author_association":"NONE","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90773/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}