{"push_id":2338464630,"size":50,"distinct_size":50,"ref":"refs/heads/branch-2.3-banzaicloud","head":"11503c37f736cb94dede79392928faf42733f5b5","before":"cb1c13563238cf9862dec15fcaec1b06590ff231","commits":[{"sha":"68f3a070c728d0af95e9b5eec2c49be274b67a20","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23268][SQL][FOLLOWUP] Reorganize packages in data source V2\n\n## What changes were proposed in this pull request?\n\nThis is a followup of https://github.com/apache/spark/pull/20435.\n\nWhile reorganizing the packages for streaming data source v2, the top level stream read/write support interfaces should not be in the reader/writer package, but should be in the `sources.v2` package, to follow the `ReadSupport`, `WriteSupport`, etc.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20509 from cloud-fan/followup.\n\n(cherry picked from commit a75f927173632eee1316879447cb62c8cf30ae37)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/68f3a070c728d0af95e9b5eec2c49be274b67a20"},{"sha":"dfb16147791ff87342ff852105420a5eac5c553b","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-23186][SQL] Initialize DriverManager first before loading JDBC Drivers\n\n## What changes were proposed in this pull request?\n\nSince some JDBC Drivers have class initialization code to call `DriverManager`, we need to initialize `DriverManager` first in order to avoid potential executor-side **deadlock** situations like the following (or [STORM-2527](https://issues.apache.org/jira/browse/STORM-2527)).\n\n```\nThread 9587: (state = BLOCKED)\n - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) bci=0 (Compiled frame; information may be imprecise)\n - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) bci=85, line=62 (Compiled frame)\n - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) bci=5, line=45 (Compiled frame)\n - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) bci=79, line=423 (Compiled frame)\n - java.lang.Class.newInstance() bci=138, line=442 (Compiled frame)\n - java.util.ServiceLoader$LazyIterator.nextService() bci=119, line=380 (Interpreted frame)\n - java.util.ServiceLoader$LazyIterator.next() bci=11, line=404 (Interpreted frame)\n - java.util.ServiceLoader$1.next() bci=37, line=480 (Interpreted frame)\n - java.sql.DriverManager$2.run() bci=21, line=603 (Interpreted frame)\n - java.sql.DriverManager$2.run() bci=1, line=583 (Interpreted frame)\n - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) bci=0 (Compiled frame)\n - java.sql.DriverManager.loadInitialDrivers() bci=27, line=583 (Interpreted frame)\n - java.sql.DriverManager.<clinit>() bci=32, line=101 (Interpreted frame)\n - org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(java.lang.String, java.lang.Integer, java.lang.String, java.util.Properties) bci=12, line=98 (Interpreted frame)\n - org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(org.apache.hadoop.conf.Configuration, java.util.Properties) bci=22, line=57 (Interpreted frame)\n - org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(org.apache.hadoop.mapreduce.JobContext, org.apache.hadoop.conf.Configuration) bci=61, line=116 (Interpreted frame)\n - org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(org.apache.hadoop.mapreduce.InputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext) bci=10, line=71 (Interpreted frame)\n - org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(org.apache.spark.rdd.NewHadoopRDD, org.apache.spark.Partition, org.apache.spark.TaskContext) bci=233, line=156 (Interpreted frame)\n\nThread 9170: (state = BLOCKED)\n - org.apache.phoenix.jdbc.PhoenixDriver.<clinit>() bci=35, line=125 (Interpreted frame)\n - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) bci=0 (Compiled frame)\n - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) bci=85, line=62 (Compiled frame)\n - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) bci=5, line=45 (Compiled frame)\n - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) bci=79, line=423 (Compiled frame)\n - java.lang.Class.newInstance() bci=138, line=442 (Compiled frame)\n - org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(java.lang.String) bci=89, line=46 (Interpreted frame)\n - org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply() bci=7, line=53 (Interpreted frame)\n - org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply() bci=1, line=52 (Interpreted frame)\n - org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.<init>(org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD, org.apache.spark.Partition, org.apache.spark.TaskContext) bci=81, line=347 (Interpreted frame)\n - org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(org.apache.spark.Partition, org.apache.spark.TaskContext) bci=7, line=339 (Interpreted frame)\n```\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #20359 from dongjoon-hyun/SPARK-23186.\n\n(cherry picked from commit 8cbcc33876c773722163b2259644037bbb259bd1)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/dfb16147791ff87342ff852105420a5eac5c553b"},{"sha":"196304a3a8ed15fd018e9c7b441954d17bd60124","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-23328][PYTHON] Disallow default value None in na.replace/replace when 'to_replace' is not a dictionary\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to disallow default value None when 'to_replace' is not a dictionary.\n\nIt seems weird we set the default value of `value` to `None` and we ended up allowing the case as below:\n\n```python\n>>> df.show()\n```\n```\n+----+------+-----+\n| age|height| name|\n+----+------+-----+\n|  10|    80|Alice|\n...\n```\n\n```python\n>>> df.na.replace('Alice').show()\n```\n```\n+----+------+----+\n| age|height|name|\n+----+------+----+\n|  10|    80|null|\n...\n```\n\n**After**\n\nThis PR targets to disallow the case above:\n\n```python\n>>> df.na.replace('Alice').show()\n```\n```\n...\nTypeError: value is required when to_replace is not a dictionary.\n```\n\nwhile we still allow when `to_replace` is a dictionary:\n\n```python\n>>> df.na.replace({'Alice': None}).show()\n```\n```\n+----+------+----+\n| age|height|name|\n+----+------+----+\n|  10|    80|null|\n...\n```\n\n## How was this patch tested?\n\nManually tested, tests were added in `python/pyspark/sql/tests.py` and doctests were fixed.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #20499 from HyukjinKwon/SPARK-19454-followup.\n\n(cherry picked from commit 4b4ee2601079f12f8f410a38d2081793cbdedc14)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/196304a3a8ed15fd018e9c7b441954d17bd60124"},{"sha":"08eb95f609f5d356c89dedcefa768b12a7a8b96c","author":{"name":"liuxian","email":"d698299a02cb9d6095b897cb57e6057a3afd73b5@zte.com.cn"},"message":"[SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result\n\n## What changes were proposed in this pull request?\nIn the `checkIndexAndDataFile`,the `blocks` is the ` Int` type,  when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result.\nIn fact, `blocks` is actually the number of partitions.\n\n## How was this patch tested?\nManual test\n\nAuthor: liuxian <liu.xian3@zte.com.cn>\n\nCloses #20544 from 10110346/overflow.\n\n(cherry picked from commit f77270b8811bbd8956d0c08fa556265d2c5ee20e)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/08eb95f609f5d356c89dedcefa768b12a7a8b96c"},{"sha":"49771ac8da8e68e8412d9f5d181953eaf0de7973","author":{"name":"Jacek Laskowski","email":"2ae8b6a7c53eb98fbee4a4146dda0b032e92cea4@japila.pl"},"message":"[MINOR][HIVE] Typo fixes\n\n## What changes were proposed in this pull request?\n\nTypo fixes (with expanding a Hive property)\n\n## How was this patch tested?\n\nlocal build. Awaiting Jenkins\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #20550 from jaceklaskowski/hiveutils-typos.\n\n(cherry picked from commit 557938e2839afce26a10a849a2a4be8fc4580427)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/49771ac8da8e68e8412d9f5d181953eaf0de7973"},{"sha":"f3a9a7f6b6eac4421bd74ff73a74105982604ce6","author":{"name":"Feng Liu","email":"053548b88f9b9bede03e83ce47a784e6fc89e414@databricks.com"},"message":"[SPARK-23275][SQL] fix the thread leaking in hive/tests\n\n## What changes were proposed in this pull request?\n\nThis is a follow up of https://github.com/apache/spark/pull/20441.\n\nThe two lines actually can trigger the hive metastore bug: https://issues.apache.org/jira/browse/HIVE-16844\n\nThe two configs are not in the default `ObjectStore` properties, so any run hive commands after these two lines will set the `propsChanged` flag in the `ObjectStore.setConf` and then cause thread leaks.\n\nI don't think the two lines are very useful. They can be removed safely.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Feng Liu <fengliu@databricks.com>\n\nCloses #20562 from liufengdb/fix-omm.\n\n(cherry picked from commit 6d7c38330e68c7beb10f54eee8b4f607ee3c4136)\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/f3a9a7f6b6eac4421bd74ff73a74105982604ce6"},{"sha":"b7571b9bfcced2e08f87e815c2ea9474bfd5fe2a","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-23360][SQL][PYTHON] Get local timezone from environment via pytz, or dateutil.\n\n## What changes were proposed in this pull request?\n\nCurrently we use `tzlocal()` to get Python local timezone, but it sometimes causes unexpected behavior.\nI changed the way to get Python local timezone to use pytz if the timezone is specified in environment variable, or timezone file via dateutil .\n\n## How was this patch tested?\n\nAdded a test and existing tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20559 from ueshin/issues/SPARK-23360/master.\n\n(cherry picked from commit 97a224a855c4410b2dfb9c0bcc6aae583bd28e92)\nSigned-off-by: hyukjinkwon <gurwls223@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/b7571b9bfcced2e08f87e815c2ea9474bfd5fe2a"},{"sha":"9fa7b0e107c283557648160195ce179077752e4c","author":{"name":"Li Jin","email":"2b7a7e99063afb1bbcc44f34d0b2f3647bc1a1ed@gmail.com"},"message":"[SPARK-23314][PYTHON] Add ambiguous=False when localizing tz-naive timestamps in Arrow codepath to deal with dst\n\n## What changes were proposed in this pull request?\nWhen tz_localize a tz-naive timetamp, pandas will throw exception if the timestamp is during daylight saving time period, e.g., `2015-11-01 01:30:00`. This PR fixes this issue by setting `ambiguous=False` when calling tz_localize, which is the same default behavior of pytz.\n\n## How was this patch tested?\nAdd `test_timestamp_dst`\n\nAuthor: Li Jin <ice.xelloss@gmail.com>\n\nCloses #20537 from icexelloss/SPARK-23314.\n\n(cherry picked from commit a34fce19bc0ee5a7e36c6ecba75d2aeb70fdcbc7)\nSigned-off-by: hyukjinkwon <gurwls223@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/9fa7b0e107c283557648160195ce179077752e4c"},{"sha":"8875e47cec01ae8da4ffb855409b54089e1016fb","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-23387][SQL][PYTHON][TEST][BRANCH-2.3] Backport assertPandasEqual to branch-2.3.\n\n## What changes were proposed in this pull request?\n\nWhen backporting a pr with tests using `assertPandasEqual` from master to branch-2.3, the tests fail because `assertPandasEqual` doesn't exist in branch-2.3.\nWe should backport `assertPandasEqual` to branch-2.3 to avoid the failures.\n\n## How was this patch tested?\n\nModified tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20577 from ueshin/issues/SPARK-23387/branch-2.3.","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/8875e47cec01ae8da4ffb855409b54089e1016fb"},{"sha":"7e2a2b33c0664b3638a1428688b28f68323994c1","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23376][SQL] creating UnsafeKVExternalSorter with BytesToBytesMap may fail\n\n## What changes were proposed in this pull request?\n\nThis is a long-standing bug in `UnsafeKVExternalSorter` and was reported in the dev list multiple times.\n\nWhen creating `UnsafeKVExternalSorter` with `BytesToBytesMap`, we need to create a `UnsafeInMemorySorter` to sort the data in `BytesToBytesMap`. The data format of the sorter and the map is same, so no data movement is required. However, both the sorter and the map need a point array for some bookkeeping work.\n\nThere is an optimization in `UnsafeKVExternalSorter`: reuse the point array between the sorter and the map, to avoid an extra memory allocation. This sounds like a reasonable optimization, the length of the `BytesToBytesMap` point array is at least 4 times larger than the number of keys(to avoid hash collision, the hash table size should be at least 2 times larger than the number of keys, and each key occupies 2 slots). `UnsafeInMemorySorter` needs the pointer array size to be 4 times of the number of entries, so we are safe to reuse the point array.\n\nHowever, the number of keys of the map doesn't equal to the number of entries in the map, because `BytesToBytesMap` supports duplicated keys. This breaks the assumption of the above optimization and we may run out of space when inserting data into the sorter, and hit error\n```\njava.lang.IllegalStateException: There is no space for new record\n   at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:239)\n   at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:149)\n...\n```\n\nThis PR fixes this bug by creating a new point array if the existing one is not big enough.\n\n## How was this patch tested?\n\na new test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20561 from cloud-fan/bug.\n\n(cherry picked from commit 4bbd7443ebb005f81ed6bc39849940ac8db3b3cc)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/7e2a2b33c0664b3638a1428688b28f68323994c1"},{"sha":"79e8650cccb00c7886efba6dd691d9733084cb81","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7\n\n## What changes were proposed in this pull request?\n\nThis test only fails with sbt on Hadoop 2.7, I can't reproduce it locally, but here is my speculation by looking at the code:\n1. FileSystem.delete doesn't delete the directory entirely, somehow we can still open the file as a 0-length empty file.(just speculation)\n2. ORC intentionally allow empty files, and the reader fails during reading without closing the file stream.\n\nThis PR improves the test to make sure all files are deleted and can't be opened.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20584 from cloud-fan/flaky-test.\n\n(cherry picked from commit 6efd5d117e98074d1b16a5c991fbd38df9aa196e)\nSigned-off-by: Sameer Agarwal <sameerag@apache.org>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/79e8650cccb00c7886efba6dd691d9733084cb81"},{"sha":"1e3118c2ee0fe7d2c59cb3e2055709bb2809a6db","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-22977][SQL] fix web UI SQL tab for CTAS\n\n## What changes were proposed in this pull request?\n\nThis is a regression in Spark 2.3.\n\nIn Spark 2.2, we have a fragile UI support for SQL data writing commands. We only track the input query plan of `FileFormatWriter` and display its metrics. This is not ideal because we don't know who triggered the writing(can be table insertion, CTAS, etc.), but it's still useful to see the metrics of the input query.\n\nIn Spark 2.3, we introduced a new mechanism: `DataWritigCommand`, to fix the UI issue entirely. Now these writing commands have real children, and we don't need to hack into the `FileFormatWriter` for the UI. This also helps with `explain`, now `explain` can show the physical plan of the input query, while in 2.2 the physical writing plan is simply `ExecutedCommandExec` and it has no child.\n\nHowever there is a regression in CTAS. CTAS commands don't extend `DataWritigCommand`, and we don't have the UI hack in `FileFormatWriter` anymore, so the UI for CTAS is just an empty node. See https://issues.apache.org/jira/browse/SPARK-22977 for more information about this UI issue.\n\nTo fix it, we should apply the `DataWritigCommand` mechanism to CTAS commands.\n\nTODO: In the future, we should refactor this part and create some physical layer code pieces for data writing, and reuse them in different writing commands. We should have different logical nodes for different operators, even some of them share some same logic, e.g. CTAS, CREATE TABLE, INSERT TABLE. Internally we can share the same physical logic.\n\n## How was this patch tested?\n\nmanually tested.\nFor data source table\n<img width=\"644\" alt=\"1\" src=\"https://user-images.githubusercontent.com/3182036/35874155-bdffab28-0ba6-11e8-94a8-e32e106ba069.png\">\nFor hive table\n<img width=\"666\" alt=\"2\" src=\"https://user-images.githubusercontent.com/3182036/35874161-c437e2a8-0ba6-11e8-98ed-7930f01432c5.png\">\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20521 from cloud-fan/UI.\n\n(cherry picked from commit 0e2c266de7189473177f45aa68ea6a45c7e47ec3)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/1e3118c2ee0fe7d2c59cb3e2055709bb2809a6db"},{"sha":"d31c4ae7ba734356c849347b9a7b448da9a5a9a1","author":{"name":"liuxian","email":"d698299a02cb9d6095b897cb57e6057a3afd73b5@zte.com.cn"},"message":"[SPARK-23391][CORE] It may lead to overflow for some integer multiplication\n\n## What changes were proposed in this pull request?\nIn the `getBlockData`,`blockId.reduceId` is the `Int` type, when it is greater than 2^28, `blockId.reduceId*8` will overflow\nIn the `decompress0`, `len` and  `unitSize` are  Int type, so `len * unitSize` may lead to  overflow\n## How was this patch tested?\nN/A\n\nAuthor: liuxian <liu.xian3@zte.com.cn>\n\nCloses #20581 from 10110346/overflow2.\n\n(cherry picked from commit 4a4dd4f36f65410ef5c87f7b61a960373f044e61)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/d31c4ae7ba734356c849347b9a7b448da9a5a9a1"},{"sha":"89f6fcbafcfb0a7aeb897fba6036cb085bd35121","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@apache.org"},"message":"Preparing Spark release v2.3.0-rc3","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/89f6fcbafcfb0a7aeb897fba6036cb085bd35121"},{"sha":"70be6038df38d5e80af8565120eedd8242c5a7c5","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@apache.org"},"message":"Preparing development version 2.3.1-SNAPSHOT","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/70be6038df38d5e80af8565120eedd8242c5a7c5"},{"sha":"4e138207ebb11a08393c15e5e39f46a5dc1e7c66","author":{"name":"James Thompson","email":"b88ddfc258f53c71f9a83629eb22bddbf8bcefa6@users.noreply.github.com"},"message":"[SPARK-23388][SQL] Support for Parquet Binary DecimalType in VectorizedColumnReader\n\n## What changes were proposed in this pull request?\n\nRe-add support for parquet binary DecimalType in VectorizedColumnReader\n\n## How was this patch tested?\n\nExisting test suite\n\nAuthor: James Thompson <jamesthomp@users.noreply.github.com>\n\nCloses #20580 from jamesthomp/jt/add-back-binary-decimal.\n\n(cherry picked from commit 5bb11411aec18b8d623e54caba5397d7cb8e89f0)\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/4e138207ebb11a08393c15e5e39f46a5dc1e7c66"},{"sha":"9632c461e6931a1a4d05684d0f62ee36f9e90b77","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-22002][SQL][FOLLOWUP][TEST] Add a test to check if the original schema doesn't have metadata.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up pr of #19231 which modified the behavior to remove metadata from JDBC table schema.\nThis pr adds a test to check if the schema doesn't have metadata.\n\n## How was this patch tested?\n\nAdded a test and existing tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20585 from ueshin/issues/SPARK-22002/fup1.\n\n(cherry picked from commit 0c66fe4f22f8af4932893134bb0fd56f00fabeae)\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/9632c461e6931a1a4d05684d0f62ee36f9e90b77"},{"sha":"2b80571e215d56d15c59f0fc5db053569a79efae","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-23313][DOC] Add a migration guide for ORC\n\n## What changes were proposed in this pull request?\n\nThis PR adds a migration guide documentation for ORC.\n\n![orc-guide](https://user-images.githubusercontent.com/9700541/36123859-ec165cae-1002-11e8-90b7-7313be7a81a5.png)\n\n## How was this patch tested?\n\nN/A.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #20484 from dongjoon-hyun/SPARK-23313.\n\n(cherry picked from commit 6cb59708c70c03696c772fbb5d158eed57fe67d4)\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/2b80571e215d56d15c59f0fc5db053569a79efae"},{"sha":"befb22de81aad41673eec9dba7585b80c6cb2564","author":{"name":"sychen","email":"5eba600a96398ad4a987521de2266b1f5e0810a3@ctrip.com"},"message":"[SPARK-23230][SQL] When hive.default.fileformat is other kinds of file types, create textfile table cause a serde error\n\nWhen hive.default.fileformat is other kinds of file types, create textfile table cause a serde error.\nWe should take the default type of textfile and sequencefile both as org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.\n\n```\nset hive.default.fileformat=orc;\ncreate table tbl( i string ) stored as textfile;\ndesc formatted tbl;\n\nSerde Library org.apache.hadoop.hive.ql.io.orc.OrcSerde\nInputFormat  org.apache.hadoop.mapred.TextInputFormat\nOutputFormat  org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\n```\n\nAuthor: sychen <sychen@ctrip.com>\n\nCloses #20406 from cxzl25/default_serde.\n\n(cherry picked from commit 4104b68e958cd13975567a96541dac7cccd8195c)\nSigned-off-by: gatorsmile <gatorsmile@gmail.com>","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/befb22de81aad41673eec9dba7585b80c6cb2564"},{"sha":"43f5e40679f771326b2ee72f14cf1ab0ed2ad692","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-23352][PYTHON][BRANCH-2.3] Explicitly specify supported types in Pandas UDFs\n\n## What changes were proposed in this pull request?\n\nThis PR backports https://github.com/apache/spark/pull/20531:\n\nIt explicitly specifies supported types in Pandas UDFs.\nThe main change here is to add a deduplicated and explicit type checking in `returnType` ahead with documenting this; however, it happened to fix multiple things.\n\n1. Currently, we don't support `BinaryType` in Pandas UDFs, for example, see:\n\n    ```python\n    from pyspark.sql.functions import pandas_udf\n    pudf = pandas_udf(lambda x: x, \"binary\")\n    df = spark.createDataFrame([[bytearray(1)]])\n    df.select(pudf(\"_1\")).show()\n    ```\n    ```\n    ...\n    TypeError: Unsupported type in conversion to Arrow: BinaryType\n    ```\n\n    We can document this behaviour for its guide.\n\n2. Since we can check the return type ahead, we can fail fast before actual execution.\n\n    ```python\n    # we can fail fast at this stage because we know the schema ahead\n    pandas_udf(lambda x: x, BinaryType())\n    ```\n\n## How was this patch tested?\n\nManually tested and unit tests for `BinaryType` and `ArrayType(...)` were added.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #20588 from HyukjinKwon/PR_TOOL_PICK_PR_20531_BRANCH-2.3.","distinct":true,"url":"https://api.github.com/repos/banzaicloud/spark/commits/43f5e40679f771326b2ee72f14cf1ab0ed2ad692"}]}
{"actor":{"display_login":"stoader"}}