{"push_id":2305380464,"size":25,"distinct_size":1,"ref":"refs/heads/SPARK-23300","head":"fe2943e00d78aff2fea41871b2d8091d4393bbef","before":"4f77fc354febc51aa8c55ff6e955caa825fb27ed","commits":[{"sha":"8bb70b068ea782e799e45238fcb093a6acb0fc9f","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues.\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up of #20450 which broke lint-java checks.\nThis pr fixes the lint-java issues.\n\n```\n[ERROR] src/main/java/org/apache/spark/sql/vectorized/ColumnVector.java:[20,8] (imports) UnusedImports: Unused import - org.apache.spark.sql.catalyst.util.MapData.\n[ERROR] src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java:[21,8] (imports) UnusedImports: Unused import - org.apache.spark.sql.catalyst.util.MapData.\n[ERROR] src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:[22,8] (imports) UnusedImports: Unused import - org.apache.spark.sql.catalyst.util.MapData.\n```\n\n## How was this patch tested?\n\nChecked manually in my local environment.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20468 from ueshin/issues/SPARK-23280/fup1.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/8bb70b068ea782e799e45238fcb093a6acb0fc9f"},{"sha":"89e8d556b93d1bf1b28fe153fd284f154045b0ee","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@databricks.com"},"message":"[SPARK-23280][SQL][FOLLOWUP] Enable `MutableColumnarRow.getMap()`.\n\n## What changes were proposed in this pull request?\n\nThis is a followup pr of #20450.\nWe should've enabled `MutableColumnarRow.getMap()` as well.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #20471 from ueshin/issues/SPARK-23280/fup2.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/89e8d556b93d1bf1b28fe153fd284f154045b0ee"},{"sha":"ffbca84519011a747e0552632e88f5e4956e493d","author":{"name":"Wang Gengliang","email":"0e8d6d10d3df7d3ddc88549e22e9db696f9a574b@gmail.com"},"message":"[SPARK-23202][SQL] Add new API in DataSourceWriter: onDataWriterCommit\n\n## What changes were proposed in this pull request?\n\nThe current DataSourceWriter API makes it hard to implement `onTaskCommit(taskCommit: TaskCommitMessage)` in `FileCommitProtocol`.\nIn general, on receiving commit message, driver can start processing messages(e.g. persist messages into files) before all the messages are collected.\n\nThe proposal to add a new API:\n`add(WriterCommitMessage message)`:  Handles a commit message on receiving from a successful data writer.\n\nThis should make the whole API of DataSourceWriter compatible with `FileCommitProtocol`, and more flexible.\n\nThere was another radical attempt in #20386.  This one should be more reasonable.\n\n## How was this patch tested?\n\nUnit test\n\nAuthor: Wang Gengliang <ltnwgl@gmail.com>\n\nCloses #20454 from gengliangwang/write_api.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/ffbca84519011a747e0552632e88f5e4956e493d"},{"sha":"ec63e2d0743a4f75e1cce21d0fe2b54407a86a4a","author":{"name":"Shixiong Zhu","email":"c8f54caa5df4e9592a01136ca859e0af6b5a2c2a@gmail.com"},"message":"[SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully\n\n## What changes were proposed in this pull request?\n\n`channel.write(buf)` may not write the whole buffer since the underlying channel is a FileChannel, we should retry until the whole buffer is written.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <zsxwing@gmail.com>\n\nCloses #20461 from zsxwing/SPARK-23289.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/ec63e2d0743a4f75e1cce21d0fe2b54407a86a4a"},{"sha":"f051f834036e63d5e480d86440ce39924f979e82","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-13983][SQL] Fix HiveThriftServer2 can not get \"--hiveconf\" and ''--hivevar\" variables since 2.0\n\n## What changes were proposed in this pull request?\n\n`--hiveconf` and `--hivevar` variables no longer work since Spark 2.0. The `spark-sql` client has fixed by [SPARK-15730](https://issues.apache.org/jira/browse/SPARK-15730) and [SPARK-18086](https://issues.apache.org/jira/browse/SPARK-18086). but `beeline`/[`Spark SQL HiveThriftServer2`](https://github.com/apache/spark/blob/v2.1.1/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala) is still broken. This pull request fix it.\n\nThis pull request works for both `JDBC client` and `beeline`.\n\n## How was this patch tested?\n\nunit tests for  `JDBC client`\nmanual tests for `beeline`:\n```\ngit checkout origin/pr/17886\n\ndev/make-distribution.sh --mvn mvn  --tgz -Phive -Phive-thriftserver -Phadoop-2.6 -DskipTests\n\ntar -zxf spark-2.3.0-SNAPSHOT-bin-2.6.5.tgz && cd spark-2.3.0-SNAPSHOT-bin-2.6.5\n\nsbin/start-thriftserver.sh\n```\n```\ncat <<EOF > test.sql\nselect '\\${a}', '\\${b}';\nEOF\n\nbeeline -u jdbc:hive2://localhost:10000 --hiveconf a=avalue --hivevar b=bvalue -f test.sql\n\n```\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\n\nCloses #17886 from wangyum/SPARK-13983-dev.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/f051f834036e63d5e480d86440ce39924f979e82"},{"sha":"73da3b6968630d9e2cafc742ccb6d4eb54957df4","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23293][SQL] fix data source v2 self join\n\n## What changes were proposed in this pull request?\n\n`DataSourceV2Relation` should extend `MultiInstanceRelation`, to take care of self-join.\n\n## How was this patch tested?\n\na new test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20466 from cloud-fan/dsv2-selfjoin.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/73da3b6968630d9e2cafc742ccb6d4eb54957df4"},{"sha":"4bcfdefb9f6d5ba88335953683a1dabbee83e9ea","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[INFRA] Close stale PRs.\n\nCloses #20334\nCloses #20262","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/4bcfdefb9f6d5ba88335953683a1dabbee83e9ea"},{"sha":"032c11b83f0d276bf8085992229b8c598f02798a","author":{"name":"Gera Shegalov","email":"230943984b795213517b61648c155c561d95d48b@apache.org"},"message":"[SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic\n\n## What changes were proposed in this pull request?\n\nInclude stacktrace in the diagnostics message upon abnormal unregister from RM\n\n## How was this patch tested?\nTested with a failing job, and confirmed a stacktrace in the client output and YARN webUI.\n\nAuthor: Gera Shegalov <gera@apache.org>\n\nCloses #20470 from gerashegalov/gera/stacktrace-diagnostics.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/032c11b83f0d276bf8085992229b8c598f02798a"},{"sha":"90848d507457d30abb36e3ba07618dfc87c34cd6","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot\n\n## What changes were proposed in this pull request?\n\nFor some ColumnVector get APIs such as getDecimal, getBinary, getStruct, getArray, getInterval, getUTF8String, we should clearly document their behaviors when accessing null slot. They should return null in this case. Then we can remove null checks from the places using above APIs.\n\nFor the APIs of primitive values like getInt, getInts, etc., this also documents their behaviors when accessing null slots. Their returning values are undefined and can be anything.\n\n## How was this patch tested?\n\nAdded tests into `ColumnarBatchSuite`.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #20455 from viirya/SPARK-23272-followup.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/90848d507457d30abb36e3ba07618dfc87c34cd6"},{"sha":"969eda4a02faa7ca6cf3aff5cd10e6d51026b845","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-23020][CORE] Fix another race in the in-process launcher test.\n\nFirst the bad news: there's an unfixable race in the launcher code.\n(By unfixable I mean it would take a lot more effort than this change\nto fix it.) The good news is that it should only affect super short\nlived applications, such as the one run by the flaky test, so it's\npossible to work around it in our test.\n\nThe fix also uncovered an issue with the recently added \"closeAndWait()\"\nmethod; closing the connection would still possibly cause data loss,\nso this change waits a while for the connection to finish itself, and\ncloses the socket if that times out. The existing connection timeout\nis reused so that if desired it's possible to control how long to wait.\n\nAs part of that I also restored the old behavior that disconnect() would\nforce a disconnection from the child app; the \"wait for data to arrive\"\napproach is only taken when disposing of the handle.\n\nI tested this by inserting a bunch of sleeps in the test and the socket\nhandling code in the launcher library; with those I was able to reproduce\nthe error from the jenkins jobs. With the changes, even with all the\nsleeps still in place, all tests pass.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #20462 from vanzin/SPARK-23020.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/969eda4a02faa7ca6cf3aff5cd10e6d51026b845"},{"sha":"b3a04283f490020c13b6750de021af734c449c3a","author":{"name":"Zhan Zhang","email":"e965a14e36a6f6b1cb0a2060e7a12b97f12d2a62@fb.com"},"message":"[SPARK-23306] Fix the oom caused by contention\n\n## What changes were proposed in this pull request?\n\nhere is race condition in TaskMemoryManger, which may cause OOM.\n\nThe memory released may be taken by another task because there is a gap between releaseMemory and acquireMemory, e.g., UnifiedMemoryManager, causing the OOM. if the current is the only one that can perform spill. It can happen to BytesToBytesMap, as it only spill required bytes.\n\nLoop on current consumer if it still has memory to release.\n\n## How was this patch tested?\n\nThe race contention is hard to reproduce, but the current logic seems causing the issue.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Zhan Zhang <zhanzhang@fb.com>\n\nCloses #20480 from zhzhan/oom.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/b3a04283f490020c13b6750de021af734c449c3a"},{"sha":"19c7c7ebdef6c1c7a02ebac9af6a24f521b52c37","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23301][SQL] data source column pruning should work for arbitrary expressions\n\n## What changes were proposed in this pull request?\n\nThis PR fixes a mistake in the `PushDownOperatorsToDataSource` rule, the column pruning logic is incorrect about `Project`.\n\n## How was this patch tested?\n\na new test case for column pruning with arbitrary expressions, and improve the existing tests to make sure the `PushDownOperatorsToDataSource` really works.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20476 from cloud-fan/push-down.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/19c7c7ebdef6c1c7a02ebac9af6a24f521b52c37"},{"sha":"b9503fcbb3f4a3ce263164d1f11a8e99b9ca5710","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23312][SQL] add a config to turn off vectorized cache reader\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-23309 reported a performance regression about cached table in Spark 2.3. While the investigating is still going on, this PR adds a conf to turn off the vectorized cache reader, to unblock the 2.3 release.\n\n## How was this patch tested?\n\na new test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20483 from cloud-fan/cache.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/b9503fcbb3f4a3ce263164d1f11a8e99b9ca5710"},{"sha":"dd52681bf542386711609cb037a55b3d264eddef","author":{"name":"Kent Yao","email":"89e6c9b08dcbb2e5502d1d6c85d07b61a2e851a7@hotmail.com"},"message":"[SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one\n\n## What changes were proposed in this pull request?\n\nShuffle Index temporay file is used for atomic creating shuffle index file, it is not needed when the index file already exists after another attempts of same task had it done.\n\n## How was this patch tested?\n\nexitsting ut\n\ncc squito\n\nAuthor: Kent Yao <yaooqinn@hotmail.com>\n\nCloses #20422 from yaooqinn/SPARK-23253.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/dd52681bf542386711609cb037a55b3d264eddef"},{"sha":"eefec93d193d43d5b71b8f8a4b1060286da971dd","author":{"name":"Kent Yao","email":"89e6c9b08dcbb2e5502d1d6c85d07b61a2e851a7@hotmail.com"},"message":"[SPARK-23295][BUILD][MINOR] Exclude Waring message when generating versions in make-distribution.sh\n\n## What changes were proposed in this pull request?\n\nWhen we specified a wrong profile to make a spark distribution, such as `-Phadoop1000`, we will get an odd package named like `spark-[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.-bin-hadoop-2.7.tgz`, which actually should be `\"spark-$VERSION-bin-$NAME.tgz\"`\n\n## How was this patch tested?\n### before\n```\nbuild/mvn help:evaluate -Dexpression=scala.binary.version -Phadoop1000 2>/dev/null | grep -v \"INFO\" | tail -n 1\n[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.\n```\n```\nbuild/mvn help:evaluate -Dexpression=project.version -Phadoop1000 2>/dev/null | grep -v \"INFO\" | tail -n 1\n[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.\n```\n### after\n```\n build/mvn help:evaluate -Dexpression=project.version -Phadoop1000 2>/dev/null | grep  -v \"INFO\" | grep -v \"WARNING\" | tail -n 1\n2.4.0-SNAPSHOT\n```\n```\nbuild/mvn help:evaluate -Dexpression=scala.binary.version -Dscala.binary.version=2.11.1 2>/dev/null | grep  -v \"INFO\" | grep -v \"WARNING\" | tail -n 1\n2.11.1\n```\n\ncloud-fan srowen\n\nAuthor: Kent Yao <yaooqinn@hotmail.com>\n\nCloses #20469 from yaooqinn/dist-minor.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/eefec93d193d43d5b71b8f8a4b1060286da971dd"},{"sha":"eaf35de2471fac4337dd2920026836d52b1ec847","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-23064][SS][DOCS] Stream-stream joins Documentation - follow up\n\n## What changes were proposed in this pull request?\nFurther clarification of caveats in using stream-stream outer joins.\n\n## How was this patch tested?\nN/A\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #20494 from tdas/SPARK-23064-2.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/eaf35de2471fac4337dd2920026836d52b1ec847"},{"sha":"3ff83ad43a704cc3354ef9783e711c065e2a1a22","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SQL] Minor doc update: Add an example in DataFrameReader.schema\n\n## What changes were proposed in this pull request?\nThis patch adds a small example to the schema string definition of schema function. It isn't obvious how to use it, so an example would be useful.\n\n## How was this patch tested?\nN/A - doc only.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #20491 from rxin/schema-doc.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/3ff83ad43a704cc3354ef9783e711c065e2a1a22"},{"sha":"fe73cb4b439169f16cc24cd851a11fd398ce7edf","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23317][SQL] rename ContinuousReader.setOffset to setStartOffset\n\n## What changes were proposed in this pull request?\n\nIn the document of `ContinuousReader.setOffset`, we say this method is used to specify the start offset. We also have a `ContinuousReader.getStartOffset` to get the value back. I think it makes more sense to rename `ContinuousReader.setOffset` to `setStartOffset`.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20486 from cloud-fan/rename.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/fe73cb4b439169f16cc24cd851a11fd398ce7edf"},{"sha":"63b49fa2e599080c2ba7d5189f9dde20a2e01fb4","author":{"name":"caoxuewen","email":"d5c631261607753c6bb0c71e4f7dc1d8bf7c3e51@zte.com.cn"},"message":"[SPARK-23311][SQL][TEST] add FilterFunction test case for test CombineTypedFilters\n\n## What changes were proposed in this pull request?\n\nIn the current test case for CombineTypedFilters, we lack the test of FilterFunction, so let's add it.\nIn addition, in TypedFilterOptimizationSuite's existing test cases, Let's extract a common LocalRelation.\n\n## How was this patch tested?\n\nadd new test cases.\n\nAuthor: caoxuewen <cao.xuewen@zte.com.cn>\n\nCloses #20482 from heary-cao/TypedFilterOptimizationSuite.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/63b49fa2e599080c2ba7d5189f9dde20a2e01fb4"},{"sha":"522e0b1866a0298669c83de5a47ba380dc0b7c84","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMissingFiles` for all file-based data sources\n\n## What changes were proposed in this pull request?\n\nLike Parquet, all file-based data source handles `spark.sql.files.ignoreMissingFiles` correctly. We had better have a test coverage for feature parity and in order to prevent future accidental regression for all data sources.\n\n## How was this patch tested?\n\nPass Jenkins with a newly added test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #20479 from dongjoon-hyun/SPARK-23305.","distinct":false,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/522e0b1866a0298669c83de5a47ba380dc0b7c84"}]}
{"actor":{"display_login":"HyukjinKwon"}}