{"push_id":2306181870,"size":13,"distinct_size":13,"ref":"refs/heads/explain","head":"a40d18ea08a62ecafa1d120bb7ce38019ba57869","before":"4ca2c40b869fc4f1334fd7bce277a0c467bd5020","commits":[{"sha":"b9503fcbb3f4a3ce263164d1f11a8e99b9ca5710","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23312][SQL] add a config to turn off vectorized cache reader\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-23309 reported a performance regression about cached table in Spark 2.3. While the investigating is still going on, this PR adds a conf to turn off the vectorized cache reader, to unblock the 2.3 release.\n\n## How was this patch tested?\n\na new test\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20483 from cloud-fan/cache.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b9503fcbb3f4a3ce263164d1f11a8e99b9ca5710"},{"sha":"dd52681bf542386711609cb037a55b3d264eddef","author":{"name":"Kent Yao","email":"89e6c9b08dcbb2e5502d1d6c85d07b61a2e851a7@hotmail.com"},"message":"[SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one\n\n## What changes were proposed in this pull request?\n\nShuffle Index temporay file is used for atomic creating shuffle index file, it is not needed when the index file already exists after another attempts of same task had it done.\n\n## How was this patch tested?\n\nexitsting ut\n\ncc squito\n\nAuthor: Kent Yao <yaooqinn@hotmail.com>\n\nCloses #20422 from yaooqinn/SPARK-23253.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/dd52681bf542386711609cb037a55b3d264eddef"},{"sha":"eefec93d193d43d5b71b8f8a4b1060286da971dd","author":{"name":"Kent Yao","email":"89e6c9b08dcbb2e5502d1d6c85d07b61a2e851a7@hotmail.com"},"message":"[SPARK-23295][BUILD][MINOR] Exclude Waring message when generating versions in make-distribution.sh\n\n## What changes were proposed in this pull request?\n\nWhen we specified a wrong profile to make a spark distribution, such as `-Phadoop1000`, we will get an odd package named like `spark-[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.-bin-hadoop-2.7.tgz`, which actually should be `\"spark-$VERSION-bin-$NAME.tgz\"`\n\n## How was this patch tested?\n### before\n```\nbuild/mvn help:evaluate -Dexpression=scala.binary.version -Phadoop1000 2>/dev/null | grep -v \"INFO\" | tail -n 1\n[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.\n```\n```\nbuild/mvn help:evaluate -Dexpression=project.version -Phadoop1000 2>/dev/null | grep -v \"INFO\" | tail -n 1\n[WARNING] The requested profile \"hadoop1000\" could not be activated because it does not exist.\n```\n### after\n```\n build/mvn help:evaluate -Dexpression=project.version -Phadoop1000 2>/dev/null | grep  -v \"INFO\" | grep -v \"WARNING\" | tail -n 1\n2.4.0-SNAPSHOT\n```\n```\nbuild/mvn help:evaluate -Dexpression=scala.binary.version -Dscala.binary.version=2.11.1 2>/dev/null | grep  -v \"INFO\" | grep -v \"WARNING\" | tail -n 1\n2.11.1\n```\n\ncloud-fan srowen\n\nAuthor: Kent Yao <yaooqinn@hotmail.com>\n\nCloses #20469 from yaooqinn/dist-minor.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/eefec93d193d43d5b71b8f8a4b1060286da971dd"},{"sha":"eaf35de2471fac4337dd2920026836d52b1ec847","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-23064][SS][DOCS] Stream-stream joins Documentation - follow up\n\n## What changes were proposed in this pull request?\nFurther clarification of caveats in using stream-stream outer joins.\n\n## How was this patch tested?\nN/A\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #20494 from tdas/SPARK-23064-2.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/eaf35de2471fac4337dd2920026836d52b1ec847"},{"sha":"3ff83ad43a704cc3354ef9783e711c065e2a1a22","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SQL] Minor doc update: Add an example in DataFrameReader.schema\n\n## What changes were proposed in this pull request?\nThis patch adds a small example to the schema string definition of schema function. It isn't obvious how to use it, so an example would be useful.\n\n## How was this patch tested?\nN/A - doc only.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #20491 from rxin/schema-doc.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/3ff83ad43a704cc3354ef9783e711c065e2a1a22"},{"sha":"fe73cb4b439169f16cc24cd851a11fd398ce7edf","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-23317][SQL] rename ContinuousReader.setOffset to setStartOffset\n\n## What changes were proposed in this pull request?\n\nIn the document of `ContinuousReader.setOffset`, we say this method is used to specify the start offset. We also have a `ContinuousReader.getStartOffset` to get the value back. I think it makes more sense to rename `ContinuousReader.setOffset` to `setStartOffset`.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #20486 from cloud-fan/rename.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/fe73cb4b439169f16cc24cd851a11fd398ce7edf"},{"sha":"63b49fa2e599080c2ba7d5189f9dde20a2e01fb4","author":{"name":"caoxuewen","email":"d5c631261607753c6bb0c71e4f7dc1d8bf7c3e51@zte.com.cn"},"message":"[SPARK-23311][SQL][TEST] add FilterFunction test case for test CombineTypedFilters\n\n## What changes were proposed in this pull request?\n\nIn the current test case for CombineTypedFilters, we lack the test of FilterFunction, so let's add it.\nIn addition, in TypedFilterOptimizationSuite's existing test cases, Let's extract a common LocalRelation.\n\n## How was this patch tested?\n\nadd new test cases.\n\nAuthor: caoxuewen <cao.xuewen@zte.com.cn>\n\nCloses #20482 from heary-cao/TypedFilterOptimizationSuite.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/63b49fa2e599080c2ba7d5189f9dde20a2e01fb4"},{"sha":"522e0b1866a0298669c83de5a47ba380dc0b7c84","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMissingFiles` for all file-based data sources\n\n## What changes were proposed in this pull request?\n\nLike Parquet, all file-based data source handles `spark.sql.files.ignoreMissingFiles` correctly. We had better have a test coverage for feature parity and in order to prevent future accidental regression for all data sources.\n\n## How was this patch tested?\n\nPass Jenkins with a newly added test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #20479 from dongjoon-hyun/SPARK-23305.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/522e0b1866a0298669c83de5a47ba380dc0b7c84"},{"sha":"4aaa7d40bf495317e740b6d6f9c2a55dfd03521b","author":{"name":"Shashwat Anand","email":"b4a5ea001e047040e8c9075b67a268293138da2c@shashwat.me"},"message":"[MINOR][DOC] Use raw triple double quotes around docstrings where there are occurrences of backslashes.\n\nFrom [PEP 257](https://www.python.org/dev/peps/pep-0257/):\n\n> For consistency, always use \"\"\"triple double quotes\"\"\" around docstrings. Use r\"\"\"raw triple double quotes\"\"\" if you use any backslashes in your docstrings. For Unicode docstrings, use u\"\"\"Unicode triple-quoted strings\"\"\".\n\nFor example, this is what help (kafka_wordcount) shows:\n\n```\nDESCRIPTION\n    Counts words in UTF8 encoded, '\n    ' delimited text received from the network every second.\n     Usage: kafka_wordcount.py <zk> <topic>\n\n     To run this on your local machine, you need to setup Kafka and create a producer first, see\n     http://kafka.apache.org/documentation.html#quickstart\n\n     and then run the example\n        `$ bin/spark-submit --jars       external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar       examples/src/main/python/streaming/kafka_wordcount.py       localhost:2181 test`\n```\n\nThis is what it shows, after the fix:\n\n```\nDESCRIPTION\n    Counts words in UTF8 encoded, '\\n' delimited text received from the network every second.\n    Usage: kafka_wordcount.py <zk> <topic>\n\n    To run this on your local machine, you need to setup Kafka and create a producer first, see\n    http://kafka.apache.org/documentation.html#quickstart\n\n    and then run the example\n       `$ bin/spark-submit --jars \\\n         external/kafka-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \\\n         examples/src/main/python/streaming/kafka_wordcount.py \\\n         localhost:2181 test`\n```\n\nThe thing worth noticing is no linebreak here in the help.\n\n## What changes were proposed in this pull request?\n\nChange triple double quotes to raw triple double quotes when there are occurrences of backslashes in docstrings.\n\n## How was this patch tested?\n\nManually as this is a doc fix.\n\nAuthor: Shashwat Anand <me@shashwat.me>\n\nCloses #20497 from ashashwat/docstring-fixes.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4aaa7d40bf495317e740b6d6f9c2a55dfd03521b"},{"sha":"551dff2bccb65e9b3f77b986f167aec90d9a6016","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-21658][SQL][PYSPARK] Revert \"[] Add default None for value in na.replace in PySpark\"\n\nThis reverts commit 0fcde87aadc9a92e138f11583119465ca4b5c518.\n\nSee the discussion in [SPARK-21658](https://issues.apache.org/jira/browse/SPARK-21658),  [SPARK-19454](https://issues.apache.org/jira/browse/SPARK-19454) and https://github.com/apache/spark/pull/16793\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #20496 from HyukjinKwon/revert-SPARK-21658.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/551dff2bccb65e9b3f77b986f167aec90d9a6016"},{"sha":"715047b02df0ac9ec16ab2a73481ab7f36ffc6ca","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-23256][ML][PYTHON] Add columnSchema method to PySpark image reader\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to add `columnSchema` in Python side too.\n\n```python\n>>> from pyspark.ml.image import ImageSchema\n>>> ImageSchema.columnSchema.simpleString()\n'struct<origin:string,height:int,width:int,nChannels:int,mode:int,data:binary>'\n```\n\n## How was this patch tested?\n\nManually tested and unittest was added in `python/pyspark/ml/tests.py`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #20475 from HyukjinKwon/SPARK-23256.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/715047b02df0ac9ec16ab2a73481ab7f36ffc6ca"},{"sha":"6fb3fd15365d43733aefdb396db205d7ccf57f75","author":{"name":"Yuming Wang","email":"b4045a50547081d83941485c6596bd2032665522@gmail.com"},"message":"[SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperations.sql\n\n## What changes were proposed in this pull request?\n\nFix decimalArithmeticOperations.sql test\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Yuming Wang <wgyumg@gmail.com>\nAuthor: wangyum <wgyumg@gmail.com>\nAuthor: Yuming Wang <yumwang@ebay.com>\n\nCloses #20498 from wangyum/SPARK-22036.","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6fb3fd15365d43733aefdb396db205d7ccf57f75"},{"sha":"a40d18ea08a62ecafa1d120bb7ce38019ba57869","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"improve the explain result for data source v2 relations","distinct":true,"url":"https://api.github.com/repos/cloud-fan/spark/commits/a40d18ea08a62ecafa1d120bb7ce38019ba57869"}]}
{"actor":{"display_login":"cloud-fan"}}