{"push_id":2712070658,"size":14,"distinct_size":14,"ref":"refs/heads/master","head":"ebf4bfb966389342bfd9bdb8e3b612828c18730c","before":"1bd3d61f4191767a94b71b42f4d00706b703e84f","commits":[{"sha":"aec966b05e8df9d459dae88d091de1923e50e2dc","author":{"name":"Xiao Li","email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com"},"message":"Revert \"[SPARK-24268][SQL] Use datatype.simpleString in error messages\"\n\nThis reverts commit 1bd3d61f4191767a94b71b42f4d00706b703e84f.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/aec966b05e8df9d459dae88d091de1923e50e2dc"},{"sha":"eb6e9880397dbac8b0b9ebc0796150b6924fc566","author":{"name":"Xiao Li","email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com"},"message":"[SPARK-24759][SQL] No reordering keys for broadcast hash join\n\n## What changes were proposed in this pull request?\n\nAs the implementation of the broadcast hash join is independent of the input hash partitioning, reordering keys is not necessary. Thus, we solve this issue by simply removing the broadcast hash join from the reordering rule in EnsureRequirements.\n\n## How was this patch tested?\nN/A\n\nAuthor: Xiao Li <gatorsmile@gmail.com>\n\nCloses #21728 from gatorsmile/cleanER.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/eb6e9880397dbac8b0b9ebc0796150b6924fc566"},{"sha":"4984f1af7e48dab1ae08021a3b17c5ad6d47a87e","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[MINOR] Add Sphinx into dev/requirements.txt\n\n## What changes were proposed in this pull request?\n\nNot a big deal but this PR adds `sphinx` into `dev/requirements.txt` since we found it needed - https://github.com/apache/spark-website/pull/122#discussion_r200896018\n\n## How was this patch tested?\n\nmanually:\n\n```\npip install -r requirements.txt\n```\n\nAuthor: hyukjinkwon <gurwls223@apache.org>\n\nCloses #21735 from HyukjinKwon/minor-dev.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/4984f1af7e48dab1ae08021a3b17c5ad6d47a87e"},{"sha":"a289009567c1566a1df4bcdfdf0111e82ae3d81d","author":{"name":"Yuming Wang","email":"a045e9e381e8d6ed98f3af6ea924f96c8dd7fbad@ebay.com"},"message":"[SPARK-24706][SQL] ByteType and ShortType support pushdown to parquet\n\n## What changes were proposed in this pull request?\n\n`ByteType` and `ShortType` support pushdown to parquet data source.\n[Benchmark result](https://issues.apache.org/jira/browse/SPARK-24706?focusedCommentId=16528878&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16528878).\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Yuming Wang <yumwang@ebay.com>\n\nCloses #21682 from wangyum/SPARK-24706.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/a289009567c1566a1df4bcdfdf0111e82ae3d81d"},{"sha":"6fe32869ccb17933e77a4dbe883e36d382fbeeec","author":{"name":"sharkdtu","email":"ee65aa1c0c6710c0c532e9940f7b27915fa5db8d@tencent.com"},"message":"[SPARK-24678][SPARK-STREAMING] Give priority in use of 'PROCESS_LOCAL' for spark-streaming\n\n## What changes were proposed in this pull request?\n\nCurrently, `BlockRDD.getPreferredLocations`  only get hosts info of blocks, which results in subsequent schedule level is not better than 'NODE_LOCAL'. We can just make a small changes, the schedule level can be improved to 'PROCESS_LOCAL'\n\n## How was this patch tested?\n\nmanual test\n\nAuthor: sharkdtu <sharkdtu@tencent.com>\n\nCloses #21658 from sharkdtu/master.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/6fe32869ccb17933e77a4dbe883e36d382fbeeec"},{"sha":"e0559f238009e02c40f65678fec691c07904e8c0","author":{"name":"Wenchen Fan","email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com"},"message":"[SPARK-21743][SQL][FOLLOWUP] free aggregate map when task ends\n\n## What changes were proposed in this pull request?\n\nThis is the first follow-up of https://github.com/apache/spark/pull/21573 , which was only merged to 2.3.\n\nThis PR fixes the memory leak in another way: free the `UnsafeExternalMap` when the task ends. All the data buffers in Spark SQL are using `UnsafeExternalMap` and `UnsafeExternalSorter` under the hood, e.g. sort, aggregate, window, SMJ, etc. `UnsafeExternalSorter` registers a task completion listener to free the resource, we should apply the same thing to `UnsafeExternalMap`.\n\nTODO in the next PR:\ndo not consume all the inputs when having limit in whole stage codegen.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #21738 from cloud-fan/limit.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/e0559f238009e02c40f65678fec691c07904e8c0"},{"sha":"32cb50835e7258625afff562939872be002232f2","author":{"name":"Mukul Murthy","email":"98f831a821649774489300334604b28b7c1df7e6@gmail.com"},"message":"[SPARK-24662][SQL][SS] Support limit in structured streaming\n\n## What changes were proposed in this pull request?\n\nSupport the LIMIT operator in structured streaming.\n\nFor streams in append or complete output mode, a stream with a LIMIT operator will return no more than the specified number of rows. LIMIT is still unsupported for the update output mode.\n\nThis change reverts https://github.com/apache/spark/commit/e4fee395ecd93ad4579d9afbf0861f82a303e563 as part of it because it is a better and more complete implementation.\n\n## How was this patch tested?\n\nNew and existing unit tests.\n\nAuthor: Mukul Murthy <mukul.murthy@gmail.com>\n\nCloses #21662 from mukulmurthy/SPARK-24662.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/32cb50835e7258625afff562939872be002232f2"},{"sha":"6078b891da8fe7fc36579699473168ae7443284c","author":{"name":"Tathagata Das","email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com"},"message":"[SPARK-24730][SS] Add policy to choose max as global watermark when streaming query has multiple watermarks\n\n## What changes were proposed in this pull request?\n\nCurrently, when a streaming query has multiple watermark, the policy is to choose the min of them as the global watermark. This is safe to do as the global watermark moves with the slowest stream, and is therefore is safe as it does not unexpectedly drop some data as late, etc. While this is indeed the safe thing to do, in some cases, you may want the watermark to advance with the fastest stream, that is, take the max of multiple watermarks. This PR is to add that configuration. It makes the following changes.\n\n- Adds a configuration to specify max as the policy.\n- Saves the configuration in OffsetSeqMetadata because changing it in the middle can lead to unpredictable results.\n   - For old checkpoints without the configuration, it assumes the default policy as min (irrespective of the policy set at the session where the query is being restarted). This is to ensure that existing queries are affected in any way.\n\nTODO\n- [ ] Add a test for recovery from existing checkpoints.\n\n## How was this patch tested?\nNew unit test\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #21701 from tdas/SPARK-24730.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/6078b891da8fe7fc36579699473168ae7443284c"},{"sha":"1f94bf492c3bce3b61f7fec6132b50e06dea94a8","author":{"name":"hyukjinkwon","email":"2680965ad5aae29275c8fb10150cb10f5554e78b@apache.org"},"message":"[SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to add `SPHINXPYTHON` environment variable to control the Python version used by Sphinx.\n\nThe motivation of this environment variable is, it seems not properly rendering some signatures in the Python documentation when Python 2 is used by Sphinx. See the JIRA's case. It should be encouraged to use Python 3, but looks we will probably live with this problem for a long while in any event.\n\nFor the default case of `make html`, it keeps previous behaviour and use `SPHINXBUILD` as it was. If `SPHINXPYTHON` is set, then it forces Sphinx to use the specific Python version.\n\n```\n$ SPHINXPYTHON=python3 make html\npython3 -msphinx -b html -d _build/doctrees   . _build/html\nRunning Sphinx v1.7.5\n...\n```\n\n1. if `SPHINXPYTHON` is set, use Python. If `SPHINXBUILD` is set, use sphinx-build.\n2. If both are set, `SPHINXBUILD` has a higher priority over `SPHINXPYTHON`\n3. By default, `SPHINXBUILD` is used as 'sphinx-build'.\n\nProbably, we can somehow work around this via explicitly setting `SPHINXBUILD` but `sphinx-build` can't be easily distinguished since it (at least in my environment and up to my knowledge) doesn't replace `sphinx-build` when newer Sphinx is installed in different Python version. It confuses and doesn't warn for its Python version.\n\n## How was this patch tested?\n\nManually tested:\n\n**`python` (Python 2.7) in the path with Sphinx:**\n\n```\n$ make html\nsphinx-build -b html -d _build/doctrees   . _build/html\nRunning Sphinx v1.7.5\n...\n```\n\n**`python` (Python 2.7) in the path without Sphinx:**\n\n```\n$ make html\nMakefile:8: *** The 'sphinx-build' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the 'sphinx-build' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/.  Stop.\n```\n\n**`SPHINXPYTHON` set `python` (Python 2.7)  with Sphinx:**\n\n```\n$ SPHINXPYTHON=python make html\nMakefile:35: *** Note that Python 3 is required to generate PySpark documentation correctly for now. Current Python executable was less than Python 3. See SPARK-24530. To force Sphinx to use a specific Python executable, please set SPHINXPYTHON to point to the Python 3 executable..  Stop.\n```\n\n**`SPHINXPYTHON` set `python` (Python 2.7)  without Sphinx:**\n\n```\n$ SPHINXPYTHON=python make html\nMakefile:35: *** Note that Python 3 is required to generate PySpark documentation correctly for now. Current Python executable was less than Python 3. See SPARK-24530. To force Sphinx to use a specific Python executable, please set SPHINXPYTHON to point to the Python 3 executable..  Stop.\n```\n\n**`SPHINXPYTHON` set `python3` with Sphinx:**\n\n```\n$ SPHINXPYTHON=python3 make html\npython3 -msphinx -b html -d _build/doctrees   . _build/html\nRunning Sphinx v1.7.5\n...\n```\n\n**`SPHINXPYTHON` set `python3` without Sphinx:**\n\n```\n$ SPHINXPYTHON=python3 make html\nMakefile:39: *** Python executable 'python3' did not have Sphinx installed. Make sure you have Sphinx installed, then set the SPHINXPYTHON environment variable to point to the Python executable having Sphinx installed. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/.  Stop.\n```\n\n**`SPHINXBUILD` set:**\n\n```\n$ SPHINXBUILD=sphinx-build make html\nsphinx-build -b html -d _build/doctrees   . _build/html\nRunning Sphinx v1.7.5\n...\n```\n\n**Both `SPHINXPYTHON` and `SPHINXBUILD` are set:**\n\n```\n$ SPHINXBUILD=sphinx-build SPHINXPYTHON=python make html\nsphinx-build -b html -d _build/doctrees   . _build/html\nRunning Sphinx v1.7.5\n...\n```\n\nAuthor: hyukjinkwon <gurwls223@apache.org>\n\nCloses #21659 from HyukjinKwon/SPARK-24530.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/1f94bf492c3bce3b61f7fec6132b50e06dea94a8"},{"sha":"74a8d6308bfa6e7ed4c64e1175c77eb3114baed5","author":{"name":"Marek Novotny","email":"51a58b3768f62be18290af0bb1b9e2d6d3d258c5@gmail.com"},"message":"[SPARK-24165][SQL] Fixing conditional expressions to handle nullability of nested types\n\n## What changes were proposed in this pull request?\nThis PR is proposing a fix for the output data type of ```If``` and ```CaseWhen``` expression. Upon till now, the implementation of exprassions has ignored nullability of nested types from different execution branches and returned the type of the first branch.\n\nThis could lead to an unwanted ```NullPointerException``` from other expressions depending on a ```If```/```CaseWhen``` expression.\n\nExample:\n```\nval rows = new util.ArrayList[Row]()\nrows.add(Row(true, (\"a\", 1)))\nrows.add(Row(false, (null, 2)))\nval schema = StructType(Seq(\n  StructField(\"cond\", BooleanType, false),\n  StructField(\"s\", StructType(Seq(\n    StructField(\"val1\", StringType, true),\n    StructField(\"val2\", IntegerType, false)\n  )), false)\n))\n\nval df = spark.createDataFrame(rows, schema)\n\ndf\n  .select(when('cond, struct(lit(\"x\").as(\"val1\"), lit(10).as(\"val2\"))).otherwise('s) as \"res\")\n  .select('res.getField(\"val1\"))\n  .show()\n```\nException:\n```\nException in thread \"main\" java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:109)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)\n\tat org.apache.spark.sql.execution.LocalTableScanExec$$anonfun$unsafeRows$1.apply(LocalTableScanExec.scala:44)\n\tat org.apache.spark.sql.execution.LocalTableScanExec$$anonfun$unsafeRows$1.apply(LocalTableScanExec.scala:44)\n...\n```\nOutput schema:\n```\nroot\n |-- res.val1: string (nullable = false)\n```\n\n## How was this patch tested?\nNew test cases added into\n- DataFrameSuite.scala\n- conditionalExpressions.scala\n\nAuthor: Marek Novotny <mn.mikke@gmail.com>\n\nCloses #21687 from mn-mikke/SPARK-24165.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/74a8d6308bfa6e7ed4c64e1175c77eb3114baed5"},{"sha":"5ff1b9ba1983d5601add62aef64a3e87d07050eb","author":{"name":"Andrew Korzhuev","email":"5b08f6bd2a52aff8d94a9f4a7c0dd2a4e05922cf@klarna.com"},"message":"[SPARK-23529][K8S] Support mounting volumes\n\nThis PR continues #21095 and intersects with #21238. I've added volume mounts as a separate step and added PersistantVolumeClaim support.\n\nThere is a fundamental problem with how we pass the options through spark conf to fabric8. For each volume type and all possible volume options we would have to implement some custom code to map config values to fabric8 calls. This will result in big body of code we would have to support and means that Spark will always be somehow out of sync with k8s.\n\nI think there needs to be a discussion on how to proceed correctly (eg use PodPreset instead)\n\n----\n\nDue to the complications of provisioning and managing actual resources this PR addresses only volume mounting of already present resources.\n\n----\n- [x] emptyDir support\n- [x] Testing\n- [x] Documentation\n- [x] KubernetesVolumeUtils tests\n\nAuthor: Andrew Korzhuev <andrew.korzhuev@klarna.com>\nAuthor: madanadit <adit@alluxio.com>\n\nCloses #21260 from andrusha/k8s-vol.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/5ff1b9ba1983d5601add62aef64a3e87d07050eb"},{"sha":"006e798e477b6871ad3ba4417d354d23f45e4013","author":{"name":"Huaxin Gao","email":"b1011540267ee7fe5a02243ad09c007ca1860809@us.ibm.com"},"message":"[SPARK-23461][R] vignettes should include model predictions for some ML models\n\n## What changes were proposed in this pull request?\n\nAdd model predictions for Linear Support Vector Machine (SVM) Classifier, Logistic Regression, GBT, RF and DecisionTree in vignettes.\n\n## How was this patch tested?\n\nManually ran the test and checked the result.\n\nAuthor: Huaxin Gao <huaxing@us.ibm.com>\n\nCloses #21678 from huaxingao/spark-23461.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/006e798e477b6871ad3ba4417d354d23f45e4013"},{"sha":"592cc84583d74c78e4cdf34a3b82692c8de8f4a9","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-24562][TESTS] Support different configs for same test in SQLQueryTestSuite\n\n## What changes were proposed in this pull request?\n\nThe PR proposes to add support for running the same SQL test input files against different configs leading to the same result.\n\n## How was this patch tested?\n\nInvolved UTs\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\n\nCloses #21568 from mgaido91/SPARK-24562.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/592cc84583d74c78e4cdf34a3b82692c8de8f4a9"},{"sha":"ebf4bfb966389342bfd9bdb8e3b612828c18730c","author":{"name":"Marco Gaido","email":"368e8ff04b169d99d4d85a32ca87c1aa4ab0fa7e@gmail.com"},"message":"[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas\n\n## What changes were proposed in this pull request?\n\nA self-join on a dataset which contains a `FlatMapGroupsInPandas` fails because of duplicate attributes. This happens because we are not dealing with this specific case in our `dedupAttr` rules.\n\nThe PR fix the issue by adding the management of the specific case\n\n## How was this patch tested?\n\nadded UT + manual tests\n\nAuthor: Marco Gaido <marcogaido91@gmail.com>\nAuthor: Marco Gaido <mgaido@hortonworks.com>\n\nCloses #21737 from mgaido91/SPARK-24208.","distinct":true,"url":"https://api.github.com/repos/datalayer-contrib/spark/commits/ebf4bfb966389342bfd9bdb8e3b612828c18730c"}]}
{"actor":{"display_login":"echarles"}}