{"push_id":1358789421,"size":1000,"distinct_size":0,"ref":"refs/heads/master","head":"444c2d22e38a8a78135adf0d3a3774f0e9fc866c","before":"02a029df43392c5d73697203bf6ff51b8d6efb83","commits":[{"sha":"ec79183ac5b842e49baa40ea1c9a72ce8f108fe5","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16340][SQL] Support column arguments for `regexp_replace` Dataset operation\n\n## What changes were proposed in this pull request?\n\nCurrently, `regexp_replace` function supports `Column` arguments in a query. This PR supports that in a `Dataset` operation, too.\n\n## How was this patch tested?\n\nPass the Jenkins tests with a updated testcase.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14060 from dongjoon-hyun/SPARK-16340.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/ec79183ac5b842e49baa40ea1c9a72ce8f108fe5"},{"sha":"5f342049cce9102fb62b4de2d8d8fa691c2e8ac4","author":{"name":"Tejas Patil","email":"6ad8f8971953bfaf2a65cdbf75ed01ef8f5bdc87@fb.com"},"message":"[SPARK-16339][CORE] ScriptTransform does not print stderr when outstream is lost\n\n## What changes were proposed in this pull request?\n\nCurrently, if due to some failure, the outstream gets destroyed or closed and later `outstream.close()` leads to IOException in such case. Due to this, the `stderrBuffer` does not get logged and there is no way for users to see why the job failed.\n\nThe change is to first display the stderr buffer and then try closing the outstream.\n\n## How was this patch tested?\n\nThe correct way to test this fix would be to grep the log to see if the `stderrBuffer` gets logged but I dont think having test cases which do that is a good idea.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\n…\n\nAuthor: Tejas Patil <tejasp@fb.com>\n\nCloses #13834 from tejasapatil/script_transform.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/5f342049cce9102fb62b4de2d8d8fa691c2e8ac4"},{"sha":"5497242c769b40338bfa57d64f2c64996dfa57e8","author":{"name":"Yuhao Yang","email":"857769fa089c7426f7be1ac29063eaedcb246e23@intel.com"},"message":"[SPARK-16249][ML] Change visibility of Object ml.clustering.LDA to public for loading\n\n## What changes were proposed in this pull request?\njira: https://issues.apache.org/jira/browse/SPARK-16249\nChange visibility of Object ml.clustering.LDA to public for loading, thus users can invoke LDA.load(\"path\").\n\n## How was this patch tested?\n\nexisting ut and manually test for load ( saved with current code)\n\nAuthor: Yuhao Yang <yuhao.yang@intel.com>\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #13941 from hhbyyh/ldapublic.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/5497242c769b40338bfa57d64f2c64996dfa57e8"},{"sha":"7e28fabdff2da1cc374efbf43372d92ae0cd07aa","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16388][SQL] Remove spark.sql.nativeView and spark.sql.nativeView.canonical config\n\n## What changes were proposed in this pull request?\nThese two configs should always be true after Spark 2.0. This patch removes them from the config list. Note that ideally this should've gone into branch-2.0, but due to the timing of the release we should only merge this in master for Spark 2.1.\n\n## How was this patch tested?\nUpdated test cases.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14061 from rxin/SPARK-16388.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/7e28fabdff2da1cc374efbf43372d92ae0cd07aa"},{"sha":"909c6d812f6ca3a3305e4611a700c8c17905b953","author":{"name":"MechCoder","email":"f3b3c28cb28c968d96a6fc2a6a7c7d189719947d@nyu.edu"},"message":"[SPARK-16307][ML] Add test to verify the predicted variances of a DT on toy data\n\n## What changes were proposed in this pull request?\n\nThe current tests assumes that `impurity.calculate()` returns the variance correctly. It should be better to make the tests independent of this assumption. In other words verify that the variance computed equals the variance computed manually on a small tree.\n\n## How was this patch tested?\n\nThe patch is a test....\n\nAuthor: MechCoder <mks542@nyu.edu>\n\nCloses #13981 from MechCoder/dt_variance.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/909c6d812f6ca3a3305e4611a700c8c17905b953"},{"sha":"21eadd1d8cbf029197e73ffca1cba54d5a890c01","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16229][SQL] Drop Empty Table After CREATE TABLE AS SELECT fails\n\n#### What changes were proposed in this pull request?\nIn `CREATE TABLE AS SELECT`, if the `SELECT` query failed, the table should not exist. For example,\n\n```SQL\nCREATE TABLE tab\nSTORED AS TEXTFILE\nSELECT 1 AS a, (SELECT a FROM (SELECT 1 AS a UNION ALL SELECT 2 AS a) t) AS b\n```\nThe above query failed as expected but an empty table `t` is created.\n\nThis PR is to drop the created table when hitting any non-fatal exception.\n\n#### How was this patch tested?\nAdded a test case to verify the behavior\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13926 from gatorsmile/dropTableAfterException.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/21eadd1d8cbf029197e73ffca1cba54d5a890c01"},{"sha":"478b71d028107d42fbb6d1bd300b86efbe0dcf7d","author":{"name":"Tao Lin","email":"12272d0502edbcb768ca9955d422d10260579899@gmail.com"},"message":"[SPARK-15591][WEBUI] Paginate Stage Table in Stages tab\n\n## What changes were proposed in this pull request?\n\nThis patch adds pagination support for the Stage Tables in the Stage tab. Pagination is provided for all of the four Job Tables (active, pending, completed, and failed). Besides, the paged stage tables are also used in JobPage (the detail page for one job) and PoolPage.\n\nInteractions (jumping, sorting, and setting page size) for paged tables are also included.\n\n## How was this patch tested?\n\nTested manually by using checking the Web UI after completing and failing hundreds of jobs.  Same as the testings for [Paginate Job Table in Jobs tab](https://github.com/apache/spark/pull/13620).\n\nThis shows the pagination for completed stages:\n![paged stage table](https://cloud.githubusercontent.com/assets/5558370/16125696/5804e35e-3427-11e6-8923-5c5948982648.png)\n\nAuthor: Tao Lin <nblintao@gmail.com>\n\nCloses #13708 from nblintao/stageTable.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/478b71d028107d42fbb6d1bd300b86efbe0dcf7d"},{"sha":"23eff5e512df5710ea6591a3fce321b53eb3fb0b","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-15979][SQL] Renames CatalystWriteSupport to ParquetWriteSupport\n\n## What changes were proposed in this pull request?\n\nPR #13696 renamed various Parquet support classes but left `CatalystWriteSupport` behind. This PR is renames it as a follow-up.\n\n## How was this patch tested?\n\nN/A.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #14070 from liancheng/spark-15979-follow-up.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/23eff5e512df5710ea6591a3fce321b53eb3fb0b"},{"sha":"b1310425b30cbd711e4834d65a0accb3c5a8403a","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[DOC][SQL] update out-of-date code snippets using SQLContext in all documents.\n\n## What changes were proposed in this pull request?\n\nI search the whole documents directory using SQLContext, and update the following places:\n\n- docs/configuration.md, sparkR code snippets.\n- docs/streaming-programming-guide.md, several example code.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14025 from WeichenXu123/WIP_SQLContext_update.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/b1310425b30cbd711e4834d65a0accb3c5a8403a"},{"sha":"4e14199ff740ea186eb2cec2e5cf901b58c5f90e","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][PYSPARK][DOC] Fix wrongly formatted examples in PySpark documentation\n\n## What changes were proposed in this pull request?\n\nThis PR fixes wrongly formatted examples in PySpark documentation as below:\n\n- **`SparkSession`**\n\n  - **Before**\n\n    ![2016-07-06 11 34 41](https://cloud.githubusercontent.com/assets/6477701/16605847/ae939526-436d-11e6-8ab8-6ad578362425.png)\n\n  - **After**\n\n    ![2016-07-06 11 33 56](https://cloud.githubusercontent.com/assets/6477701/16605845/ace9ee78-436d-11e6-8923-b76d4fc3e7c3.png)\n\n- **`Builder`**\n\n  - **Before**\n    ![2016-07-06 11 34 44](https://cloud.githubusercontent.com/assets/6477701/16605844/aba60dbc-436d-11e6-990a-c87bc0281c6b.png)\n\n  - **After**\n    ![2016-07-06 1 26 37](https://cloud.githubusercontent.com/assets/6477701/16607562/586704c0-437d-11e6-9483-e0af93d8f74e.png)\n\nThis PR also fixes several similar instances across the documentation in `sql` PySpark module.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14063 from HyukjinKwon/minor-pyspark-builder.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/4e14199ff740ea186eb2cec2e5cf901b58c5f90e"},{"sha":"480357cc6d71c682fe703611c71c1e6a36e6ce9a","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16304] LinkageError should not crash Spark executor\n\n## What changes were proposed in this pull request?\nThis patch updates the failure handling logic so Spark executor does not crash when seeing LinkageError.\n\n## How was this patch tested?\nAdded an end-to-end test in FailureSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #13982 from petermaxlee/SPARK-16304.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/480357cc6d71c682fe703611c71c1e6a36e6ce9a"},{"sha":"4f8ceed59367319300e4bfa5b957c387be81ffa3","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16371][SQL] Do not push down filters incorrectly when inner name and outer name are the same in Parquet\n\n## What changes were proposed in this pull request?\n\nCurrently, if there is a schema as below:\n\n```\nroot\n  |-- _1: struct (nullable = true)\n  |    |-- _1: integer (nullable = true)\n```\n\nand if we execute the codes below:\n\n```scala\ndf.filter(\"_1 IS NOT NULL\").count()\n```\n\nThis pushes down a filter although this filter is being applied to `StructType`.(If my understanding is correct, Spark does not pushes down filters for those).\n\nThe reason is, `ParquetFilters.getFieldMap` produces results below:\n\n```\n(_1,StructType(StructField(_1,IntegerType,true)))\n(_1,IntegerType)\n```\n\nand then it becomes a `Map`\n\n```\n(_1,IntegerType)\n```\n\nNow, because of ` ....lift(dataTypeOf(name)).map(_(name, value))`, this pushes down filters for `_1` which Parquet thinks is `IntegerType`. However, it is actually `StructType`.\n\nSo, Parquet filter2 produces incorrect results, for example, the codes below:\n\n```\ndf.filter(\"_1 IS NOT NULL\").count()\n```\n\nproduces always 0.\n\nThis PR prevents this by not finding nested fields.\n\n## How was this patch tested?\n\nUnit test in `ParquetFilterSuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14067 from HyukjinKwon/SPARK-16371.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/4f8ceed59367319300e4bfa5b957c387be81ffa3"},{"sha":"040f6f9f468f153e4c4db78c26ced0299245fb6f","author":{"name":"tmnd1991","email":"445e5c80fecd10c2370c17d3256340eaba5b8c1c@studio.unibo.it"},"message":"[SPARK-15740][MLLIB] Word2VecSuite \"big model load / save\" caused OOM in maven jenkins builds\n\n## What changes were proposed in this pull request?\n\"test big model load / save\" in Word2VecSuite, lately resulted into OOM.\nTherefore we decided to make the partitioning adaptive (not based on spark default \"spark.kryoserializer.buffer.max\" conf) and then testing it using a small buffer size in order to trigger partitioning without allocating too much memory for the test.\n\n## How was this patch tested?\nIt was tested running the following unit test:\norg.apache.spark.mllib.feature.Word2VecSuite\n\nAuthor: tmnd1991 <antonio.murgia2@studio.unibo.it>\n\nCloses #13509 from tmnd1991/SPARK-15740.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/040f6f9f468f153e4c4db78c26ced0299245fb6f"},{"sha":"a8f89df3b391e7a3fa9f73d9ec730d6eaa95bb09","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16379][CORE][MESOS] Spark on mesos is broken due to race condition in Logging\n\n## What changes were proposed in this pull request?\n\nThe commit https://github.com/apache/spark/commit/044971eca0ff3c2ce62afa665dbd3072d52cbbec introduced a lazy val to simplify code in Logging. Simple enough, though one side effect is that accessing log now means grabbing the instance's lock. This in turn turned up a form of deadlock in the Mesos code. It was arguably a bit of a problem in how this code is structured, but, in any event the safest thing to do seems to be to revert the commit, and that's 90% of the change here; it's just not worth the risk of similar more subtle issues.\n\nWhat I didn't revert here was the removal of this odd override of log in the Mesos code. In retrospect it might have been put in place at some stage as a defense against this type of problem. After all the Logging code still involved a lock at initialization before the change in question.\n\nEven after the revert, it doesn't seem like it does anything, given how Logging works now, so I left it removed. However, I also removed the particular log message that ended up playing a part in this problem anyway, maybe being paranoid, to make sure this type of problem can't happen even with how the current locking works in logging initialization.\n\n## How was this patch tested?\n\nJenkins tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14069 from srowen/SPARK-16379.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/a8f89df3b391e7a3fa9f73d9ec730d6eaa95bb09"},{"sha":"9c041990cf4d0138d9104207b5c2e7a319b42615","author":{"name":"Michael Gummelt","email":"997b1eeb9667009a1fe95939cfe9ef99c4f83395@mesosphere.io"},"message":"[MESOS] expand coarse-grained mode docs\n\n## What changes were proposed in this pull request?\n\ndocs\n\n## How was this patch tested?\n\nviewed the docs in github\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #14059 from mgummelt/coarse-grained.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/9c041990cf4d0138d9104207b5c2e7a319b42615"},{"sha":"8e3e4ed6c090d18675d49eec46b3ee572457db95","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16371][SQL] Two follow-up tasks\n\n## What changes were proposed in this pull request?\nThis is a small follow-up for SPARK-16371:\n\n1. Hide removeMetadata from public API.\n2. Add JIRA ticket number to test case name.\n\n## How was this patch tested?\nUpdated a test comment.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14074 from rxin/parquet-filter.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/8e3e4ed6c090d18675d49eec46b3ee572457db95"},{"sha":"b8ebf63c1e1fa1ab53ea760fa293051c08ce5f59","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-16212][STREAMING][KAFKA] apply test tweaks from 0-10 to 0-8 as well\n\n## What changes were proposed in this pull request?\n\nBring the kafka-0-8 subproject up to date with some test modifications from development on 0-10.\n\nMain changes are\n- eliminating waits on concurrent queue in favor of an assert on received results,\n- atomics instead of volatile (although this probably doesn't matter)\n- increasing uniqueness of topic names\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #14073 from koeninger/kafka-0-8-test-direct-cleanup.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/b8ebf63c1e1fa1ab53ea760fa293051c08ce5f59"},{"sha":"44c7c62bcfca74c82ffc4e3c53997fff47bfacac","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-16021] Fill freed memory in test to help catch correctness bugs\n\n## What changes were proposed in this pull request?\n\nThis patches `MemoryAllocator` to fill clean and freed memory with known byte values, similar to https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Find-a-memory-corruption-bug . Memory filling is flag-enabled in test only by default.\n\n## How was this patch tested?\n\nUnit test that it's on in test.\n\ncc sameeragarwal\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #13983 from ericl/spark-16021.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/44c7c62bcfca74c82ffc4e3c53997fff47bfacac"},{"sha":"34283de160808324da02964cd5dc5df80e59ae71","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-14839][SQL] Support for other types for `tableProperty` rule in SQL syntax\n\n## What changes were proposed in this pull request?\n\nCurrently, Scala API supports to take options with the types, `String`, `Long`, `Double` and `Boolean` and Python API also supports other types.\n\nThis PR corrects `tableProperty` rule to support other types (string, boolean, double and integer) so that support the options for data sources in a consistent way. This will affect other rules such as DBPROPERTIES and TBLPROPERTIES (allowing other types as values).\n\nAlso, `TODO add bucketing and partitioning.` was removed because it was resolved in https://github.com/apache/spark/commit/24bea000476cdd0b43be5160a76bc5b170ef0b42\n\n## How was this patch tested?\n\nUnit test in `MetastoreDataSourcesSuite.scala`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #13517 from HyukjinKwon/SPARK-14839.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/34283de160808324da02964cd5dc5df80e59ae71"},{"sha":"42279bff686f9808ec7a9e8f4da95c717edc6026","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16374][SQL] Remove Alias from MetastoreRelation and SimpleCatalogRelation\n\n#### What changes were proposed in this pull request?\nDifferent from the other leaf nodes, `MetastoreRelation` and `SimpleCatalogRelation` have a pre-defined `alias`, which is used to change the qualifier of the node. However, based on the existing alias handling, alias should be put in `SubqueryAlias`.\n\nThis PR is to separate alias handling from `MetastoreRelation` and `SimpleCatalogRelation` to make it consistent with the other nodes. It simplifies the signature and conversion to a `BaseRelation`.\n\nFor example, below is an example query for `MetastoreRelation`,  which is converted to a `LogicalRelation`:\n```SQL\nSELECT tmp.a + 1 FROM test_parquet_ctas tmp WHERE tmp.a > 2\n```\n\nBefore changes, the analyzed plan is\n```\n== Analyzed Logical Plan ==\n(a + 1): int\nProject [(a#951 + 1) AS (a + 1)#952]\n+- Filter (a#951 > 2)\n   +- SubqueryAlias tmp\n      +- Relation[a#951] parquet\n```\nAfter changes, the analyzed plan becomes\n```\n== Analyzed Logical Plan ==\n(a + 1): int\nProject [(a#951 + 1) AS (a + 1)#952]\n+- Filter (a#951 > 2)\n   +- SubqueryAlias tmp\n      +- SubqueryAlias test_parquet_ctas\n         +- Relation[a#951] parquet\n```\n\n**Note: the optimized plans are the same.**\n\nFor `SimpleCatalogRelation`, the existing code always generates two Subqueries. Thus, no change is needed.\n\n#### How was this patch tested?\nAdded test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14053 from gatorsmile/removeAliasFromMetastoreRelation.","distinct":false,"url":"https://api.github.com/repos/mengxr/spark/commits/42279bff686f9808ec7a9e8f4da95c717edc6026"}]}
{"actor":{"display_login":"mengxr"}}