{"push_id":1240308884,"size":61,"distinct_size":2,"ref":"refs/heads/SPARK-16779","head":"e4d8452f540140b7ade22a472b5334dae88df559","before":"df67cdec42d52f87c55a52b2d1ceb785ee0bf443","commits":[{"sha":"bbc247548ac6faeca15afc05c266cee37ef13416","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-16748][SQL] SparkExceptions during planning should not wrapped in TreeNodeException\n\n## What changes were proposed in this pull request?\nWe do not want SparkExceptions from job failures in the planning phase to create TreeNodeException. Hence do not wrap SparkException in TreeNodeException.\n\n## How was this patch tested?\nNew unit test\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14395 from tdas/SPARK-16748.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/bbc247548ac6faeca15afc05c266cee37ef13416"},{"sha":"0dc4310b470c7e4355c0da67ca3373c3013cc9dd","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16694][CORE] Use for/foreach rather than map for Unit expressions whose side effects are required\n\n## What changes were proposed in this pull request?\n\nUse foreach/for instead of map where operation requires execution of body, not actually defining a transformation\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14332 from srowen/SPARK-16694.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/0dc4310b470c7e4355c0da67ca3373c3013cc9dd"},{"sha":"bce354c1d4e2b97b1159913085e9883a26bc605a","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16696][ML][MLLIB] destroy KMeans bcNewCenters when loop finished and update code where should release unused broadcast/RDD in proper time\n\n## What changes were proposed in this pull request?\n\nupdate unused broadcast in KMeans/Word2Vec,\nuse destroy(false) to release memory in time.\n\nand several place destroy() update to destroy(false) so that it will be async-called,\nit will better than blocking called.\n\nand update bcNewCenters in KMeans to make it destroy in correct time.\nI use a list to store all historical `bcNewCenters` generated in each loop iteration and delay them to release at the end of loop.\n\nfix TODO in `BisectingKMeans.run` \"unpersist old indices\",\nImplements the pattern \"persist current step RDD, and unpersist previous one\" in the loop iteration.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14333 from WeichenXu123/broadvar_unpersist_to_destroy.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/bce354c1d4e2b97b1159913085e9883a26bc605a"},{"sha":"a6290e51e402e8434d6207d553db1f551e714fde","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-16800][EXAMPLES][ML] Fix Java examples that fail to run due to exception\n\n## What changes were proposed in this pull request?\nSome Java examples are using mllib.linalg.Vectors instead of ml.linalg.Vectors and causes an exception when run.  Also there are some Java examples that incorrectly specify data types in the schema, also causing an exception.\n\n## How was this patch tested?\nRan corrected examples locally\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #14405 from BryanCutler/java-examples-ml.Vectors-fix-SPARK-16800.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/a6290e51e402e8434d6207d553db1f551e714fde"},{"sha":"957a8ab3743521850fb1c0106c37c5d3997b9e56","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-16818] Exchange reuse incorrectly reuses scans over different sets of partitions\n\n## What changes were proposed in this pull request?\n\nThis fixes a bug wherethe file scan operator does not take into account partition pruning in its implementation of `sameResult()`. As a result, executions may be incorrect on self-joins over the same base file relation.\n\nThe patch here is minimal, but we should reconsider relying on `metadata` for implementing sameResult() in the future, as string representations may not be uniquely identifying.\n\ncc rxin\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #14425 from ericl/spark-16818.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/957a8ab3743521850fb1c0106c37c5d3997b9e56"},{"sha":"7c27d075c39ebaf3e762284e2536fe7be0e3da87","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16812] Open up SparkILoop.getAddedJars\n\n## What changes were proposed in this pull request?\nThis patch makes SparkILoop.getAddedJars a public developer API. It is a useful function to get the list of jars added.\n\n## How was this patch tested?\nN/A - this is a simple visibility change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14417 from rxin/SPARK-16812.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/7c27d075c39ebaf3e762284e2536fe7be0e3da87"},{"sha":"064d91ff7342002414d3274694a8e2e37f154986","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16813][SQL] Remove private[sql] and private[spark] from catalyst package\n\n## What changes were proposed in this pull request?\nThe catalyst package is meant to be internal, and as a result it does not make sense to mark things as private[sql] or private[spark]. It simply makes debugging harder when Spark developers need to inspect the plans at runtime.\n\nThis patch removes all private[sql] and private[spark] visibility modifiers in org.apache.spark.sql.catalyst.\n\n## How was this patch tested?\nN/A - just visibility changes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14418 from rxin/SPARK-16813.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/064d91ff7342002414d3274694a8e2e37f154986"},{"sha":"301fb0d7236eb55d53c9cd60804a2d755b4ad3b2","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16731][SQL] use StructType in CatalogTable and remove CatalogColumn\n\n## What changes were proposed in this pull request?\n\n`StructField` has very similar semantic with `CatalogColumn`, except that `CatalogColumn` use string to express data type. I think it's reasonable to use `StructType` as the `CatalogTable.schema` and remove `CatalogColumn`.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14363 from cloud-fan/column.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/301fb0d7236eb55d53c9cd60804a2d755b4ad3b2"},{"sha":"579fbcf3bd9717003025caecc0c0b85bcff7ac7f","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16805][SQL] Log timezone when query result does not match\n\n## What changes were proposed in this pull request?\nIt is useful to log the timezone when query result does not match, especially on build machines that have different timezone from AMPLab Jenkins.\n\n## How was this patch tested?\nThis is a test-only change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14413 from rxin/SPARK-16805.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/579fbcf3bd9717003025caecc0c0b85bcff7ac7f"},{"sha":"64d8f37c717cbc9c1c3649cae4c7cc4e628cd72d","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16726][SQL] Improve `Union/Intersect/Except` error messages on incompatible types\n\n## What changes were proposed in this pull request?\n\nCurrently, `UNION` queries on incompatible types show misleading error messages, i.e., `unresolved operator Union`. We had better show a more correct message. This will help users in the situation of [SPARK-16704](https://issues.apache.org/jira/browse/SPARK-16704).\n\n**Before**\n```scala\nscala> sql(\"select 1,2,3 union (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Union;\nscala> sql(\"select 1,2,3 intersect (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Intersect;\nscala> sql(\"select 1,2,3 except (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: unresolved operator 'Except;\n```\n\n**After**\n```scala\nscala> sql(\"select 1,2,3 union (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the second column of the second table;\nscala> sql(\"select 1,2,3 intersect (select 1,array(2),3)\")\norg.apache.spark.sql.AnalysisException: Intersect can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the second column of the second table;\nscala> sql(\"select 1,2,3 except (select array(1),array(2),3)\")\norg.apache.spark.sql.AnalysisException: Except can only be performed on tables with the compatible column types. ArrayType(IntegerType,false) <> IntegerType at the first column of the second table;\n```\n\n## How was this patch tested?\n\nPass the Jenkins test with a new test case.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14355 from dongjoon-hyun/SPARK-16726.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/64d8f37c717cbc9c1c3649cae4c7cc4e628cd72d"},{"sha":"2a0de7dc995844984d7dbb4238418967c6bbac70","author":{"name":"Shuai Lin","email":"7a96e9fba54b1a616c0c679715dfd11c0089fe85@gmail.com"},"message":"[SPARK-16485][DOC][ML] Remove useless latex in a log messge.\n\n## What changes were proposed in this pull request?\n\nRemoved useless latex in a log messge.\n\n## How was this patch tested?\n\nCheck generated scaladoc.\n\nAuthor: Shuai Lin <linshuai2012@gmail.com>\n\nCloses #14380 from lins05/fix-docs-formatting.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/2a0de7dc995844984d7dbb4238418967c6bbac70"},{"sha":"1e9b59b73bdb8aacf5a85e0eed29efc6485a3bc3","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-16778][SQL][TRIVIAL] Fix deprecation warning with SQLContext\n\n## What changes were proposed in this pull request?\n\nChange to non-deprecated constructor for SQLContext.\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #14406 from holdenk/SPARK-16778-fix-use-of-deprecated-SQLContext-constructor.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/1e9b59b73bdb8aacf5a85e0eed29efc6485a3bc3"},{"sha":"f93ad4fe7c9728c8dd67a8095de3d39fad21d03f","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16776][STREAMING] Replace deprecated API in KafkaTestUtils for 0.10.0.\n\n## What changes were proposed in this pull request?\n\nThis PR replaces the old Kafka API to 0.10.0 ones in `KafkaTestUtils`.\n\nThe change include:\n\n - `Producer` to `KafkaProducer`\n - Change configurations to equalvant ones. (I referred [here](http://kafka.apache.org/documentation.html#producerconfigs) for 0.10.0 and [here](http://kafka.apache.org/082/documentation.html#producerconfigs\n) for old, 0.8.2).\n\nThis PR will remove the build warning as below:\n\n```scala\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:71: class Producer in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.KafkaProducer instead.\n[WARNING]   private var producer: Producer[String, String] = _\n[WARNING]                         ^\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:181: class Producer in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.KafkaProducer instead.\n[WARNING]     producer = new Producer[String, String](new ProducerConfig(producerConfiguration))\n[WARNING]                    ^\n[WARNING] .../spark/streaming/kafka010/KafkaTestUtils.scala:181: class ProducerConfig in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.ProducerConfig instead.\n[WARNING]     producer = new Producer[String, String](new ProducerConfig(producerConfiguration))\n[WARNING]                                                 ^\n[WARNING] .../spark/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala:182: class KeyedMessage in package producer is deprecated: This class has been deprecated and will be removed in a future release. Please use org.apache.kafka.clients.producer.ProducerRecord instead.\n[WARNING]     producer.send(messages.map { new KeyedMessage[String, String](topic, _ ) }: _*)\n[WARNING]                                      ^\n[WARNING] four warnings found\n[WARNING] warning: [options] bootstrap class path not set in conjunction with -source 1.7\n[WARNING] 1 warning\n```\n\n## How was this patch tested?\n\nExisting tests that use `KafkaTestUtils` should cover this.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14416 from HyukjinKwon/SPARK-16776.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/f93ad4fe7c9728c8dd67a8095de3d39fad21d03f"},{"sha":"338a98d65c8efe0c41f39a8dddeab7040dcda125","author":{"name":"eyal farago","email":"74d0e2451c1814707187b75c7c67acf59bad9214"},"message":"[SPARK-16791][SQL] cast struct with timestamp field fails\n\n## What changes were proposed in this pull request?\na failing test case + fix to SPARK-16791 (https://issues.apache.org/jira/browse/SPARK-16791)\n\n## How was this patch tested?\nadded a failing test case to CastSuit, then fixed the Cast code and rerun the entire CastSuit\n\nAuthor: eyal farago <eyal farago>\nAuthor: Eyal Farago <eyal.farago@actimize.com>\n\nCloses #14400 from eyalfa/SPARK-16791_cast_struct_with_timestamp_field_fails.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/338a98d65c8efe0c41f39a8dddeab7040dcda125"},{"sha":"ab1e761f9691b41385e2ed2202c5a671c63c963d","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[SPARK-16774][SQL] Fix use of deprecated timestamp constructor & improve timezone handling\n\n## What changes were proposed in this pull request?\n\nRemoves the deprecated timestamp constructor and incidentally fixes the use which was using system timezone rather than the one specified when working near DST.\n\nThis change also causes the roundtrip tests to fail since it now actually uses all the timezones near DST boundaries where it didn't before.\n\nNote: this is only a partial the solution, longer term we should follow up with https://issues.apache.org/jira/browse/SPARK-16788 to avoid this problem & simplify our timezone handling code.\n\n## How was this patch tested?\n\nNew tests for two timezones added so even if user timezone happens to coincided with one, the other tests should still fail. Important note: this (temporarily) disables the round trip tests until we can fix the issue more thoroughly.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #14398 from holdenk/SPARK-16774-fix-use-of-deprecated-timestamp-constructor.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/ab1e761f9691b41385e2ed2202c5a671c63c963d"},{"sha":"03d46aafe561b03e25f4e25cf01e631c18dd827c","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-15869][STREAMING] Fix a potential NPE in StreamingJobProgressListener.getBatchUIData\n\n## What changes were proposed in this pull request?\n\nMoved `asScala` to a `map` to avoid NPE.\n\n## How was this patch tested?\n\nExisting unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14443 from zsxwing/SPARK-15869.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/03d46aafe561b03e25f4e25cf01e631c18dd827c"},{"sha":"2eedc00b04ef8ca771ff64c4f834c25f835f5f44","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16828][SQL] remove MaxOf and MinOf\n\n## What changes were proposed in this pull request?\n\nThese 2 expressions are not needed anymore after we have `Greatest` and `Least`. This PR removes them and related tests.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14434 from cloud-fan/minor1.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/2eedc00b04ef8ca771ff64c4f834c25f835f5f44"},{"sha":"5184df06b347f86776c8ac87415b8002a5942a35","author":{"name":"jiangxingbo","email":"8a33a35109e04ce846544d0ef94489482c135310@meituan.com"},"message":"[SPARK-16793][SQL] Set the temporary warehouse path to sc'conf in TestHive.\n\n## What changes were proposed in this pull request?\n\nWith SPARK-15034, we could use the value of spark.sql.warehouse.dir to set the warehouse location. In TestHive, we can now simply set the temporary warehouse path in sc's conf, and thus, param \"warehousePath\" could be removed.\n\n## How was this patch tested?\n\nexsiting testsuites.\n\nAuthor: jiangxingbo <jiangxingbo@meituan.com>\n\nCloses #14401 from jiangxb1987/warehousePath.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/5184df06b347f86776c8ac87415b8002a5942a35"},{"sha":"10e1c0e638774f5d746771b6dd251de2480f94eb","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-16734][EXAMPLES][SQL] Revise examples of all language bindings\n\n## What changes were proposed in this pull request?\n\nThis PR makes various minor updates to examples of all language bindings to make sure they are consistent with each other. Some typos and missing parts (JDBC example in Scala/Java/Python) are also fixed.\n\n## How was this patch tested?\n\nManually tested.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #14368 from liancheng/revise-examples.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/10e1c0e638774f5d746771b6dd251de2480f94eb"},{"sha":"a1ff72e1cce6f22249ccc4905e8cef30075beb2f","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16850][SQL] Improve type checking error message for greatest/least\n\n## What changes were proposed in this pull request?\nGreatest/least function does not have the most friendly error message for data types. This patch improves the error message to not show the Seq type, and use more human readable data types.\n\nBefore:\n```\norg.apache.spark.sql.AnalysisException: cannot resolve 'greatest(CAST(1.0 AS DECIMAL(2,1)), \"1.0\")' due to data type mismatch: The expressions should all have the same type, got GREATEST (ArrayBuffer(DecimalType(2,1), StringType)).; line 1 pos 7\n```\n\nAfter:\n```\norg.apache.spark.sql.AnalysisException: cannot resolve 'greatest(CAST(1.0 AS DECIMAL(2,1)), \"1.0\")' due to data type mismatch: The expressions should all have the same type, got GREATEST(decimal(2,1), string).; line 1 pos 7\n```\n\n## How was this patch tested?\nManually verified the output and also added unit tests to ConditionalExpressionSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #14453 from petermaxlee/SPARK-16850.","distinct":false,"url":"https://api.github.com/repos/holdenk/spark/commits/a1ff72e1cce6f22249ccc4905e8cef30075beb2f"}]}
{"actor":{"display_login":"holdenk"}}