{"push_id":1094112877,"size":1000,"distinct_size":1,"ref":"refs/heads/master","head":"ccbf731c07664c088dffd24d9f7aa3879c6d8035","before":"08c781ca672820be9ba32838bbe40d2643c4bde4","commits":[{"sha":"9e01fe2ed1e834710f4ee6a02864ab0fcc528fef","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-13535][SQL] Fix Analysis Exceptions when Using Backticks in Transform Clause\n\n#### What changes were proposed in this pull request?\n```SQL\nFROM\n(FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string)) t\nSELECT thing1 + 1\n```\nThis query returns an analysis error, like:\n```\nFailed to analyze query: org.apache.spark.sql.AnalysisException: cannot resolve '`thing1`' given input columns: [`thing1`, thing2]; line 3 pos 7\n'Project [unresolvedalias(('thing1 + 1), None)]\n+- SubqueryAlias t\n   +- ScriptTransformation [key#2,value#3], cat, [`thing1`#6,thing2#7], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim,\t)),List((field.delim,\t)),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),false)\n      +- SubqueryAlias test\n         +- Project [_1#0 AS key#2,_2#1 AS value#3]\n            +- LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3],[4,4],[5,5]]\n```\n\nThe backpacks of \\`thing1\\` should be cleaned before entering Parser/Analyzer. This PR fixes this issue.\n\n#### How was this patch tested?\n\nAdded a test case and modified an existing test case\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11415 from gatorsmile/scriptTransform.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9e01fe2ed1e834710f4ee6a02864ab0fcc528fef"},{"sha":"b5a59a0fe2ea703ce2712561e7b9044f772660a2","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13601] call failure callbacks before writer.close()\n\n## What changes were proposed in this pull request?\n\nIn order to tell OutputStream that the task has failed or not, we should call the failure callbacks BEFORE calling writer.close().\n\n## How was this patch tested?\n\nAdded new unit tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11450 from davies/callback.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b5a59a0fe2ea703ce2712561e7b9044f772660a2"},{"sha":"e2780ce8252ded93a695125c0a745d8b93193cca","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-13574] [SQL] Add benchmark to measure string dictionary decode.\n\n## What changes were proposed in this pull request?\n\nAlso updated the other benchmarks when the default to use vectorized decode was flipped.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #11454 from nongli/benchmark.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e2780ce8252ded93a695125c0a745d8b93193cca"},{"sha":"4dd24811d9035c52c5965fca2fc6431aac6963fc","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13594][SQL] remove typed operations(e.g. map, flatMap) from python DataFrame\n\n## What changes were proposed in this pull request?\n\nRemove `map`, `flatMap`, `mapPartitions` from python DataFrame, to prepare for Dataset API in the future.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11445 from cloud-fan/python-clean.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/4dd24811d9035c52c5965fca2fc6431aac6963fc"},{"sha":"6250cf1e00f6b0bacca73ad785fa402f59bd6232","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-13528][SQL] Make the short names of compression codecs consistent in ParquetRelation\n\n## What changes were proposed in this pull request?\nThis pr to make the short names of compression codecs in `ParquetRelation` consistent against other ones. This pr comes from #11324.\n\n## How was this patch tested?\nAdd more tests in `TextSuite`.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #11408 from maropu/SPARK-13528.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/6250cf1e00f6b0bacca73ad785fa402f59bd6232"},{"sha":"b60b8137992641b9193e57061aa405f908b0f267","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13617][SQL] remove unnecessary GroupingAnalytics trait\n\n## What changes were proposed in this pull request?\n\nThe `trait GroupingAnalytics` only has one implementation, it's an unnecessary abstraction. This PR removes it, and does some code simplification when resolving `GroupingSet`.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11469 from cloud-fan/groupingset.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b60b8137992641b9193e57061aa405f908b0f267"},{"sha":"9c274ac4a6a190a436180ffb0db0c926d6abd0b0","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-13627][SQL][YARN] Fix simple deprecation warnings.\n\n## What changes were proposed in this pull request?\n\nThis PR aims to fix the following deprecation warnings.\n  * MethodSymbolApi.paramss--> paramLists\n  * AnnotationApi.tpe -> tree.tpe\n  * BufferLike.readOnly -> toList.\n  * StandardNames.nme -> termNames\n  * scala.tools.nsc.interpreter.AbstractFileClassLoader -> scala.reflect.internal.util.AbstractFileClassLoader\n  * TypeApi.declarations-> decls\n\n## How was this patch tested?\n\nCheck the compile build log and pass the tests.\n```\n./build/sbt\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11479 from dongjoon-hyun/SPARK-13627.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9c274ac4a6a190a436180ffb0db0c926d6abd0b0"},{"sha":"f87ce0504ea0697969ac3e67690c78697b76e94a","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13616][SQL] Let SQLBuilder convert logical plan without a project on top of it\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13616\n\n## What changes were proposed in this pull request?\n\nIt is possibly that a logical plan has been removed `Project` from the top of it. Or the plan doesn't has a top `Project` from the beginning because it is not necessary. Currently the `SQLBuilder` can't convert such plans back to SQL. This change is to add this feature.\n\n## How was this patch tested?\n\nA test is added to `LogicalPlanToSQLSuite`.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11466 from viirya/sqlbuilder-notopselect.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/f87ce0504ea0697969ac3e67690c78697b76e94a"},{"sha":"56e3d007151d77c0ef2d84335adb8614c48c092a","author":{"email":"1d6ff74e691c4679535b09af3c8ed5d448e8a009@apache.org","name":"Devaraj K"},"message":"[SPARK-13621][CORE] TestExecutor.scala needs to be moved to test package\n\nMoved TestExecutor.scala from src to test package and removed the unused file TestClient.scala.\n\nAuthor: Devaraj K <devaraj@apache.org>\n\nCloses #11474 from devaraj-kavali/SPARK-13621.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/56e3d007151d77c0ef2d84335adb8614c48c092a"},{"sha":"1085bd862a3f69e14e00cec11077e54ab153098b","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13635] [SQL] Enable LimitPushdown optimizer rule because we have whole-stage codegen for Limit\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13635\n\n## What changes were proposed in this pull request?\n\nLimitPushdown optimizer rule has been disabled due to no whole-stage codegen for Limit. As we have whole-stage codegen for Limit now, we should enable it.\n\n## How was this patch tested?\n\nAs we only re-enable LimitPushdown optimizer rule, no need to add new tests for it.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11483 from viirya/enable-limitpushdown.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/1085bd862a3f69e14e00cec11077e54ab153098b"},{"sha":"7b25dc7b7e5a098552c0d640eee132b83d42db56","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13466] [SQL] Remove projects that become redundant after column pruning rule\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13466\n\n## What changes were proposed in this pull request?\n\nWith column pruning rule in optimizer, some Project operators will become redundant. We should remove these redundant Projects.\n\nFor an example query:\n\n    val input = LocalRelation('key.int, 'value.string)\n\n    val query =\n      Project(Seq($\"x.key\", $\"y.key\"),\n        Join(\n          SubqueryAlias(\"x\", input),\n          BroadcastHint(SubqueryAlias(\"y\", input)), Inner, None))\n\nAfter the first run of column pruning, it would like:\n\n    Project(Seq($\"x.key\", $\"y.key\"),\n      Join(\n        Project(Seq($\"x.key\"), SubqueryAlias(\"x\", input)),\n        Project(Seq($\"y.key\"),      <-- inserted by the rule\n        BroadcastHint(SubqueryAlias(\"y\", input))),\n        Inner, None))\n\nActually we don't need the outside Project now. This patch will remove it:\n\n    Join(\n      Project(Seq($\"x.key\"), SubqueryAlias(\"x\", input)),\n      Project(Seq($\"y.key\"),\n      BroadcastHint(SubqueryAlias(\"y\", input))),\n      Inner, None)\n\n## How was the this patch tested?\n\nUnit test is added into ColumnPruningSuite.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11341 from viirya/remove-redundant-project.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/7b25dc7b7e5a098552c0d640eee132b83d42db56"},{"sha":"02b7677e9584f5ccd68869abdb0bf980dc847ce1","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[HOT-FIX] Recover some deprecations for 2.10 compatibility.\n\n## What changes were proposed in this pull request?\n\n#11479 [SPARK-13627] broke 2.10 compatibility: [2.10-Build](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-scala-2.10/292/console)\nAt this moment, we need to support both 2.10 and 2.11.\nThis PR recovers some deprecated methods which were replace by [SPARK-13627].\n\n## How was this patch tested?\n\nJenkins build: Both 2.10, 2.11.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11488 from dongjoon-hyun/hotfix_compatibility_with_2.10.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/02b7677e9584f5ccd68869abdb0bf980dc847ce1"},{"sha":"e97fc7f176f8bf501c9b3afd8410014e3b0e1602","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-13423][WIP][CORE][SQL][STREAMING] Static analysis fixes for 2.x\n\n## What changes were proposed in this pull request?\n\nMake some cross-cutting code improvements according to static analysis. These are individually up for discussion since they exist in separate commits that can be reverted. The changes are broadly:\n\n- Inner class should be static\n- Mismatched hashCode/equals\n- Overflow in compareTo\n- Unchecked warnings\n- Misuse of assert, vs junit.assert\n- get(a) + getOrElse(b) -> getOrElse(a,b)\n- Array/String .size -> .length (occasionally, -> .isEmpty / .nonEmpty) to avoid implicit conversions\n- Dead code\n- tailrec\n- exists(_ == ) -> contains find + nonEmpty -> exists filter + size -> count\n- reduce(_+_) -> sum map + flatten -> map\n\nThe most controversial may be .size -> .length simply because of its size. It is intended to avoid implicits that might be expensive in some places.\n\n## How was the this patch tested?\n\nExisting Jenkins unit tests.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #11292 from srowen/SPARK-13423.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/e97fc7f176f8bf501c9b3afd8410014e3b0e1602"},{"sha":"b5f02d6743ecb1633b7b13382f76cb8bfc2aa95c","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-13583][CORE][STREAMING] Remove unused imports and add checkstyle rule\n\n## What changes were proposed in this pull request?\n\nAfter SPARK-6990, `dev/lint-java` keeps Java code healthy and helps PR review by saving much time.\nThis issue aims remove unused imports from Java/Scala code and add `UnusedImports` checkstyle rule to help developers.\n\n## How was this patch tested?\n```\n./dev/lint-java\n./build/sbt compile\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11438 from dongjoon-hyun/SPARK-13583.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/b5f02d6743ecb1633b7b13382f76cb8bfc2aa95c"},{"sha":"645c3a85e2029928d37ec2de9ef5a2d884620b9b","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-13423][HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10\n\n## What changes were proposed in this pull request?\n\nFixes compile problem due to inadvertent use of `Option.contains`, only in Scala 2.11. The change should have been to replace `Option.exists(_ == x)` with `== Some(x)`. Replacing exists with contains only makes sense for collections. Replacing use of `Option.exists` still makes sense though as it's misleading.\n\n## How was this patch tested?\n\nJenkins tests / compilation\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #11493 from srowen/SPARK-13423.2.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/645c3a85e2029928d37ec2de9ef5a2d884620b9b"},{"sha":"70f6f9649bdb13b6745473b7edc4cd06b10f99d2","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-13013][DOCS] Replace example code in mllib-clustering.md using include_example\n\nReplace example code in mllib-clustering.md using include_example\nhttps://issues.apache.org/jira/browse/SPARK-13013\n\nThe example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.\n\nGoal is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.\n`{% include_example scala/org/apache/spark/examples/mllib/KMeansExample.scala %}`\nJekyll will find `examples/src/main/scala/org/apache/spark/examples/mllib/KMeansExample.scala` and pick code blocks marked \"example\" and replace code block in\n`{% highlight %}`\n in the markdown.\n\nSee more sub-tasks in parent ticket: https://issues.apache.org/jira/browse/SPARK-11337\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #11116 from keypointt/SPARK-13013.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/70f6f9649bdb13b6745473b7edc4cd06b10f99d2"},{"sha":"9a48c656eec8e47a55e5e0ecc6a7901c5be31566","author":{"email":"ec969b9b5c2f2a49a0d9175cec61f09554863a3d@hortonworks.com","name":"Steve Loughran"},"message":"[SPARK-13599][BUILD] remove transitive groovy dependencies from Hive\n\n## What changes were proposed in this pull request?\n\nModifies the dependency declarations of the all the hive artifacts, to explicitly exclude the groovy-all JAR.\n\nThis stops the groovy classes *and everything else in that uber-JAR* from getting into spark-assembly JAR.\n\n## How was this patch tested?\n\n1. Pre-patch build was made: `mvn clean install -Pyarn,hive,hive-thriftserver`\n1. spark-assembly expanded, observed to have the org.codehaus.groovy packages and JARs\n1. A maven dependency tree was created `mvn dependency:tree -Pyarn,hive,hive-thriftserver  -Dverbose > target/dependencies.txt`\n1. This text file examined to confirm that groovy was being imported as a dependency of `org.spark-project.hive`\n1. Patch applied\n1. Repeated step1: clean build of project with ` -Pyarn,hive,hive-thriftserver` set\n1. Examined created spark-assembly, verified no org.codehaus packages\n1. Verified that the maven dependency tree no longer references groovy\n\nNote also that the size of the assembly JAR was 181628646 bytes before this patch, 166318515 after —15MB smaller. That's a good metric of things being excluded\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\n\nCloses #11449 from steveloughran/fixes/SPARK-13599-groovy-dependency.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/9a48c656eec8e47a55e5e0ecc6a7901c5be31566"},{"sha":"511d4929c87ebf364b96bd46890628f736eaa026","author":{"email":"f97d5112937f779dff37a1cf186f37e361a6b63d@gmail.com","name":"JeremyNixon"},"message":"[SPARK-12877][ML] Add train-validation-split to pyspark\n\n## What changes were proposed in this pull request?\nThe changes proposed were to add train-validation-split to pyspark.ml.tuning.\n\n## How was the this patch tested?\nThis patch was tested through unit tests located in pyspark/ml/test.py.\n\nThis is my original work and I license it to Spark.\n\nAuthor: JeremyNixon <jnixon2@gmail.com>\n\nCloses #11335 from JeremyNixon/tvs_pyspark.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/511d4929c87ebf364b96bd46890628f736eaa026"},{"sha":"cf95d728c64f76e8b1065d7cacf1c3ad7769e935","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-13543][SQL] Support for specifying compression codec for Parquet/ORC via option()\n\n## What changes were proposed in this pull request?\n\nThis PR adds the support to specify compression codecs for both ORC and Parquet.\n\n## How was this patch tested?\n\nunittests within IDE and code style tests with `dev/run_tests`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #11464 from HyukjinKwon/SPARK-13543.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/cf95d728c64f76e8b1065d7cacf1c3ad7769e935"},{"sha":"ce58e99aae668d83cf69aa45993fb10a078394d9","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[MINOR][ML][DOC] Remove duplicated periods at the end of some sharedParam\n\n## What changes were proposed in this pull request?\nRemove duplicated periods at the end of some sharedParams in ScalaDoc, such as [here](https://github.com/apache/spark/pull/11344/files#diff-9edc669edcf2c0c7cf1efe4a0a57da80L367)\ncc mengxr srowen\n## How was this patch tested?\nDocuments change, no test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #11344 from yanboliang/shared-cleanup.","distinct":false,"url":"https://api.github.com/repos/cloud-fan/spark/commits/ce58e99aae668d83cf69aa45993fb10a078394d9"}]}