{"push_id":1132963141,"size":47,"distinct_size":47,"ref":"refs/heads/master","head":"019afd9c78a9f40e1d07f0a74868010206e90ed5","before":"c875d81a3de3f209b9eb03adf96b7c740b2c7b52","commits":[{"sha":"698ef762f80cf4c84bc7b7cf083aa97d44b87170","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-14269][SCHEDULER] Eliminate unnecessary submitStage() call.\n\n## What changes were proposed in this pull request?\n\nCurrently a method `submitStage()` for waiting stages is called on every iteration of the event loop in `DAGScheduler` to submit all waiting stages, but most of them are not necessary because they are not related to Stage status.\nThe case we should try to submit waiting stages is only when their parent stages are successfully completed.\n\nThis elimination can improve `DAGScheduler` performance.\n\n## How was this patch tested?\n\nAdded some checks and other existing tests, and our projects.\n\nWe have a project bottle-necked by `DAGScheduler`, having about 2000 stages.\n\nBefore this patch the almost all execution time in `Driver` process was spent to process `submitStage()` of `dag-scheduler-event-loop` thread but after this patch the performance was improved as follows:\n\n|        | total execution time | `dag-scheduler-event-loop` thread time | `submitStage()` |\n|--------|---------------------:|---------------------------------------:|----------------:|\n| Before |              760 sec |                                710 sec |         667 sec |\n| After  |              440 sec |                                 14 sec |          10 sec |\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #12060 from ueshin/issues/SPARK-14269.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/698ef762f80cf4c84bc7b7cf083aa97d44b87170"},{"sha":"9c297df3d4d5fa4bbfdffdaad15f362586db384b","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[MINOR] [PYSPARK] [EXAMPLES] Changed examples to use SparkSession.sparkContext instead of _sc\n\n## What changes were proposed in this pull request?\n\nSome PySpark examples need a SparkContext and get it by accessing _sc directly from the session.  These examples should use the provided property `sparkContext` in `SparkSession` instead.\n\n## How was this patch tested?\nRan modified examples\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #13303 from BryanCutler/pyspark-session-sparkContext-MINOR.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/9c297df3d4d5fa4bbfdffdaad15f362586db384b"},{"sha":"589cce93c821ac28e9090a478f6e7465398b7c30","author":{"name":"Gio Borje","email":"ab9d6223c603ca520cf1d254720a3e673a4cfe69@linkedin.com"},"message":"Log warnings for numIterations * miniBatchFraction < 1.0\n\n## What changes were proposed in this pull request?\n\nAdd a warning log for the case that `numIterations * miniBatchFraction <1.0` during gradient descent. If the product of those two numbers is less than `1.0`, then not all training examples will be used during optimization. To put this concretely, suppose that `numExamples = 100`, `miniBatchFraction = 0.2` and `numIterations = 3`. Then, 3 iterations will occur each sampling approximately 6 examples each. In the best case, each of the 6 examples are unique; hence 18/100 examples are used.\n\nThis may be counter-intuitive to most users and led to the issue during the development of another Spark  ML model: https://github.com/zhengruifeng/spark-libFM/issues/11. If a user actually does not require the training data set, it would be easier and more intuitive to use `RDD.sample`.\n\n## How was this patch tested?\n\n`build/mvn -DskipTests clean package` build succeeds\n\nAuthor: Gio Borje <gborje@linkedin.com>\n\nCloses #13265 from Hydrotoast/master.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/589cce93c821ac28e9090a478f6e7465398b7c30"},{"sha":"ee682fe293b47988056b540ee46ca49861309982","author":{"name":"Andrew Or","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@databricks.com"},"message":"[SPARK-15534][SPARK-15535][SQL] Truncate table fixes\n\n## What changes were proposed in this pull request?\n\nTwo changes:\n- When things fail, `TRUNCATE TABLE` just returns nothing. Instead, we should throw exceptions.\n- Remove `TRUNCATE TABLE ... COLUMN`, which was never supported by either Spark or Hive.\n\n## How was this patch tested?\nJenkins.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13302 from andrewor14/truncate-table.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/ee682fe293b47988056b540ee46ca49861309982"},{"sha":"527499b624e743583fe0f93ea0b487031891ac3a","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@questtec.nl"},"message":"[SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT plugin\n\n## What changes were proposed in this pull request?\nThe ANTLR4 SBT plugin has been moved from its own repo to one on bintray. The version was also changed from `0.7.10` to `0.7.11`. The latter actually broke our build (ihji has fixed this by also adding `0.7.10` and others to the bin-tray repo).\n\nThis PR upgrades the SBT-ANTLR4 plugin and ANTLR4 to their most recent versions (`0.7.11`/`4.5.3`). I have also removed a few obsolete build configurations.\n\n## How was this patch tested?\nManually running SBT/Maven builds.\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #13299 from hvanhovell/SPARK-15525.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/527499b624e743583fe0f93ea0b487031891ac3a"},{"sha":"06ed1fa3e45adfc11b0f615cb8b97c99fadc735f","author":{"name":"Sameer Agarwal","email":"73cebedf715c5415a2956d45e4fbb64de14b0dfd@databricks.com"},"message":"[SPARK-15533][SQL] Deprecate Dataset.explode\n\n## What changes were proposed in this pull request?\n\nThis patch deprecates `Dataset.explode` and documents appropriate workarounds to use `flatMap()` or `functions.explode()` instead.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #13312 from sameeragarwal/deprecate.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/06ed1fa3e45adfc11b0f615cb8b97c99fadc735f"},{"sha":"06bae8af17d9478c889d206a4556a697b5d629e7","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-15439][SPARKR] Failed to run unit test in SparkR\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\nThere are some failures when running SparkR unit tests.\nIn this PR, I fixed two of these failures in test_context.R and test_sparkSQL.R\nThe first one is due to different masked name. I added missed names in the expected arrays.\nThe second one is because one PR removed the logic of a previous fix of missing subset method.\n\nThe file privilege issue is still there. I am debugging it. SparkR shell can run the test case successfully.\ntest_that(\"pipeRDD() on RDDs\", {\n  actual <- collect(pipeRDD(rdd, \"more\"))\nWhen using run-test script, it complains no such directories as below:\ncannot open file '/tmp/Rtmp4FQbah/filee2273f9d47f7': No such file or directory\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\nManually test it\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #13284 from wangmiao1981/R.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/06bae8af17d9478c889d206a4556a697b5d629e7"},{"sha":"dfc9fc02ccbceb09213c394177d54b9ca56b6f24","author":{"name":"Imran Rashid","email":"63f21f1cb7b694d385ee53376e18c9c9be6c6f98@cloudera.com"},"message":"[SPARK-10372] [CORE] basic test framework for entire spark scheduler\n\nThis is a basic framework for testing the entire scheduler.  The tests this adds aren't very interesting -- the point of this PR is just to setup the framework, to keep the initial change small, but it can be built upon to test more features (eg., speculation, killing tasks, blacklisting, etc.).\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #8559 from squito/SPARK-10372-scheduler-integs.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/dfc9fc02ccbceb09213c394177d54b9ca56b6f24"},{"sha":"361ebc282b2d09dc6dcf21419a53c5c617b1b6bd","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15543][SQL] Rename DefaultSources to make them more self-describing\n\n## What changes were proposed in this pull request?\nThis patch renames various DefaultSources to make their names more self-describing. The choice of \"DefaultSource\" was from the days when we did not have a good way to specify short names.\n\nThey are now named:\n- LibSVMFileFormat\n- CSVFileFormat\n- JdbcRelationProvider\n- JsonFileFormat\n- ParquetFileFormat\n- TextFileFormat\n\nBackward compatibility is maintained through aliasing.\n\n## How was this patch tested?\nUpdated relevant test cases too.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13311 from rxin/SPARK-15543.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/361ebc282b2d09dc6dcf21419a53c5c617b1b6bd"},{"sha":"53d4abe9e996e53c1bdcd5ac4cb8cbf08b9ec8b5","author":{"name":"Bo Meng","email":"bebef4876565d254ea7bafca8629fd3eeb8eb055@hotmail.com"},"message":"[SPARK-15537][SQL] fix dir delete issue\n\n## What changes were proposed in this pull request?\n\nFor some of the test cases, e.g. `OrcSourceSuite`, it will create temp folders and temp files inside them. But after tests finish, the folders are not removed. This will cause lots of temp files created and space occupied, if we keep running the test cases.\n\nThe reason is dir.delete() won't work if dir is not empty. We need to recursively delete the content before deleting the folder.\n\n## How was this patch tested?\n\nManually checked the temp folder to make sure the temp files were deleted.\n\nAuthor: Bo Meng <mengbo@hotmail.com>\n\nCloses #13304 from bomeng/SPARK-15537.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/53d4abe9e996e53c1bdcd5ac4cb8cbf08b9ec8b5"},{"sha":"e451f7f0c3857cdbbca98e66928a97f797f2fc6b","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-15492][ML][DOC] Binarization scala example copy & paste to spark-shell error\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\nThe Binarization scala example val dataFrame : Dataframe = spark.createDataFrame(data).toDF(\"label\", \"feature\"), which can't be pasted in the spark-shell as Dataframe is not imported. Compared with other examples, this explicit type is not required.\n\nSo I removed Dataframe in the code.\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\nManually tested\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #13266 from wangmiao1981/unit.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/e451f7f0c3857cdbbca98e66928a97f797f2fc6b"},{"sha":"6d506c9ae9a2519d1a93e788ae5316d4f942d35d","author":{"name":"Villu Ruusmann","email":"788d50e01b5fa791ba3b57b72ea9a03a6c129fa9@gmail.com"},"message":"[SPARK-15523][ML][MLLIB] Update JPMML to 1.2.15\n\n## What changes were proposed in this pull request?\n\nSee https://issues.apache.org/jira/browse/SPARK-15523\n\nThis PR replaces PR #13293. It's isolated to a new branch, and contains some more squashed changes.\n\n## How was this patch tested?\n\n1. Executed `mvn clean package` in `mllib` directory\n2. Executed `dev/test-dependencies.sh --replace-manifest` in the root directory.\n\nAuthor: Villu Ruusmann <villu.ruusmann@gmail.com>\n\nCloses #13297 from vruusmann/update-jpmml.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/6d506c9ae9a2519d1a93e788ae5316d4f942d35d"},{"sha":"c76457c8e422ce6fbf72a8fe5db94565783b12d0","author":{"name":"felixcheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-10903][SPARKR] R - Simplify SQLContext method signatures and use a singleton\n\nEliminate the need to pass sqlContext to method since it is a singleton - and we don't want to support multiple contexts in a R session.\n\nChanges are done in a back compat way with deprecation warning added. Method signature for S3 methods are added in a concise, clean approach such that in the next release the deprecated signature can be taken out easily/cleanly (just delete a few lines per method).\n\nCustom method dispatch is implemented to allow for multiple JVM reference types that are all 'jobj' in R and to avoid having to add 30 new exports.\n\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #9192 from felixcheung/rsqlcontext.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/c76457c8e422ce6fbf72a8fe5db94565783b12d0"},{"sha":"01b350a4f7c17d6516b27b6cd27ba8390834d40c","author":{"name":"Steve Loughran","email":"504d4a581a50fc7a86a0698985fb52745234a417@hortonworks.com"},"message":"[SPARK-13148][YARN] document zero-keytab Oozie application launch; add diagnostics\n\nThis patch provides detail on what to do for keytabless Oozie launches of spark apps, and adds some debug-level diagnostics of what credentials have been submitted\n\nAuthor: Steve Loughran <stevel@hortonworks.com>\nAuthor: Steve Loughran <stevel@apache.org>\n\nCloses #11033 from steveloughran/stevel/feature/SPARK-13148-oozie.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/01b350a4f7c17d6516b27b6cd27ba8390834d40c"},{"sha":"2b1ac6cea882246ef0e655bb2c134ef1656a5068","author":{"name":"Andrew Or","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@databricks.com"},"message":"[SPARK-15539][SQL] DROP TABLE throw exception if table doesn't exist\n\n## What changes were proposed in this pull request?\n\nSame as #13302, but for DROP TABLE.\n\n## How was this patch tested?\n\n`DDLSuite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13307 from andrewor14/drop-table.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/2b1ac6cea882246ef0e655bb2c134ef1656a5068"},{"sha":"594a1bf200fea8d6bcf25839a49186f66f922bc8","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark\n\n## What changes were proposed in this pull request?\n\nAlso sets confs in the underlying sc when using SparkSession.builder.getOrCreate(). This is a bug-fix from a post-merge comment in https://github.com/apache/spark/pull/13289\n\n## How was this patch tested?\n\nPython doc-tests.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #13309 from ericl/spark-15520-1.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/594a1bf200fea8d6bcf25839a49186f66f922bc8"},{"sha":"0f61d6efb45b9ee94fa663f67c4489fbdae2eded","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15552][SQL] Remove unnecessary private[sql] methods in SparkSession\n\n## What changes were proposed in this pull request?\nSparkSession has a list of unnecessary private[sql] methods. These methods cause some trouble because private[sql] doesn't apply in Java. In the cases that they are easy to remove, we can simply remove them. This patch does that.\n\nAs part of this pull request, I also replaced a bunch of protected[sql] with private[sql], to tighten up visibility.\n\n## How was this patch tested?\nUpdated test cases to reflect the changes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13319 from rxin/SPARK-15552.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/0f61d6efb45b9ee94fa663f67c4489fbdae2eded"},{"sha":"b0a03feef2cf4daa7642ec7f4dc479dbd473b581","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-15457][MLLIB][ML] Eliminate some warnings from MLlib about deprecations\n\n## What changes were proposed in this pull request?\n\nSeveral classes and methods have been deprecated and are creating lots of build warnings in branch-2.0. This issue is to identify and fix those items:\n* WithSGD classes: Change to make class not deprecated, object deprecated, and public class constructor deprecated. Any public use will require a deprecated API. We need to keep a non-deprecated private API since we cannot eliminate certain uses: Python API, streaming algs, and examples.\n  * Use in PythonMLlibAPI: Change to using private constructors\n  * Streaming algs: No warnings after we un-deprecate the classes\n  * Examples: Deprecate or change ones which use deprecated APIs\n* MulticlassMetrics fields (precision, etc.)\n* LinearRegressionSummary.model field\n\n## How was this patch tested?\n\nExisting tests.  Checked for warnings manually.\n\nAuthor: Sean Owen <sowen@cloudera.com>\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #13314 from jkbradley/warning-cleanups.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/b0a03feef2cf4daa7642ec7f4dc479dbd473b581"},{"sha":"b5859e0bb8cc147858cb28d8bdb5ca3b4a2cec77","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-13445][SQL] Improves error message and add test coverage for Window function\n\n## What changes were proposed in this pull request?\n\nAdd more verbose error message when order by clause is missed when using Window function.\n\n## How was this patch tested?\n\nUnit test.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #13333 from clockfly/spark-13445.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/b5859e0bb8cc147858cb28d8bdb5ca3b4a2cec77"},{"sha":"fe6de16f781ff659b34e0ddda427d371d3d94536","author":{"name":"Sameer Agarwal","email":"73cebedf715c5415a2956d45e4fbb64de14b0dfd@databricks.com"},"message":"[SPARK-8428][SPARK-13850] Fix integer overflows in TimSort\n\n## What changes were proposed in this pull request?\n\nThis patch fixes a few integer overflows in `UnsafeSortDataFormat.copyRange()` and `ShuffleSortDataFormat copyRange()` that seems to be the most likely cause behind a number of `TimSort` contract violation errors seen in Spark 2.0 and Spark 1.6 while sorting large datasets.\n\n## How was this patch tested?\n\nAdded a test in `ExternalSorterSuite` that instantiates a large array of the form of [150000000, 150000001, 150000002, ...., 300000000, 0, 1, 2, ..., 149999999] that triggers a `copyRange` in `TimSort.mergeLo` or `TimSort.mergeHi`. Note that the input dataset should contain at least 268.43 million rows with a certain data distribution for an overflow to occur.\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #13336 from sameeragarwal/timsort-bug.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/fe6de16f781ff659b34e0ddda427d371d3d94536"}]}
{"actor":{"display_login":"bomeng"}}