{"push_id":998977709,"size":496,"distinct_size":81,"ref":"refs/heads/block-manager-cleanup","head":"7444171097832b040fe35a00ecdf0ae884290e48","before":"80d375a8962cb79a3a6da1f176f82c709e6a493c","commits":[{"sha":"cd81fc9e8652c07b84f0887a24d67381b4e605fa","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-12400][SHUFFLE] Avoid generating temp shuffle files for empty partitions\n\nThis problem lies in `BypassMergeSortShuffleWriter`, empty partition will also generate a temp shuffle file with several bytes. So here change to only create file when partition is not empty.\n\nThis problem only lies in here, no such issue in `HashShuffleWriter`.\n\nPlease help to review, thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #10376 from jerryshao/SPARK-12400.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/cd81fc9e8652c07b84f0887a24d67381b4e605fa"},{"sha":"021dafc6a05a31dc22c9f9110dedb47a1f913087","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-12026][MLLIB] ChiSqTest gets slower and slower over time when number of features is large\n\njira: https://issues.apache.org/jira/browse/SPARK-12026\n\nThe issue is valid as features.toArray.view.zipWithIndex.slice(startCol, endCol) becomes slower as startCol gets larger.\n\nI tested on local and the change can improve the performance and the running time was stable.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #10146 from hhbyyh/chiSq.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/021dafc6a05a31dc22c9f9110dedb47a1f913087"},{"sha":"20d8ef858af6e13db59df118b562ea33cba5464d","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-12703][MLLIB][DOC][PYTHON] Fixed pyspark.mllib.clustering.KMeans user guide example\n\nFixed WSSSE computeCost in Python mllib KMeans user guide example by using new computeCost method API in Python.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #10707 from jkbradley/kmeans-doc-fix.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/20d8ef858af6e13db59df118b562ea33cba5464d"},{"sha":"e2ae7bd046f6d8d6a375c2e81e5a51d7d78ca984","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12819] Deprecate TaskContext.isRunningLocally()\n\nWe've already removed local execution but didn't deprecate `TaskContext.isRunningLocally()`; we should deprecate it for 2.0.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10751 from JoshRosen/remove-local-exec-from-taskcontext.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/e2ae7bd046f6d8d6a375c2e81e5a51d7d78ca984"},{"sha":"962e9bcf94da6f5134983f2bf1e56c5cd84f2bf7","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-12756][SQL] use hash expression in Exchange\n\nThis PR makes bucketing and exchange share one common hash algorithm, so that we can guarantee the data distribution is same between shuffle and bucketed data source, which enables us to only shuffle one side when join a bucketed table and a normal one.\n\nThis PR also fixes the tests that are broken by the new hash behaviour in shuffle.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #10703 from cloud-fan/use-hash-expr-in-shuffle.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/962e9bcf94da6f5134983f2bf1e56c5cd84f2bf7"},{"sha":"8f13cd4cc8dcf638b178774418669a2e247d0652","author":{"email":"ced9a24e2e494bb60341a9f26a61a78e7a11abad@apache.org","name":"Jeff Zhang"},"message":"[SPARK-12707][SPARK SUBMIT] Remove submit python/R scripts through py…\n\n…spark/sparkR\n\nAuthor: Jeff Zhang <zjffdu@apache.org>\n\nCloses #10658 from zjffdu/SPARK-12707.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/8f13cd4cc8dcf638b178774418669a2e247d0652"},{"sha":"56cdbd654d54bf07a063a03a5c34c4165818eeb2","author":{"email":"060dd14c05a8ffa8f201883ef57d0676b7c9c04a@gmail.com","name":"Bryan Cutler"},"message":"[SPARK-9844][CORE] File appender race condition during shutdown\n\nWhen an Executor process is destroyed, the FileAppender that is asynchronously reading the stderr stream of the process can throw an IOException during read because the stream is closed.  Before the ExecutorRunner destroys the process, the FileAppender thread is flagged to stop.  This PR wraps the inputStream.read call of the FileAppender in a try/catch block so that if an IOException is thrown and the thread has been flagged to stop, it will safely ignore the exception.  Additionally, the FileAppender thread was changed to use Utils.tryWithSafeFinally to better log any exception that do occur.  Added unit tests to verify a IOException is thrown and logged if FileAppender is not flagged to stop, and that no IOException when the flag is set.\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #10714 from BryanCutler/file-appender-read-ioexception-SPARK-9844.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/56cdbd654d54bf07a063a03a5c34c4165818eeb2"},{"sha":"501e99ef0fbd2f2165095548fe67a3447ccbfc91","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-12784][UI] Fix Spark UI IndexOutOfBoundsException with dynamic allocation\n\nAdd `listener.synchronized` to get `storageStatusList` and `execInfo` atomically.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #10728 from zsxwing/SPARK-12784.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/501e99ef0fbd2f2165095548fe67a3447ccbfc91"},{"sha":"902667fd2766f0472a15851b1ed8fb5859593f97","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12771][SQL] Simplify CaseWhen code generation\n\nThe generated code for CaseWhen uses a control variable \"got\" to make sure we do not evaluate more branches once a branch is true. Changing that to generate just simple \"if / else\" would be slightly more efficient.\n\nThis closes #10737.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10755 from rxin/SPARK-12771.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/902667fd2766f0472a15851b1ed8fb5859593f97"},{"sha":"bcc7373f673d1a51b48fb95432ba5c4644dd5d23","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-12821][BUILD] Style checker should run when some configuration files for style are modified but any source files are not.\n\nWhen running the `run-tests` script, style checkers run only when any source files are modified but they should run when configuration files related to style are modified.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #10754 from sarutak/SPARK-12821.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/bcc7373f673d1a51b48fb95432ba5c4644dd5d23"},{"sha":"25782981cf58946dc7c186acadd2beec5d964461","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test\n\nThis patch significantly speeds up the BlockManagerSuite's \"SPARK-9591: getRemoteBytes from another location when Exception throw\" test, reducing the test time from 45s to ~250ms. The key change was to set `spark.shuffle.io.maxRetries` to 0 (the code previously set `spark.network.timeout` to `2s`, but this didn't make a difference because the slowdown was not due to this timeout).\n\nAlong the way, I also cleaned up the way that we handle SparkConf in BlockManagerSuite: previously, each test would mutate a shared SparkConf instance, while now each test gets a fresh SparkConf.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #10759 from JoshRosen/SPARK-12174.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/25782981cf58946dc7c186acadd2beec5d964461"},{"sha":"cc7af86afd3e769d1e2a581f31bb3db5a3d0229f","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-12813][SQL] Eliminate serialization for back to back operations\n\nThe goal of this PR is to eliminate unnecessary translations when there are back-to-back `MapPartitions` operations.  In order to achieve this I also made the following simplifications:\n\n - Operators no longer have hold encoders, instead they have only the expressions that they need.  The benefits here are twofold: the expressions are visible to transformations so go through the normal resolution/binding process.  now that they are visible we can change them on a case by case basis.\n - Operators no longer have type parameters.  Since the engine is responsible for its own type checking, having the types visible to the complier was an unnecessary complication.  We still leverage the scala compiler in the companion factory when constructing a new operator, but after this the types are discarded.\n\nDeferred to a follow up PR:\n - Remove as much of the resolution/binding from Dataset/GroupedDataset as possible. We should still eagerly check resolution and throw an error though in the case of mismatches for an `as` operation.\n - Eliminate serializations in more cases by adding more cases to `EliminateSerialization`\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #10747 from marmbrus/encoderExpressions.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/cc7af86afd3e769d1e2a581f31bb3db5a3d0229f"},{"sha":"32cca933546b4aaf0fc040b9cfd1a5968171b423","author":{"email":"9d2446cf15e9c733109bd67e0b3993aaff884674@oss.nttdata.co.jp","name":"Koyo Yoshida"},"message":"[SPARK-12708][UI] Sorting task error in Stages Page when yarn mode.\n\nIf sort column contains slash(e.g. \"Executor ID / Host\") when yarn mode,sort fail with following message.\n\n![spark-12708](https://cloud.githubusercontent.com/assets/6679275/12193320/80814f8c-b62a-11e5-9914-7bf3907029df.png)\n\nＩt's similar to SPARK-4313 .\n\nAuthor: root <root@R520T1.(none)>\nAuthor: Koyo Yoshida <koyo0615@gmail.com>\n\nCloses #10663 from yoshidakuy/SPARK-12708.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/32cca933546b4aaf0fc040b9cfd1a5968171b423"},{"sha":"591c88c9e2a6c2e2ca84f1b66c635f198a16d112","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12829] Turn Java style checker on\n\nIt was previously turned off because there was a problem with a pull request. We should turn it on now.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10763 from rxin/SPARK-12829.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/591c88c9e2a6c2e2ca84f1b66c635f198a16d112"},{"sha":"fe7246fea67e1d71fba679dee3eb2c7386b4f4e2","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12830] Java style: disallow trailing whitespaces.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10764 from rxin/SPARK-12830.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/fe7246fea67e1d71fba679dee3eb2c7386b4f4e2"},{"sha":"d0a5c32bd05841f411a342a80c5da9f73f30d69a","author":{"email":"69d9c961933324ebb2a99682f1939032e0ff256e@us.ibm.com","name":"Jason Lee"},"message":"[SPARK-12655][GRAPHX] GraphX does not unpersist RDDs\n\nSome VertexRDD and EdgeRDD are created during the intermediate step of g.connectedComponents() but unnecessarily left cached after the method is done. The fix is to unpersist these RDDs once they are no longer in use.\n\nA test case is added to confirm the fix for the reported bug.\n\nAuthor: Jason Lee <cjlee@us.ibm.com>\n\nCloses #10713 from jasoncl/SPARK-12655.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/d0a5c32bd05841f411a342a80c5da9f73f30d69a"},{"sha":"96fb894d4b33e293625fa92bbeccbbf5e688015e","author":{"email":"a1a405b82800b492a7268cba6bc58270d2b527de@yahoo-inc.com","name":"Tom Graves"},"message":"[SPARK-2930] clarify docs on using webhdfs with spark.yarn.access.nam…\n\n…enodes\n\nAuthor: Tom Graves <tgraves@yahoo-inc.com>\n\nCloses #10699 from tgravescs/SPARK-2930.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/96fb894d4b33e293625fa92bbeccbbf5e688015e"},{"sha":"ba4a641902f95c5a9b3a6bebcaa56039eca2720d","author":{"email":"4aa8789978b9b9ff21ab5ed4497ed9d6a98ad6f4@oscars-mbp.usca.ibm.com","name":"Oscar D. Lara Yejas"},"message":"[SPARK-11031][SPARKR] Method str() on a DataFrame\n\nAuthor: Oscar D. Lara Yejas <odlaraye@oscars-mbp.usca.ibm.com>\nAuthor: Oscar D. Lara Yejas <olarayej@mail.usf.edu>\nAuthor: Oscar D. Lara Yejas <oscar.lara.yejas@us.ibm.com>\nAuthor: Oscar D. Lara Yejas <odlaraye@oscars-mbp.attlocal.net>\n\nCloses #9613 from olarayej/SPARK-11031.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/ba4a641902f95c5a9b3a6bebcaa56039eca2720d"},{"sha":"c5e7076da72657ea35a0aa388f8d2e6411d39280","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[MINOR] [SQL] GeneratedExpressionCode -> ExprCode\n\nGeneratedExpressionCode is too long\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10767 from davies/renaming.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/c5e7076da72657ea35a0aa388f8d2e6411d39280"},{"sha":"5f83c6991c95616ecbc2878f8860c69b2826f56c","author":{"email":"5ec9aca3de67ebdf671d9e5cddd02b8276856562@databricks.com","name":"Hossein"},"message":"[SPARK-12833][SQL] Initial import of spark-csv\n\nCSV is the most common data format in the \"small data\" world. It is often the first format people want to try when they see Spark on a single node. Having to rely on a 3rd party component for this leads to poor user experience for new users. This PR merges the popular spark-csv data source package (https://github.com/databricks/spark-csv) with SparkSQL.\n\nThis is a first PR to bring the functionality to spark 2.0 master. We will complete items outlines in the design document (see JIRA attachment) in follow up pull requests.\n\nAuthor: Hossein <hossein@databricks.com>\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #10766 from rxin/csv.","distinct":false,"url":"https://api.github.com/repos/JoshRosen/spark/commits/5f83c6991c95616ecbc2878f8860c69b2826f56c"}]}