{"push_id":965304941,"size":21,"distinct_size":21,"ref":"refs/heads/master","head":"82d84ff2dd3efb3bda20b529f09a4022586fb722","before":"915a75398ecbccdbf9a1e07333104c857ae1ce5e","commits":[{"sha":"de0914522fc5b2658959f9e2272b4e3162b14978","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13131] [SQL] Use best and average time in benchmark\n\nBest time is stabler than average time, also added a column for nano seconds per row (which could be used to estimate contributions of each components in a query).\n\nHaving best time and average time together for more information (we can see kind of variance).\n\nrate, time per row and relative are all calculated using best time.\n\nThe result looks like this:\n```\nIntel(R) Core(TM) i7-4558U CPU  2.80GHz\nrang/filter/sum:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n-------------------------------------------------------------------------------------------\nrang/filter/sum codegen=false          14332 / 16646         36.0          27.8       1.0X\nrang/filter/sum codegen=true              845 /  940        620.0           1.6      17.0X\n```\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11018 from davies/gen_bench.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/de0914522fc5b2658959f9e2272b4e3162b14978"},{"sha":"a8e2ba776b20c8054918af646d8228bba1b87c9b","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13152][CORE] Fix task metrics deprecation warning\n\nMake an internal non-deprecated version of incBytesRead and incRecordsRead so we don't have unecessary deprecation warnings in our build.\n\nRight now incBytesRead and incRecordsRead are marked as deprecated and for internal use only. We should make private[spark] versions which are not deprecated and switch to those internally so as to not clutter up the warning messages when building.\n\ncc andrewor14 who did the initial deprecation\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #11056 from holdenk/SPARK-13152-fix-task-metrics-deprecation-warnings.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/a8e2ba776b20c8054918af646d8228bba1b87c9b"},{"sha":"a64831124c215f56f124747fa241560c70cf0a36","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13079][SQL] Extend and implement InMemoryCatalog\n\nThis is a step towards consolidating `SQLContext` and `HiveContext`.\n\nThis patch extends the existing Catalog API added in #10982 to include methods for handling table partitions. In particular, a partition is identified by `PartitionSpec`, which is just a `Map[String, String]`. The Catalog is still not used by anything yet, but its API is now more or less complete and an implementation is fully tested.\n\nAbout 200 lines are test code.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11069 from andrewor14/catalog.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/a64831124c215f56f124747fa241560c70cf0a36"},{"sha":"0f81318ae217346c20894572795e1a9cee2ebc8f","author":{"email":"c2bf8a29ab4e9ca2b842ca45622f95cb9164f566@intel.com","name":"Daoyuan Wang"},"message":"[SPARK-12828][SQL] add natural join support\n\nJira:\nhttps://issues.apache.org/jira/browse/SPARK-12828\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #10762 from adrian-wang/naturaljoin.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/0f81318ae217346c20894572795e1a9cee2ebc8f"},{"sha":"c2c956bcd1a75fd01868ee9ad2939a6d3de52bc2","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[ML][DOC] fix wrong api link in ml onevsrest\n\nminor fix for api link in ml onevsrest\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #11068 from hhbyyh/onevsrestDoc.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/c2c956bcd1a75fd01868ee9ad2939a6d3de52bc2"},{"sha":"d39087147ff1052b623cdba69ffbde28b266745f","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13113] [CORE] Remove unnecessary bit operation when decoding page number\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13113\n\nAs we shift bits right, looks like the bitwise AND operation is unnecessary.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11002 from viirya/improve-decodepagenumber.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/d39087147ff1052b623cdba69ffbde28b266745f"},{"sha":"dee801adb78d6abd0abbf76b4dfa71aa296b4f0b","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-12828][SQL] Natural join follow-up\n\nThis is a small addendum to #10762 to make the code more robust again future changes.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11070 from rxin/SPARK-12828-natural-join.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/dee801adb78d6abd0abbf76b4dfa71aa296b4f0b"},{"sha":"2eaeafe8a2aa31be9b230b8d53d3baccd32535b1","author":{"email":"cbdb0cc7f3f5b4be81a75fa7242590e3e9882e1e@allen-net.com","name":"Charles Allen"},"message":"[SPARK-12330][MESOS] Fix mesos coarse mode cleanup\n\nIn the current implementation the mesos coarse scheduler does not wait for the mesos tasks to complete before ending the driver. This causes a race where the task has to finish cleaning up before the mesos driver terminates it with a SIGINT (and SIGKILL after 3 seconds if the SIGINT doesn't work).\n\nThis PR causes the mesos coarse scheduler to wait for the mesos tasks to finish (with a timeout defined by `spark.mesos.coarse.shutdown.ms`)\n\nThis PR also fixes a regression caused by [SPARK-10987] whereby submitting a shutdown causes a race between the local shutdown procedure and the notification of the scheduler driver disconnection. If the scheduler driver disconnection wins the race, the coarse executor incorrectly exits with status 1 (instead of the proper status 0)\n\nWith this patch the mesos coarse scheduler terminates properly, the executors clean up, and the tasks are reported as `FINISHED` in the Mesos console (as opposed to `KILLED` in < 1.6 or `FAILED` in 1.6 and later)\n\nAuthor: Charles Allen <charles@allen-net.com>\n\nCloses #10319 from drcrallen/SPARK-12330.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/2eaeafe8a2aa31be9b230b8d53d3baccd32535b1"},{"sha":"62a7c28388539e6fc7d16ee3009f2cf79d8635bd","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13164][CORE] Replace deprecated synchronized buffer in core\n\nBuilding with scala 2.11 results in the warning trait SynchronizedBuffer in package mutable is deprecated: Synchronization via traits is deprecated as it is inherently unreliable. Consider java.util.concurrent.ConcurrentLinkedQueue as an alternative. Investigation shows we are already using ConcurrentLinkedQueue in other locations so switch our uses of SynchronizedBuffer to ConcurrentLinkedQueue.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #11059 from holdenk/SPARK-13164-replace-deprecated-synchronized-buffer-in-core.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/62a7c28388539e6fc7d16ee3009f2cf79d8635bd"},{"sha":"4120bcbaffe92da40486b469334119ed12199f4f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13162] Standalone mode does not respect initial executors\n\nCurrently the Master would always set an application's initial executor limit to infinity. If the user specified `spark.dynamicAllocation.initialExecutors`, the config would not take effect. This is similar to #11047 but for standalone mode.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11054 from andrewor14/standalone-da-initial.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/4120bcbaffe92da40486b469334119ed12199f4f"},{"sha":"15205da817b24ef0e349ec24d84034dc30b501f8","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13053][TEST] Unignore tests in InternalAccumulatorSuite\n\nThese were ignored because they are incorrectly written; they don't actually trigger stage retries, which is what the tests are testing. These tests are now rewritten to induce stage retries through fetch failures.\n\nNote: there were 2 tests before and now there's only 1. What happened? It turns out that the case where we only resubmit a subset of of the original missing partitions is very difficult to simulate in tests without potentially introducing flakiness. This is because the `DAGScheduler` removes all map outputs associated with a given executor when this happens, and we will need multiple executors to trigger this case, and sometimes the scheduler still removes map outputs from all executors.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #10969 from andrewor14/unignore-accum-test.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/15205da817b24ef0e349ec24d84034dc30b501f8"},{"sha":"085f510ae554e2739a38ee0bc7210c4ece902f3f","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"MAINTENANCE: Automated closing of pull requests.\n\nThis commit exists to close the following pull requests on Github:\n\nCloses #7971 (requested by yhuai)\nCloses #8539 (requested by srowen)\nCloses #8746 (requested by yhuai)\nCloses #9288 (requested by andrewor14)\nCloses #9321 (requested by andrewor14)\nCloses #9935 (requested by JoshRosen)\nCloses #10442 (requested by andrewor14)\nCloses #10585 (requested by srowen)\nCloses #10785 (requested by srowen)\nCloses #10832 (requested by andrewor14)\nCloses #10941 (requested by marmbrus)\nCloses #11024 (requested by andrewor14)","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/085f510ae554e2739a38ee0bc7210c4ece902f3f"},{"sha":"33212cb9a13a6012b4c19ccfc0fb3db75de304da","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13168][SQL] Collapse adjacent repartition operators\n\nSpark SQL should collapse adjacent `Repartition` operators and only keep the last one.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11064 from JoshRosen/collapse-repartition.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/33212cb9a13a6012b4c19ccfc0fb3db75de304da"},{"sha":"c756bda477f458ba4aad7fdb2026263507e0ad9b","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-12330][MESOS][HOTFIX] Rename timeout config\n\nThe config already describes time and accepts a general format\nthat is not restricted to ms. This commit renames the internal\nconfig to use a format that's consistent in Spark.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/c756bda477f458ba4aad7fdb2026263507e0ad9b"},{"sha":"bd38dd6f75c4af0f8f32bb21a82da53fffa5e825","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13079][SQL] InMemoryCatalog follow-ups\n\nThis patch incorporates review feedback from #11069, which is already merged.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11080 from andrewor14/catalog-follow-ups.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/bd38dd6f75c4af0f8f32bb21a82da53fffa5e825"},{"sha":"8e2f296306131e6c7c2f06d6672995d3ff8ab021","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-13195][STREAMING] Fix NoSuchElementException when a state is not set but timeoutThreshold is defined\n\nCheck the state Existence before calling get.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11081 from zsxwing/SPARK-13195.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/8e2f296306131e6c7c2f06d6672995d3ff8ab021"},{"sha":"7a4b37f02cffd6d971c07716688a7cb6cee26c8b","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[HOTFIX] Fix style violation caused by c756bda","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/7a4b37f02cffd6d971c07716688a7cb6cee26c8b"},{"sha":"6dbfc40776514c3a5667161ebe7829f4cc9c7529","author":{"email":"3e1f34debcd061268cc0e0a3b81ce7a1e5e5f57e@gmail.com","name":"Raafat Akkad"},"message":"[SPARK-13052] waitingApps metric doesn't show the number of apps currently in the WAITING state\n\nAuthor: Raafat Akkad <raafat.akkad@gmail.com>\n\nCloses #10959 from RaafatAkkad/master.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/6dbfc40776514c3a5667161ebe7829f4cc9c7529"},{"sha":"e3c75c6398b1241500343ff237e9bcf78b5396f9","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12850][SQL] Support Bucket Pruning (Predicate Pushdown for Bucketed Tables)\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12850\n\nThis PR is to support bucket pruning when the predicates are `EqualTo`, `EqualNullSafe`, `IsNull`, `In`, and `InSet`.\n\nLike HIVE, in this PR, the bucket pruning works when the bucketing key has one and only one column.\n\nSo far, I do not find a way to verify how many buckets are actually scanned. However, I did verify it when doing the debug. Could you provide a suggestion how to do it properly? Thank you! cloud-fan yhuai rxin marmbrus\n\nBTW, we can add more cases to support complex predicate including `Or` and `And`. Please let me know if I should do it in this PR.\n\nMaybe we also need to add test cases to verify if bucket pruning works well for each data type.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #10942 from gatorsmile/pruningBuckets.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/e3c75c6398b1241500343ff237e9bcf78b5396f9"},{"sha":"352102ed0b7be8c335553d7e0389fd7ce83f5fbf","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-13208][CORE] Replace use of Pairs with Tuple2s\n\nAnother trivial deprecation fix for Scala 2.11\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #11089 from jodersky/SPARK-13208.","distinct":true,"url":"https://api.github.com/repos/rlugojr/spark/commits/352102ed0b7be8c335553d7e0389fd7ce83f5fbf"}]}