{"push_id":984102187,"size":23,"distinct_size":23,"ref":"refs/heads/followup","head":"95e1ab223e87fc216f3256d404fe3be50d111a9d","before":"a8bbc4f50ef3faacf4b7fe865a29144ea87f0796","commits":[{"sha":"56d49397e01306637edf23bbb4f3b9d396cdc6ff","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-12995][GRAPHX] Remove deprecate APIs from Pregel\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10918 from maropu/RemoveDeprecateInPregel.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/56d49397e01306637edf23bbb4f3b9d396cdc6ff"},{"sha":"adb548365012552e991d51740bfd3c25abf0adec","author":{"email":"f97d5112937f779dff37a1cf186f37e361a6b63d@gmail.com","name":"JeremyNixon"},"message":"[SPARK-13312][MLLIB] Update java train-validation-split example in ml-guide\n\nResponse to JIRA https://issues.apache.org/jira/browse/SPARK-13312.\n\nThis contribution is my original work and I license the work to this project.\n\nAuthor: JeremyNixon <jnixon2@gmail.com>\n\nCloses #11199 from JeremyNixon/update_train_val_split_example.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/adb548365012552e991d51740bfd3c25abf0adec"},{"sha":"cbeb006f23838b2f19e700e20b25003aeb3dfb01","author":{"email":"08143e5b7f3ef9b2c05b03e8981fb36a6affd6c2@gmail.com","name":"seddonm1"},"message":"[SPARK-13097][ML] Binarizer allowing Double AND Vector input types\n\nThis enhancement extends the existing SparkML Binarizer [SPARK-5891] to allow Vector in addition to the existing Double input column type.\n\nA use case for this enhancement is for when a user wants to Binarize many similar feature columns at once using the same threshold value (for example a binary threshold applied to many pixels in an image).\n\nThis contribution is my original work and I license the work to the project under the project's open source license.\n\nviirya mengxr\n\nAuthor: seddonm1 <seddonm1@gmail.com>\n\nCloses #10976 from seddonm1/master.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/cbeb006f23838b2f19e700e20b25003aeb3dfb01"},{"sha":"e4675c240255207c5dd812fa657e6aca2dc9cfeb","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-13018][DOCS] Replace example code in mllib-pmml-model-export.md using include_example\n\nReplace example code in mllib-pmml-model-export.md using include_example\nhttps://issues.apache.org/jira/browse/SPARK-13018\n\nThe example code in the user guide is embedded in the markdown and hence it is not easy to test. It would be nice to automatically test them. This JIRA is to discuss options to automate example code testing and see what we can do in Spark 1.6.\n\nGoal is to move actual example code to spark/examples and test compilation in Jenkins builds. Then in the markdown, we can reference part of the code to show in the user guide. This requires adding a Jekyll tag that is similar to https://github.com/jekyll/jekyll/blob/master/lib/jekyll/tags/include.rb, e.g., called include_example.\n`{% include_example scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala %}`\nJekyll will find `examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala` and pick code blocks marked \"example\" and replace code block in\n`{% highlight %}`\n in the markdown.\n\nSee more sub-tasks in parent ticket: https://issues.apache.org/jira/browse/SPARK-11337\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #11126 from keypointt/SPARK-13018.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/e4675c240255207c5dd812fa657e6aca2dc9cfeb"},{"sha":"fee739f07b3bc37dd65682e93e60e0add848f583","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-13221] [SQL] Fixing GroupingSets when Aggregate Functions Containing GroupBy Columns\n\nUsing GroupingSets will generate a wrong result when Aggregate Functions containing GroupBy columns.\n\nThis PR is to fix it. Since the code changes are very small. Maybe we also can merge it to 1.6\n\nFor example, the following query returns a wrong result:\n```scala\nsql(\"select course, sum(earnings) as sum from courseSales group by course, earnings\" +\n     \" grouping sets((), (course), (course, earnings))\" +\n     \" order by course, sum\").show()\n```\nBefore the fix, the results are like\n```\n[null,null]\n[Java,null]\n[Java,20000.0]\n[Java,30000.0]\n[dotNET,null]\n[dotNET,5000.0]\n[dotNET,10000.0]\n[dotNET,48000.0]\n```\nAfter the fix, the results become correct:\n```\n[null,113000.0]\n[Java,20000.0]\n[Java,30000.0]\n[Java,50000.0]\n[dotNET,5000.0]\n[dotNET,10000.0]\n[dotNET,48000.0]\n[dotNET,63000.0]\n```\n\nUPDATE:  This PR also deprecated the external column: GROUPING__ID.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11100 from gatorsmile/groupingSets.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/fee739f07b3bc37dd65682e93e60e0add848f583"},{"sha":"827ed1c06785692d14857bd41f1fd94a0853874a","author":{"email":"54234b27fef504da340909df6dcfdd8e7524fa74@databricks.com","name":"Miles Yucht"},"message":"Correct SparseVector.parse documentation\n\nThere's a small typo in the SparseVector.parse docstring (which says that it returns a DenseVector rather than a SparseVector), which seems to be incorrect.\n\nAuthor: Miles Yucht <miles@databricks.com>\n\nCloses #11213 from mgyucht/fix-sparsevector-docs.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/827ed1c06785692d14857bd41f1fd94a0853874a"},{"sha":"00c72d27bf2e3591c4068fb344fa3edf1662ad81","author":{"email":"07c870ffe79a3d57c30fc67475f42d361ec88a03@gmail.com","name":"BenFradet"},"message":"[SPARK-12247][ML][DOC] Documentation for spark.ml's ALS and collaborative filtering in general\n\nThis documents the implementation of ALS in `spark.ml` with example code in scala, java and python.\n\nAuthor: BenFradet <benjamin.fradet@gmail.com>\n\nCloses #10411 from BenFradet/SPARK-12247.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/00c72d27bf2e3591c4068fb344fa3edf1662ad81"},{"sha":"19dc69de795eb08f3bab4988ad88732bf8ca7bae","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-12976][SQL] Add LazilyGenerateOrdering and use it for RangePartitioner of Exchange.\n\nAdd `LazilyGenerateOrdering` to support generated ordering for `RangePartitioner` of `Exchange` instead of `InterpretedOrdering`.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #10894 from ueshin/issues/SPARK-12976.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/19dc69de795eb08f3bab4988ad88732bf8ca7bae"},{"sha":"c7d00a24da317c9601a9239ac1cf185fb6647352","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-13280][STREAMING] Use a better logger name for FileBasedWriteAheadLog.\n\nThe new logger name is under the org.apache.spark namespace.\nThe detection of the caller name was also enhanced a bit to ignore\nsome common things that show up in the call stack.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #11165 from vanzin/SPARK-13280.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/c7d00a24da317c9601a9239ac1cf185fb6647352"},{"sha":"5f37aad48cb729a80c4cc25347460f12aafec9fb","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-13308] ManagedBuffers passed to OneToOneStreamManager need to be freed in non-error cases\n\nManagedBuffers that are passed to `OneToOneStreamManager.registerStream` need to be freed by the manager once it's done using them. However, the current code only frees them in certain error-cases and not during typical operation. This isn't a major problem today, but it will cause memory leaks after we implement better locking / pinning in the BlockManager (see #10705).\n\nThis patch modifies the relevant network code so that the ManagedBuffers are freed as soon as the messages containing them are processed by the lower-level Netty message sending code.\n\n/cc zsxwing for review.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #11193 from JoshRosen/add-missing-release-calls-in-network-layer.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/5f37aad48cb729a80c4cc25347460f12aafec9fb"},{"sha":"7218c0eba957e0a079a407b79c3a050cce9647b2","author":{"email":"204e86d2224e126835d0a0ef58ab8b5e624f862d@mogujie.com","name":"junhao"},"message":"[SPARK-11627] Add initial input rate limit for spark streaming backpressure mechanism.\n\nhttps://issues.apache.org/jira/browse/SPARK-11627\n\nSpark Streaming backpressure mechanism has no initial input rate limit, it might cause OOM exception.\nIn the firest batch task ,receivers receive data at the maximum speed they can reach,it might exhaust executors memory resources. Add a initial input rate limit value can make sure the Streaming job execute  success in the first batch,then the backpressure mechanism can adjust receiving rate adaptively.\n\nAuthor: junhao <junhao@mogujie.com>\n\nCloses #9593 from junhaoMg/junhao-dev.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/7218c0eba957e0a079a407b79c3a050cce9647b2"},{"sha":"1e1e31e03df14f2e7a9654e640fb2796cf059fe0","author":{"email":"651d9e2ab320d6d21a117141758c2477389f223d@fb.com","name":"Sital Kedia"},"message":"[SPARK-13279] Remove O(n^2) operation from scheduler.\n\nThis commit removes an unnecessary duplicate check in addPendingTask that meant\nthat scheduling a task set took time proportional to (# tasks)^2.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #11175 from sitalkedia/fix_stuck_driver.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/1e1e31e03df14f2e7a9654e640fb2796cf059fe0"},{"sha":"04e8afe362f3bf36097688f9574c66002535ac0f","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-13357][SQL] Use generated projection and ordering for TakeOrderedAndProjectNode\n\n`TakeOrderedAndProjectNode` should use generated projection and ordering like other `LocalNode`s.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #11230 from ueshin/issues/SPARK-13357.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/04e8afe362f3bf36097688f9574c66002535ac0f"},{"sha":"a7c74d7563926573c01baf613708a0f105a03e57","author":{"email":"711c73f64afdce07b7e38039a96d2224209e9a6c@chrisaycock.com","name":"Christopher C. Aycock"},"message":"[SPARK-13350][DOCS] Config doc updated to state that PYSPARK_PYTHON's default is \"python2.7\"\n\nAuthor: Christopher C. Aycock <chris@chrisaycock.com>\n\nCloses #11239 from chrisaycock/master.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/a7c74d7563926573c01baf613708a0f105a03e57"},{"sha":"1eac3800087a804c4b58a4af8b65eeb0879b6528","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-13109][BUILD] Fix SBT publishLocal issue\n\nAdd local ivy repo to the SBT build file to fix this.\n\nScaladoc compile error is fixed.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #11001 from jerryshao/SPARK-13109.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/1eac3800087a804c4b58a4af8b65eeb0879b6528"},{"sha":"97ee85daf68345cf5c3c11ae5bf288cc697bdf9e","author":{"email":"412185dfb2d05d147507face6410e08d48a8c8c1@163.com","name":"shijinkui"},"message":"[SPARK-12953][EXAMPLES] RDDRelation writer set overwrite mode\n\nhttps://issues.apache.org/jira/browse/SPARK-12953\n\nfix error when run RDDRelation.main():\n\"path file:/Users/sjk/pair.parquet already exists\"\n\nSet DataFrameWriter's mode to SaveMode.Overwrite\n\nAuthor: shijinkui <shijinkui666@163.com>\n\nCloses #10864 from shijinkui/set_mode.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/97ee85daf68345cf5c3c11ae5bf288cc697bdf9e"},{"sha":"9451fed52cb8a00c706b582a0b51d8cd832f9350","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13344][TEST] Fix harmless accumulator not found exceptions\n\nSee [JIRA](https://issues.apache.org/jira/browse/SPARK-13344) for more detail. This was caused by #10835.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11222 from andrewor14/fix-test-accum-exceptions.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/9451fed52cb8a00c706b582a0b51d8cd832f9350"},{"sha":"0088b252bf56d391025a19b70af03f3ce6d7574c","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[MINOR][MLLIB] fix mllib compile warnings\n\nThis PR fixes some warnings found by `build/sbt mllib/test:compile`.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #11227 from mengxr/fix-mllib-warnings-201602.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/0088b252bf56d391025a19b70af03f3ce6d7574c"},{"sha":"b84404865b90fc07858a8b28c2a7028f96517167","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-13324][CORE][BUILD] Update plugin, test, example dependencies for 2.x\n\nPhase 1: update plugin versions, test dependencies, some example and third-party versions\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #11206 from srowen/SPARK-13324.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/b84404865b90fc07858a8b28c2a7028f96517167"},{"sha":"892b2dd6dd00d2088c75ab3c8443e8b3e44e5803","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"Add github pull request template","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/892b2dd6dd00d2088c75ab3c8443e8b3e44e5803"}]}