{"push_id":1002124647,"size":11,"distinct_size":11,"ref":"refs/heads/master","head":"366f26d2da0437aab99fd88b70ca12ae18958451","before":"3c5f5e3b5c4eb69472fdd8124aa9988bd8d933b5","commits":[{"sha":"12a2a57e1af21da0aa4275971365d76a8fc84a43","author":{"email":"e2aa89bc956dec4bf9c6f5e2da3f58db06fb878d@oss.nttdata.co.jp","name":"Masayoshi TSUZUKI"},"message":"[SPARK-13592][WINDOWS] fix path of spark-submit2.cmd in spark-submit.cmd\n\n## What changes were proposed in this pull request?\n\nThis patch fixes the problem that pyspark fails on Windows because pyspark can't find ```spark-submit2.cmd```.\n\n## How was this patch tested?\n\nmanual tests:\n  I ran ```bin\\pyspark.cmd``` and checked if pyspark is launched correctly after this patch is applyed.\n\nAuthor: Masayoshi TSUZUKI <tsudukim@oss.nttdata.co.jp>\n\nCloses #11442 from tsudukim/feature/SPARK-13592.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/12a2a57e1af21da0aa4275971365d76a8fc84a43"},{"sha":"c43899a04e4de18e238a1761bf4fe9f54e182320","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13511] [SQL] Add wholestage codegen for limit\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13511\n\n## What changes were proposed in this pull request?\n\nCurrent limit operator doesn't support wholestage codegen. This is open to add support for it.\n\nIn the `doConsume` of `GlobalLimit` and `LocalLimit`, we use a count term to count the processed rows. Once the row numbers catches the limit number, we set the variable `stopEarly` of `BufferedRowIterator` newly added in this pr to `true` that indicates we want to stop processing remaining rows. Then when the wholestage codegen framework checks `shouldStop()`, it will stop the processing of the row iterator.\n\nBefore this, the executed plan for a query `sqlContext.range(N).limit(100).groupBy().sum()` is:\n\n    TungstenAggregate(key=[], functions=[(sum(id#5L),mode=Final,isDistinct=false)], output=[sum(id)#6L])\n    +- TungstenAggregate(key=[], functions=[(sum(id#5L),mode=Partial,isDistinct=false)], output=[sum#9L])\n       +- GlobalLimit 100\n          +- Exchange SinglePartition, None\n             +- LocalLimit 100\n                +- Range 0, 1, 1, 524288000, [id#5L]\n\nAfter add wholestage codegen support:\n\n    WholeStageCodegen\n    :  +- TungstenAggregate(key=[], functions=[(sum(id#40L),mode=Final,isDistinct=false)], output=[sum(id)#41L])\n    :     +- TungstenAggregate(key=[], functions=[(sum(id#40L),mode=Partial,isDistinct=false)], output=[sum#44L])\n    :        +- GlobalLimit 100\n    :           +- INPUT\n    +- Exchange SinglePartition, None\n       +- WholeStageCodegen\n          :  +- LocalLimit 100\n          :     +- Range 0, 1, 1, 524288000, [id#40L]\n\n## How was this patch tested?\n\nA test is added into BenchmarkWholeStageCodegen.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11391 from viirya/wholestage-limit.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/c43899a04e4de18e238a1761bf4fe9f54e182320"},{"sha":"5ed48dd84d38dfe621428e164a02e74ddbdbc622","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12811][ML] Estimator for Generalized Linear Models(GLMs)\n\nEstimator for Generalized Linear Models(GLMs) which will be solved by IRLS.\n\ncc mengxr\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #11136 from yanboliang/spark-12811.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/5ed48dd84d38dfe621428e164a02e74ddbdbc622"},{"sha":"c37bbb3a1cbd93c749aaaeca1345817e0c20094f","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"Closes #11320\nCloses #10940\nCloses #11302\nCloses #11430\nCloses #10912","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/c37bbb3a1cbd93c749aaaeca1345817e0c20094f"},{"sha":"c27ba0d547a0cd3fd00bb42c76ad971b2d48b4a0","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13582] [SQL] defer dictionary decoding in parquet reader\n\n## What changes were proposed in this pull request?\n\nThis PR defer the resolution from a id of dictionary to value until the column is actually accessed (inside getInt/getLong), this is very useful for those columns and rows that are filtered out. It's also useful for binary type, we will not need to copy all the byte arrays.\n\nThis PR also change the underlying type for small decimal that could be fit within a Int, in order to use getInt() to lookup the value from IntDictionary.\n\n## How was this patch tested?\n\nManually test TPCDS Q7 with scale factor 10, saw about 30% improvements (after PR #11274).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11437 from davies/decode_dict.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/c27ba0d547a0cd3fd00bb42c76ad971b2d48b4a0"},{"sha":"b0ee7d43730469ad61fdf6b7b75cc1b1efb62c31","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-13548][BUILD] Move tags and unsafe modules into common\n\n## What changes were proposed in this pull request?\nThis patch moves tags and unsafe modules into common directory to remove 2 top level non-user-facing directories.\n\n## How was this patch tested?\nJenkins should suffice.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11426 from rxin/SPARK-13548.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/b0ee7d43730469ad61fdf6b7b75cc1b1efb62c31"},{"sha":"a640c5b4fbd653919a5897a7b11f16328f2094eb","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13598] [SQL] remove LeftSemiJoinBNL\n\n## What changes were proposed in this pull request?\n\nBroadcast left semi join without joining keys is already supported in BroadcastNestedLoopJoin, it has the same implementation as LeftSemiJoinBNL, we should remove that.\n\n## How was this patch tested?\n\nUpdated unit tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11448 from davies/remove_bnl.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/a640c5b4fbd653919a5897a7b11f16328f2094eb"},{"sha":"e42724b12b976b3276accc1132f446fa67f7f981","author":{"email":"d5493c25651f198266b02bc9287959eaadab29fc@gmail.com","name":"sureshthalamati"},"message":"[SPARK-13167][SQL] Include rows with null values for partition column when reading from JDBC datasources.\n\nRows with null values in partition column are not included in the results because none of the partition\nwhere clause specify is null predicate on the partition column. This fix adds is null predicate on the partition column  to the first JDBC partition where clause.\n\nExample:\nJDBCPartition(THEID < 1 or THEID is null, 0),JDBCPartition(THEID >= 1 AND THEID < 2,1),\nJDBCPartition(THEID >= 2, 2)\n\nAuthor: sureshthalamati <suresh.thalamati@gmail.com>\n\nCloses #11063 from sureshthalamati/nullable_jdbc_part_col_spark-13167.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/e42724b12b976b3276accc1132f446fa67f7f981"},{"sha":"9495c40f227b785d852abdc307461d2e7e5c2011","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-13008][ML][PYTHON] Put one alg per line in pyspark.ml all lists\n\nThis is to fix a long-time annoyance: Whenever we add a new algorithm to pyspark.ml, we have to add it to the ```__all__``` list at the top.  Since we keep it alphabetized, it often creates a lot more changes than needed.  It is also easy to add the Estimator and forget the Model.  I'm going to switch it to have one algorithm per line.\n\nThis also alphabetizes a few out-of-place classes in pyspark.ml.feature.  No changes have been made to the moved classes.\n\nCC: thunterdb\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #10927 from jkbradley/ml-python-all-list.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/9495c40f227b785d852abdc307461d2e7e5c2011"},{"sha":"b4d096ded6540c46a2b07b0b0897cbb0b43ba1e0","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[BUILD][MINOR] Fix SBT build error with network-yarn module\n\n## What changes were proposed in this pull request?\n\n```\nerror] Expected ID character\n[error] Not a valid command: common (similar: completions)\n[error] Expected project ID\n[error] Expected configuration\n[error] Expected ':' (if selecting a configuration)\n[error] Expected key\n[error] Not a valid key: common (similar: commands)\n[error] common/network-yarn/test\n```\n\n`common/network-yarn` is not a valid sbt project, we should change to `network-yarn`.\n\n## How was this patch tested?\n\nLocally run the the unit-test.\n\nCC rxin , we should either change here, or change the sbt project name.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #11456 from jerryshao/build-fix.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/b4d096ded6540c46a2b07b0b0897cbb0b43ba1e0"},{"sha":"366f26d2da0437aab99fd88b70ca12ae18958451","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[MINOR][STREAMING] Replace deprecated `apply` with `create` in example.\n\n## What changes were proposed in this pull request?\n\nTwitter Algebird deprecated `apply` in HyperLogLog.scala.\n```\ndeprecated(\"Use toHLL\", since = \"0.10.0 / 2015-05\")\ndef apply[T <% Array[Byte]](t: T) = create(t)\n```\nThis PR replace the deprecated usage `apply` with new `create`\naccording to the upstream change.\n\n## How was this patch tested?\nmanual.\n```\n/bin/spark-submit --class org.apache.spark.examples.streaming.TwitterAlgebirdHLL examples/target/scala-2.11/spark-examples-2.0.0-SNAPSHOT-hadoop2.2.0.jar\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11451 from dongjoon-hyun/replace_deprecated_hll_apply.","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/366f26d2da0437aab99fd88b70ca12ae18958451"}]}