{"push_id":1182159751,"size":35,"distinct_size":35,"ref":"refs/heads/branch-2.0","head":"b52bd8070dc852b419283f8a14595e42c179d3d0","before":"c7704099da6c2b949a17abb3a75b7b4e01ff869a","commits":[{"sha":"4c4f7775cbf5dd69e688350ee59a9319bcaa56fe","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARK-16143][R] group AFT survival regression methods docs in a single Rd\n\n## What changes were proposed in this pull request?\n\nThis PR groups `spark.survreg`, `summary(AFT)`, `predict(AFT)`, `write.ml(AFT)` for survival regression into a single Rd.\n\n## How was this patch tested?\n\nManually checked generated HTML doc. See attached screenshots.\n\n![screen shot 2016-06-27 at 10 28 20 am](https://cloud.githubusercontent.com/assets/15318264/16392008/a14cf472-3c5e-11e6-9ce5-490ed1a52249.png)\n![screen shot 2016-06-27 at 10 28 35 am](https://cloud.githubusercontent.com/assets/15318264/16392009/a14e333c-3c5e-11e6-8bd7-c2e9ba71f8e2.png)\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #13927 from junyangq/SPARK-16143.\n\n(cherry picked from commit 1b7fc5817203db5a56489b289fb1a0dd44b2e26b)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/4c4f7775cbf5dd69e688350ee59a9319bcaa56fe"},{"sha":"7177e184368393b1f5ecd246635168ffc10bf91e","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16111][SQL][DOC] Hide SparkOrcNewRecordReader in API docs\n\n## What changes were proposed in this pull request?\n\nCurrently, Spark Scala/Java API documents shows **org.apache.hadoop.hive.ql.io.orc** package at the top.\n\nhttp://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.package\nhttp://spark.apache.org/docs/2.0.0-preview/api/java/index.html\n\nThis PR hides `SparkOrcNewRecordReader` from API docs.\n\n## How was this patch tested?\n\nManual. (`build/sbt unidoc`).\n\nThe following is the screenshot after this PR.\n\n**Scala API doc**\n\n![Scala API doc](https://app.box.com/representation/file_version_75673952621/image_2048/1.png?shared_name=2mdqydygs8le6q9x00356898662zjwz6)\n\n**Java API doc**\n\n![Java API doc](https://app.box.com/representation/file_version_75673951725/image_2048/1.png?shared_name=iv23eeqy3avvkqz203v9ygfaqeyml85j)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13914 from dongjoon-hyun/SPARK-16111.\n\n(cherry picked from commit 50fdd866b55cb9b51427095e56b2aafea12a7c23)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/7177e184368393b1f5ecd246635168ffc10bf91e"},{"sha":"af70ad02859900e8c890e38b6fec0d12d42461f2","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-15863][SQL][DOC][FOLLOW-UP] Update SQL programming guide.\n\n## What changes were proposed in this pull request?\nThis PR makes several updates to SQL programming guide.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13938 from yhuai/doc.\n\n(cherry picked from commit dd6b7dbe7043f3fa3d2e3993d2e13f87231a59ca)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/af70ad02859900e8c890e38b6fec0d12d42461f2"},{"sha":"b349237e4b4cf60fccf9bfcf76deca78f1224bf1","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16242][MLLIB][PYSPARK] Conversion between old/new matrix columns in a DataFrame (Python)\n\n## What changes were proposed in this pull request?\nThis PR implements python wrappers for #13888 to convert old/new matrix columns in a DataFrame.\n\n## How was this patch tested?\nDoctest in python.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13935 from yanboliang/spark-16242.\n\n(cherry picked from commit e158478a9fff5e63ae0336a54b3f360d0cd38921)\nSigned-off-by: Yanbo Liang <ybliang8@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/b349237e4b4cf60fccf9bfcf76deca78f1224bf1"},{"sha":"4c5e16f58043b3103bbd59c5fa8fec4c411e5e11","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf\n\n## What changes were proposed in this pull request?\nWhen we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called).\n\n## How was this patch tested?\nNew tests and manual tests.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13931 from yhuai/SPARK-16224.\n\n(cherry picked from commit 0923c4f5676691e28e70ecb05890e123540b91f0)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/4c5e16f58043b3103bbd59c5fa8fec4c411e5e11"},{"sha":"e68872f2ef89e85ab7a856bee82d16c938de6db0","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16181][SQL] outer join with isNull filter may return wrong result\n\n## What changes were proposed in this pull request?\n\nThe root cause is: the output attributes of outer join are derived from its children, while they are actually different attributes(outer join can return null).\n\nWe have already added some special logic to handle it, e.g. `PushPredicateThroughJoin` won't push down predicates through outer join side, `FixNullability`.\n\nThis PR adds one more special logic in `FoldablePropagation`.\n\n## How was this patch tested?\n\nnew test in `DataFrameSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13884 from cloud-fan/bug.\n\n(cherry picked from commit 1f2776df6e87a84991537ac20e4b8829472d3462)\nSigned-off-by: Yin Huai <yhuai@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/e68872f2ef89e85ab7a856bee82d16c938de6db0"},{"sha":"a1d04cc03edac729488e9eb2b9e34e6c951ddbae","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-15643][DOC][ML] Update spark.ml and spark.mllib migration guide from 1.6 to 2.0\n\n## What changes were proposed in this pull request?\nUpdate ```spark.ml``` and ```spark.mllib``` migration guide from 1.6 to 2.0.\n\n## How was this patch tested?\nDocs update, no tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13378 from yanboliang/spark-13448.\n\n(cherry picked from commit 26252f7064ba852e1bce6d8233a95aeb395f826a)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/a1d04cc03edac729488e9eb2b9e34e6c951ddbae"},{"sha":"c86d29b2e6bfda05124c20ba3c6db9275c24faa8","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[MINOR][SPARKR] update sparkR DataFrame.R comment\n\n## What changes were proposed in this pull request?\n\nupdate sparkR DataFrame.R comment\nSQLContext ==> SparkSession\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #13946 from WeichenXu123/sparkR_comment_update_sparkSession.\n\n(cherry picked from commit d59ba8e30751bbf91d49f5530b8242a12bbfb569)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/c86d29b2e6bfda05124c20ba3c6db9275c24faa8"},{"sha":"5c9555e1115ce52954db2a1b18f78cd77ec8c15f","author":{"name":"Tom Magrino","email":"b4ff0ed6b2f378ecb18a44daba12916fd6d7add8@fb.com"},"message":"[SPARK-16148][SCHEDULER] Allow for underscores in TaskLocation in the Executor ID\n\n## What changes were proposed in this pull request?\n\nPreviously, the TaskLocation implementation would not allow for executor ids which include underscores.  This tweaks the string split used to get the hostname and executor id, allowing for underscores in the executor id.\n\nThis addresses the JIRA found here: https://issues.apache.org/jira/browse/SPARK-16148\n\nThis is moved over from a previous PR against branch-1.6: https://github.com/apache/spark/pull/13857\n\n## How was this patch tested?\n\nRan existing unit tests for core and streaming.  Manually ran a simple streaming job with an executor whose id contained underscores and confirmed that the job ran successfully.\n\nThis is my original work and I license the work to the project under the project's open source license.\n\nAuthor: Tom Magrino <tmagrino@fb.com>\n\nCloses #13858 from tmagrino/fixtasklocation.\n\n(cherry picked from commit ae14f362355b131fcb3e3633da7bb14bdd2b6893)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/5c9555e1115ce52954db2a1b18f78cd77ec8c15f"},{"sha":"43bd612f35490c11a76d5379d723ba65f7afbefd","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16175] [PYSPARK] handle None for UDT\n\n## What changes were proposed in this pull request?\n\nScala UDT will bypass all the null and will not pass them into serialize() and deserialize() of UDT, this PR update the Python UDT to do this as well.\n\n## How was this patch tested?\n\nAdded tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13878 from davies/udt_null.\n\n(cherry picked from commit 35438fb0ad3bcda5c5a3a0ccde1a620699d012db)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/43bd612f35490c11a76d5379d723ba65f7afbefd"},{"sha":"5626a0af598168a15d68a8817d1dec2a0e3dec7e","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16236][SQL] Add Path Option back to Load API in DataFrameReader\n\n#### What changes were proposed in this pull request?\nkoertkuipers identified the PR https://github.com/apache/spark/pull/13727/ changed the behavior of `load` API. After the change, the `load` API does not add the value of `path` into the `options`. Thank you!\n\nThis PR is to add the option `path` back to `load()` API in `DataFrameReader`, if and only if users specify one and only one `path` in the `load` API. For example, users can see the `path` option after the following API call,\n```Scala\nspark.read\n  .format(\"parquet\")\n  .load(\"/test\")\n```\n\n#### How was this patch tested?\nAdded test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13933 from gatorsmile/optionPath.\n\n(cherry picked from commit 25520e976275e0d1e3bf9c73128ef4dec4618568)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/5626a0af598168a15d68a8817d1dec2a0e3dec7e"},{"sha":"d73c38ed0e129bdcb634000153516fca4b31b9d0","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16100][SQL] fix bug when use Map as the buffer type of Aggregator\n\n## What changes were proposed in this pull request?\n\nThe root cause is in `MapObjects`. Its parameter `loopVar` is not declared as child, but sometimes can be same with `lambdaFunction`(e.g. the function that takes `loopVar` and produces `lambdaFunction` may be `identity`), which is a child. This brings trouble when call `withNewChildren`, it may mistakenly treat `loopVar` as a child and cause `IndexOutOfBoundsException: 0` later.\n\nThis PR fixes this bug by simply pulling out the paremters from `LambdaVariable` and pass them to `MapObjects` directly.\n\n## How was this patch tested?\n\nnew test in `DatasetAggregatorSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13835 from cloud-fan/map-objects.\n\n(cherry picked from commit 8a977b065418f07d2bf4fe1607a5534c32d04c47)\nSigned-off-by: Cheng Lian <lian@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/d73c38ed0e129bdcb634000153516fca4b31b9d0"},{"sha":"5fb7804e55e50ba61c3a780b771d9b20b0bf2424","author":{"name":"James Thomas","email":"4c3f70f7e07111950bf31cd977787d757ef55837@gmail.com"},"message":"[SPARK-16114][SQL] structured streaming network word count examples\n\n## What changes were proposed in this pull request?\n\nNetwork word count example for structured streaming\n\n## How was this patch tested?\n\nRun locally\n\nAuthor: James Thomas <jamesjoethomas@gmail.com>\nAuthor: James Thomas <jamesthomas@Jamess-MacBook-Pro.local>\n\nCloses #13816 from jjthomas/master.\n\n(cherry picked from commit 3554713a163c58ca176ffde87d2c6e4a91bacb50)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/5fb7804e55e50ba61c3a780b771d9b20b0bf2424"},{"sha":"52c9d69f7da05c45cb191fef8f7ce54c8f40b1bb","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[MINOR][DOCS][STRUCTURED STREAMING] Minor doc fixes around `DataFrameWriter` and `DataStreamWriter`\n\n## What changes were proposed in this pull request?\n\nFixes a couple old references to `DataFrameWriter.startStream` to `DataStreamWriter.start\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #13952 from brkyvz/minor-doc-fix.\n\n(cherry picked from commit 5545b791096756b07b3207fb3de13b68b9a37b00)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/52c9d69f7da05c45cb191fef8f7ce54c8f40b1bb"},{"sha":"d7a59f1f450aae06baac96867a26042bd1ccd1d5","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR] add csv tests\n\n## What changes were proposed in this pull request?\n\nAdd unit tests for csv data for SPARKR\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13904 from felixcheung/rcsv.\n\n(cherry picked from commit 823518c2b5259c8a954431467639198c808c9198)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/d7a59f1f450aae06baac96867a26042bd1ccd1d5"},{"sha":"835c5a3bd549811178f5b455dc127401c5551866","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16268][PYSPARK] SQLContext should import DataStreamReader\n\n## What changes were proposed in this pull request?\n\nFixed the following error:\n```\n>>> sqlContext.readStream\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\n  File \"...\", line 442, in readStream\n    return DataStreamReader(self._wrapped)\nNameError: global name 'DataStreamReader' is not defined\n```\n\n## How was this patch tested?\n\nThe added test.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13958 from zsxwing/fix-import.\n\n(cherry picked from commit 5bf8881b34a18f25acc10aeb28a06af4c44a6ac8)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/835c5a3bd549811178f5b455dc127401c5551866"},{"sha":"dd70a115cd562223e97f0b5e6172a9ea758be95d","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16248][SQL] Whitelist the list of Hive fallback functions\n\n## What changes were proposed in this pull request?\nThis patch removes the blind fallback into Hive for functions. Instead, it creates a whitelist and adds only a small number of functions to the whitelist, i.e. the ones we intend to support in the long run in Spark.\n\n## How was this patch tested?\nUpdated tests to reflect the change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13939 from rxin/hive-whitelist.\n\n(cherry picked from commit 363bcedeea40fe3f1a92271b96af2acba63e058c)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/dd70a115cd562223e97f0b5e6172a9ea758be95d"},{"sha":"22b4072e704f9a68a605e9a4cebf54d2122fe448","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16245][ML] model loading backward compatibility for ml.feature.PCA\n\n## What changes were proposed in this pull request?\nmodel loading backward compatibility for ml.feature.PCA.\n\n## How was this patch tested?\nexisting ut and manual test for loading models saved by Spark 1.6.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13937 from yanboliang/spark-16245.\n\n(cherry picked from commit 0df5ce1bc1387a58b33cd185008f4022bd3dcc69)\nSigned-off-by: Xiangrui Meng <meng@databricks.com>","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/22b4072e704f9a68a605e9a4cebf54d2122fe448"},{"sha":"345212b9fc91638f6cda8519ddbfec6a780854c1","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16259][PYSPARK] cleanup options in DataFrame read/write API\n\n## What changes were proposed in this pull request?\n\nThere are some duplicated code for options in DataFrame reader/writer API, this PR clean them up, it also fix a bug for `escapeQuotes` of csv().\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13948 from davies/csv_options.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/345212b9fc91638f6cda8519ddbfec6a780854c1"},{"sha":"6650c0533e5c60f8653d2e0a608a42d5838fa553","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-16266][SQL][STREAING] Moved DataStreamReader/Writer from pyspark.sql to pyspark.sql.streaming\n\n## What changes were proposed in this pull request?\n\n- Moved DataStreamReader/Writer from pyspark.sql to pyspark.sql.streaming to make them consistent with scala packaging\n- Exposed the necessary classes in sql.streaming package so that they appear in the docs\n- Added pyspark.sql.streaming module to the docs\n\n## How was this patch tested?\n- updated unit tests.\n- generated docs for testing visibility of pyspark.sql.streaming classes.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #13955 from tdas/SPARK-16266.","distinct":true,"url":"https://api.github.com/repos/chuckchen/spark/commits/6650c0533e5c60f8653d2e0a608a42d5838fa553"}]}
{"actor":{"display_login":"chuckchen"}}