{"push_id":1191373611,"size":110,"distinct_size":110,"ref":"refs/heads/master","head":"480357cc6d71c682fe703611c71c1e6a36e6ce9a","before":"c15b552dd547a129c7f0d082dab4eebbd64bee02","commits":[{"sha":"02a029df43392c5d73697203bf6ff51b8d6efb83","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-16220][SQL] Add scope to show functions\n\n## What changes were proposed in this pull request?\nSpark currently shows all functions when issue a `SHOW FUNCTIONS` command. This PR refines the `SHOW FUNCTIONS` command by allowing users to select all functions, user defined function or system functions. The following syntax can be used:\n\n**ALL** (default)\n```SHOW FUNCTIONS```\n```SHOW ALL FUNCTIONS```\n\n**SYSTEM**\n```SHOW SYSTEM FUNCTIONS```\n\n**USER**\n```SHOW USER FUNCTIONS```\n## How was this patch tested?\nUpdated tests and added tests to the DDLSuite\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #13929 from hvanhovell/SPARK-16220.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/02a029df43392c5d73697203bf6ff51b8d6efb83"},{"sha":"1b7fc5817203db5a56489b289fb1a0dd44b2e26b","author":{"name":"Junyang Qian","email":"d5766da94a207df74ee6bb6bd7d2e5654c64d4d7@databricks.com"},"message":"[SPARK-16143][R] group AFT survival regression methods docs in a single Rd\n\n## What changes were proposed in this pull request?\n\nThis PR groups `spark.survreg`, `summary(AFT)`, `predict(AFT)`, `write.ml(AFT)` for survival regression into a single Rd.\n\n## How was this patch tested?\n\nManually checked generated HTML doc. See attached screenshots.\n\n![screen shot 2016-06-27 at 10 28 20 am](https://cloud.githubusercontent.com/assets/15318264/16392008/a14cf472-3c5e-11e6-9ce5-490ed1a52249.png)\n![screen shot 2016-06-27 at 10 28 35 am](https://cloud.githubusercontent.com/assets/15318264/16392009/a14e333c-3c5e-11e6-8bd7-c2e9ba71f8e2.png)\n\nAuthor: Junyang Qian <junyangq@databricks.com>\n\nCloses #13927 from junyangq/SPARK-16143.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/1b7fc5817203db5a56489b289fb1a0dd44b2e26b"},{"sha":"50fdd866b55cb9b51427095e56b2aafea12a7c23","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16111][SQL][DOC] Hide SparkOrcNewRecordReader in API docs\n\n## What changes were proposed in this pull request?\n\nCurrently, Spark Scala/Java API documents shows **org.apache.hadoop.hive.ql.io.orc** package at the top.\n\nhttp://spark.apache.org/docs/2.0.0-preview/api/scala/index.html#org.apache.spark.package\nhttp://spark.apache.org/docs/2.0.0-preview/api/java/index.html\n\nThis PR hides `SparkOrcNewRecordReader` from API docs.\n\n## How was this patch tested?\n\nManual. (`build/sbt unidoc`).\n\nThe following is the screenshot after this PR.\n\n**Scala API doc**\n\n![Scala API doc](https://app.box.com/representation/file_version_75673952621/image_2048/1.png?shared_name=2mdqydygs8le6q9x00356898662zjwz6)\n\n**Java API doc**\n\n![Java API doc](https://app.box.com/representation/file_version_75673951725/image_2048/1.png?shared_name=iv23eeqy3avvkqz203v9ygfaqeyml85j)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13914 from dongjoon-hyun/SPARK-16111.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/50fdd866b55cb9b51427095e56b2aafea12a7c23"},{"sha":"a0da854fb3748aca0128377f0955600cb7a2b5bc","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16221][SQL] Redirect Parquet JUL logger via SLF4J for WRITE operations\n\n## What changes were proposed in this pull request?\n\n[SPARK-8118](https://github.com/apache/spark/pull/8196) implements redirecting Parquet JUL logger via SLF4J, but it is currently applied only when READ operations occurs. If users use only WRITE operations, there occurs many Parquet logs.\n\nThis PR makes the redirection work on WRITE operations, too.\n\n**Before**\n```scala\nscala> spark.range(10).write.format(\"parquet\").mode(\"overwrite\").save(\"/tmp/p\")\nSLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".\nSLF4J: Defaulting to no-operation (NOP) logger implementation\nSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\nJun 26, 2016 9:04:38 PM INFO: org.apache.parquet.hadoop.codec.CodecConfig: Compression: SNAPPY\n............ about 70 lines Parquet Log .............\nscala> spark.range(10).write.format(\"parquet\").mode(\"overwrite\").save(\"/tmp/p\")\n............ about 70 lines Parquet Log .............\n```\n\n**After**\n```scala\nscala> spark.range(10).write.format(\"parquet\").mode(\"overwrite\").save(\"/tmp/p\")\nSLF4J: Failed to load class \"org.slf4j.impl.StaticLoggerBinder\".\nSLF4J: Defaulting to no-operation (NOP) logger implementation\nSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.\nscala> spark.range(10).write.format(\"parquet\").mode(\"overwrite\").save(\"/tmp/p\")\n```\n\nThis PR also fixes some typos.\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13918 from dongjoon-hyun/SPARK-16221.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/a0da854fb3748aca0128377f0955600cb7a2b5bc"},{"sha":"dd6b7dbe7043f3fa3d2e3993d2e13f87231a59ca","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-15863][SQL][DOC][FOLLOW-UP] Update SQL programming guide.\n\n## What changes were proposed in this pull request?\nThis PR makes several updates to SQL programming guide.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13938 from yhuai/doc.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/dd6b7dbe7043f3fa3d2e3993d2e13f87231a59ca"},{"sha":"4cbf611c1dc88111ff49d005e902ad5864799ede","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16202][SQL][DOC] Correct The Description of CreatableRelationProvider's createRelation\n\n#### What changes were proposed in this pull request?\nThe API description of `createRelation` in `CreatableRelationProvider` is misleading. The current description only expects users to return the relation.\n\n```Scala\ntrait CreatableRelationProvider {\n  def createRelation(\n      sqlContext: SQLContext,\n      mode: SaveMode,\n      parameters: Map[String, String],\n      data: DataFrame): BaseRelation\n}\n```\n\nHowever, the major goal of this API should also include saving the `DataFrame`.\n\nSince this API is critical for Data Source API developers, this PR is to correct the description.\n\n#### How was this patch tested?\nN/A\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13903 from gatorsmile/readUnderscoreFiles.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/4cbf611c1dc88111ff49d005e902ad5864799ede"},{"sha":"f6b497fcdddc705a9e1022e20b0dbc15da1b5a5a","author":{"name":"Prashant Sharma","email":"f36d2a4f118ffe5bbe5ae27e218772484c048736@in.ibm.com"},"message":"[SPARK-16128][SQL] Allow setting length of characters to be truncated to, in Dataset.show function.\n\n## What changes were proposed in this pull request?\n\nAllowing truncate to a specific number of character is convenient at times, especially while operating from the REPL. Sometimes those last few characters make all the difference, and showing everything brings in whole lot of noise.\n\n## How was this patch tested?\nExisting tests. + 1 new test in DataFrameSuite.\n\nFor SparkR and pyspark, existing tests and manual testing.\n\nAuthor: Prashant Sharma <prashsh1@in.ibm.com>\nAuthor: Prashant Sharma <prashant@apache.org>\n\nCloses #13839 from ScrapCodes/add_truncateTo_DF.show.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/f6b497fcdddc705a9e1022e20b0dbc15da1b5a5a"},{"sha":"e158478a9fff5e63ae0336a54b3f360d0cd38921","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-16242][MLLIB][PYSPARK] Conversion between old/new matrix columns in a DataFrame (Python)\n\n## What changes were proposed in this pull request?\nThis PR implements python wrappers for #13888 to convert old/new matrix columns in a DataFrame.\n\n## How was this patch tested?\nDoctest in python.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13935 from yanboliang/spark-16242.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/e158478a9fff5e63ae0336a54b3f360d0cd38921"},{"sha":"0923c4f5676691e28e70ecb05890e123540b91f0","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf\n\n## What changes were proposed in this pull request?\nWhen we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called).\n\n## How was this patch tested?\nNew tests and manual tests.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13931 from yhuai/SPARK-16224.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/0923c4f5676691e28e70ecb05890e123540b91f0"},{"sha":"1f2776df6e87a84991537ac20e4b8829472d3462","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16181][SQL] outer join with isNull filter may return wrong result\n\n## What changes were proposed in this pull request?\n\nThe root cause is: the output attributes of outer join are derived from its children, while they are actually different attributes(outer join can return null).\n\nWe have already added some special logic to handle it, e.g. `PushPredicateThroughJoin` won't push down predicates through outer join side, `FixNullability`.\n\nThis PR adds one more special logic in `FoldablePropagation`.\n\n## How was this patch tested?\n\nnew test in `DataFrameSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13884 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/1f2776df6e87a84991537ac20e4b8829472d3462"},{"sha":"26252f7064ba852e1bce6d8233a95aeb395f826a","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-15643][DOC][ML] Update spark.ml and spark.mllib migration guide from 1.6 to 2.0\n\n## What changes were proposed in this pull request?\nUpdate ```spark.ml``` and ```spark.mllib``` migration guide from 1.6 to 2.0.\n\n## How was this patch tested?\nDocs update, no tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13378 from yanboliang/spark-13448.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/26252f7064ba852e1bce6d8233a95aeb395f826a"},{"sha":"d59ba8e30751bbf91d49f5530b8242a12bbfb569","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[MINOR][SPARKR] update sparkR DataFrame.R comment\n\n## What changes were proposed in this pull request?\n\nupdate sparkR DataFrame.R comment\nSQLContext ==> SparkSession\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #13946 from WeichenXu123/sparkR_comment_update_sparkSession.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/d59ba8e30751bbf91d49f5530b8242a12bbfb569"},{"sha":"ae14f362355b131fcb3e3633da7bb14bdd2b6893","author":{"name":"Tom Magrino","email":"b4ff0ed6b2f378ecb18a44daba12916fd6d7add8@fb.com"},"message":"[SPARK-16148][SCHEDULER] Allow for underscores in TaskLocation in the Executor ID\n\n## What changes were proposed in this pull request?\n\nPreviously, the TaskLocation implementation would not allow for executor ids which include underscores.  This tweaks the string split used to get the hostname and executor id, allowing for underscores in the executor id.\n\nThis addresses the JIRA found here: https://issues.apache.org/jira/browse/SPARK-16148\n\nThis is moved over from a previous PR against branch-1.6: https://github.com/apache/spark/pull/13857\n\n## How was this patch tested?\n\nRan existing unit tests for core and streaming.  Manually ran a simple streaming job with an executor whose id contained underscores and confirmed that the job ran successfully.\n\nThis is my original work and I license the work to the project under the project's open source license.\n\nAuthor: Tom Magrino <tmagrino@fb.com>\n\nCloses #13858 from tmagrino/fixtasklocation.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/ae14f362355b131fcb3e3633da7bb14bdd2b6893"},{"sha":"1aad8c6e59c1e8b18a3eaa8ded93ff6ad05d83df","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16259][PYSPARK] cleanup options in DataFrame read/write API\n\n## What changes were proposed in this pull request?\n\nThere are some duplicated code for options in DataFrame reader/writer API, this PR clean them up, it also fix a bug for `escapeQuotes` of csv().\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13948 from davies/csv_options.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/1aad8c6e59c1e8b18a3eaa8ded93ff6ad05d83df"},{"sha":"35438fb0ad3bcda5c5a3a0ccde1a620699d012db","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-16175] [PYSPARK] handle None for UDT\n\n## What changes were proposed in this pull request?\n\nScala UDT will bypass all the null and will not pass them into serialize() and deserialize() of UDT, this PR update the Python UDT to do this as well.\n\n## How was this patch tested?\n\nAdded tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13878 from davies/udt_null.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/35438fb0ad3bcda5c5a3a0ccde1a620699d012db"},{"sha":"25520e976275e0d1e3bf9c73128ef4dec4618568","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16236][SQL] Add Path Option back to Load API in DataFrameReader\n\n#### What changes were proposed in this pull request?\nkoertkuipers identified the PR https://github.com/apache/spark/pull/13727/ changed the behavior of `load` API. After the change, the `load` API does not add the value of `path` into the `options`. Thank you!\n\nThis PR is to add the option `path` back to `load()` API in `DataFrameReader`, if and only if users specify one and only one `path` in the `load` API. For example, users can see the `path` option after the following API call,\n```Scala\nspark.read\n  .format(\"parquet\")\n  .load(\"/test\")\n```\n\n#### How was this patch tested?\nAdded test cases.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13933 from gatorsmile/optionPath.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/25520e976275e0d1e3bf9c73128ef4dec4618568"},{"sha":"8a977b065418f07d2bf4fe1607a5534c32d04c47","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16100][SQL] fix bug when use Map as the buffer type of Aggregator\n\n## What changes were proposed in this pull request?\n\nThe root cause is in `MapObjects`. Its parameter `loopVar` is not declared as child, but sometimes can be same with `lambdaFunction`(e.g. the function that takes `loopVar` and produces `lambdaFunction` may be `identity`), which is a child. This brings trouble when call `withNewChildren`, it may mistakenly treat `loopVar` as a child and cause `IndexOutOfBoundsException: 0` later.\n\nThis PR fixes this bug by simply pulling out the paremters from `LambdaVariable` and pass them to `MapObjects` directly.\n\n## How was this patch tested?\n\nnew test in `DatasetAggregatorSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13835 from cloud-fan/map-objects.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/8a977b065418f07d2bf4fe1607a5534c32d04c47"},{"sha":"3554713a163c58ca176ffde87d2c6e4a91bacb50","author":{"name":"James Thomas","email":"4c3f70f7e07111950bf31cd977787d757ef55837@gmail.com"},"message":"[SPARK-16114][SQL] structured streaming network word count examples\n\n## What changes were proposed in this pull request?\n\nNetwork word count example for structured streaming\n\n## How was this patch tested?\n\nRun locally\n\nAuthor: James Thomas <jamesjoethomas@gmail.com>\nAuthor: James Thomas <jamesthomas@Jamess-MacBook-Pro.local>\n\nCloses #13816 from jjthomas/master.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/3554713a163c58ca176ffde87d2c6e4a91bacb50"},{"sha":"5545b791096756b07b3207fb3de13b68b9a37b00","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[MINOR][DOCS][STRUCTURED STREAMING] Minor doc fixes around `DataFrameWriter` and `DataStreamWriter`\n\n## What changes were proposed in this pull request?\n\nFixes a couple old references to `DataFrameWriter.startStream` to `DataStreamWriter.start\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #13952 from brkyvz/minor-doc-fix.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/5545b791096756b07b3207fb3de13b68b9a37b00"},{"sha":"823518c2b5259c8a954431467639198c808c9198","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR] add csv tests\n\n## What changes were proposed in this pull request?\n\nAdd unit tests for csv data for SPARKR\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13904 from felixcheung/rcsv.","distinct":true,"url":"https://api.github.com/repos/jodersky/spark/commits/823518c2b5259c8a954431467639198c808c9198"}]}
{"actor":{"display_login":"jodersky"}}