{"push_id":1168200009,"size":15,"distinct_size":15,"ref":"refs/heads/master","head":"f3a768b7b96f00f33d2fe4e6c0bf4acf373ad4f4","before":"e2b7eba87cdf67fa737c32f5f6ca075445ff28cb","commits":[{"sha":"a46553cbacf0e4012df89fe55385dec5beaa680a","author":{"name":"Davies Liu","email":"0974b7f8fcee704ff653a2542bba493bdfd47646@gmail.com"},"message":"[SPARK-16086] [SQL] fix Python UDF without arguments (for 1.6)\n\nFix the bug for Python UDF that does not have any arguments.\n\nAdded regression tests.\n\nAuthor: Davies Liu <davies.liu@gmail.com>\n\nCloses #13793 from davies/fix_no_arguments.\n\n(cherry picked from commit abe36c53d126bb580e408a45245fd8e81806869c)\nSigned-off-by: Davies Liu <davies.liu@gmail.com>","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/a46553cbacf0e4012df89fe55385dec5beaa680a"},{"sha":"217db56ba11fcdf9e3a81946667d1d99ad7344ee","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-15294][R] Add `pivot` to SparkR\n\n## What changes were proposed in this pull request?\n\nThis PR adds `pivot` function to SparkR for API parity. Since this PR is based on https://github.com/apache/spark/pull/13295 , mhnatiuk should be credited for the work he did.\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new testcase.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13786 from dongjoon-hyun/SPARK-15294.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/217db56ba11fcdf9e3a81946667d1d99ad7344ee"},{"sha":"c775bf09e0c3540f76de3f15d3fd35112a4912c1","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-13792][SQL] Limit logging of bad records in CSV data source\n\n## What changes were proposed in this pull request?\nThis pull request adds a new option (maxMalformedLogPerPartition) in CSV reader to limit the maximum of logging message Spark generates per partition for malformed records.\n\nThe error log looks something like\n```\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: Dropping malformed line: adsf,1,4\n16/06/20 18:50:14 WARN CSVRelation: More than 10 malformed records have been found on this partition. Malformed records from now on will not be logged.\n```\n\nCloses #12173\n\n## How was this patch tested?\nManually tested.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13795 from rxin/SPARK-13792.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/c775bf09e0c3540f76de3f15d3fd35112a4912c1"},{"sha":"d9a3a2a0bec504d17d3b94104d449ee3bd850120","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16056][SPARK-16057][SPARK-16058][SQL] Fix Multiple Bugs in Column Partitioning in JDBC Source\n\n#### What changes were proposed in this pull request?\nThis PR is to fix the following bugs:\n\n**Issue 1: Wrong Results when lowerBound is larger than upperBound in Column Partitioning**\n```scala\nspark.read.jdbc(\n  url = urlWithUserAndPass,\n  table = \"TEST.seq\",\n  columnName = \"id\",\n  lowerBound = 4,\n  upperBound = 0,\n  numPartitions = 3,\n  connectionProperties = new Properties)\n```\n**Before code changes:**\nThe returned results are wrong and the generated partitions are wrong:\n```\n  Part 0 id < 3 or id is null\n  Part 1 id >= 3 AND id < 2\n  Part 2 id >= 2\n```\n**After code changes:**\nIssue an `IllegalArgumentException` exception:\n```\nOperation not allowed: the lower bound of partitioning column is larger than the upper bound. lowerBound: 5; higherBound: 1\n```\n**Issue 2: numPartitions is more than the number of key values between upper and lower bounds**\n```scala\nspark.read.jdbc(\n  url = urlWithUserAndPass,\n  table = \"TEST.seq\",\n  columnName = \"id\",\n  lowerBound = 1,\n  upperBound = 5,\n  numPartitions = 10,\n  connectionProperties = new Properties)\n```\n**Before code changes:**\nReturned correct results but the generated partitions are very inefficient, like:\n```\nPartition 0: id < 1 or id is null\nPartition 1: id >= 1 AND id < 1\nPartition 2: id >= 1 AND id < 1\nPartition 3: id >= 1 AND id < 1\nPartition 4: id >= 1 AND id < 1\nPartition 5: id >= 1 AND id < 1\nPartition 6: id >= 1 AND id < 1\nPartition 7: id >= 1 AND id < 1\nPartition 8: id >= 1 AND id < 1\nPartition 9: id >= 1\n```\n**After code changes:**\nAdjust `numPartitions` and can return the correct answers:\n```\nPartition 0: id < 2 or id is null\nPartition 1: id >= 2 AND id < 3\nPartition 2: id >= 3 AND id < 4\nPartition 3: id >= 4\n```\n**Issue 3: java.lang.ArithmeticException when numPartitions is zero**\n```Scala\nspark.read.jdbc(\n  url = urlWithUserAndPass,\n  table = \"TEST.seq\",\n  columnName = \"id\",\n  lowerBound = 0,\n  upperBound = 4,\n  numPartitions = 0,\n  connectionProperties = new Properties)\n```\n**Before code changes:**\nGot the following exception:\n```\n  java.lang.ArithmeticException: / by zero\n```\n**After code changes:**\nAble to return a correct answer by disabling column partitioning when numPartitions is equal to or less than zero\n\n#### How was this patch tested?\nAdded test cases to verify the results\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13773 from gatorsmile/jdbcPartitioning.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/d9a3a2a0bec504d17d3b94104d449ee3bd850120"},{"sha":"18a8a9b1f4114211cd108efda5672f2bd2c6e5cd","author":{"name":"Xiangrui Meng","email":"e4720f5dc69a5967cefd679a46054c60c7e8d310@databricks.com"},"message":"[SPARK-16074][MLLIB] expose VectorUDT/MatrixUDT in a public API\n\n## What changes were proposed in this pull request?\n\nBoth VectorUDT and MatrixUDT are private APIs, because UserDefinedType itself is private in Spark. However, in order to let developers implement their own transformers and estimators, we should expose both types in a public API to simply the implementation of transformSchema, transform, etc. Otherwise, they need to get the data types using reflection.\n\n## How was this patch tested?\n\nUnit tests in Scala and Java.\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #13789 from mengxr/SPARK-16074.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/18a8a9b1f4114211cd108efda5672f2bd2c6e5cd"},{"sha":"4f7f1c436205630ab77d3758d7210cc1a2f0d04a","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16044][SQL] input_file_name() returns empty strings in data sources based on NewHadoopRDD\n\n## What changes were proposed in this pull request?\n\nThis PR makes `input_file_name()` function return the file paths not empty strings for external data sources based on `NewHadoopRDD`, such as [spark-redshift](https://github.com/databricks/spark-redshift/blob/cba5eee1ab79ae8f0fa9e668373a54d2b5babf6b/src/main/scala/com/databricks/spark/redshift/RedshiftRelation.scala#L149) and [spark-xml](https://github.com/databricks/spark-xml/blob/master/src/main/scala/com/databricks/spark/xml/util/XmlFile.scala#L39-L47).\n\nThe codes with the external data sources below:\n\n```scala\ndf.select(input_file_name).show()\n```\n\nwill produce\n\n- **Before**\n  ```\n+-----------------+\n|input_file_name()|\n+-----------------+\n|                 |\n+-----------------+\n```\n\n- **After**\n  ```\n+--------------------+\n|   input_file_name()|\n+--------------------+\n|file:/private/var...|\n+--------------------+\n```\n\n## How was this patch tested?\n\nUnit tests in `ColumnExpressionSuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #13759 from HyukjinKwon/SPARK-16044.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/4f7f1c436205630ab77d3758d7210cc1a2f0d04a"},{"sha":"07367533de68817e1e6cf9cf2b056a04dd160c8a","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-16025][CORE] Document OFF_HEAP storage level in 2.0\n\nThis has changed from 1.6, and now stores memory off-heap using spark's off-heap support instead of in tachyon.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #13744 from ericl/spark-16025.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/07367533de68817e1e6cf9cf2b056a04dd160c8a"},{"sha":"58f6e27dd70f476f99ac8204e6b405bced4d6de1","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15863][SQL][DOC][SPARKR] sql programming guide updates to include sparkSession in R\n\n## What changes were proposed in this pull request?\n\nUpdate doc as per discussion in PR #13592\n\n## How was this patch tested?\n\nmanual\n\nshivaram liancheng\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13799 from felixcheung/rsqlprogrammingguide.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/58f6e27dd70f476f99ac8204e6b405bced4d6de1"},{"sha":"41e0ffb19f678e9b1e87f747a5e4e3d44964e39a","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-15894][SQL][DOC] Update docs for controlling #partitions\n\n## What changes were proposed in this pull request?\nUpdate docs for two parameters `spark.sql.files.maxPartitionBytes` and `spark.sql.files.openCostInBytes ` in Other Configuration Options.\n\n## How was this patch tested?\nN/A\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #13797 from maropu/SPARK-15894-2.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/41e0ffb19f678e9b1e87f747a5e4e3d44964e39a"},{"sha":"09f4ceaeb0a99874f774e09d868fdf907ecf256f","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR][DOCS] R code doc cleanup\n\n## What changes were proposed in this pull request?\n\nI ran a full pass from A to Z and fixed the obvious duplications, improper grouping etc.\n\nThere are still more doc issues to be cleaned up.\n\n## How was this patch tested?\n\nmanual tests\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13798 from felixcheung/rdocseealso.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/09f4ceaeb0a99874f774e09d868fdf907ecf256f"},{"sha":"843a1eba8ec9d5a7beac0c74b54d24cb3c41b45a","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-15319][SPARKR][DOCS] Fix SparkR doc layout for corr and other DataFrame stats functions\n\n## What changes were proposed in this pull request?\n\nDoc only changes. Please see screenshots.\n\nBefore:\nhttp://spark.apache.org/docs/latest/api/R/statfunctions.html\n![image](https://cloud.githubusercontent.com/assets/8969467/15264110/cd458826-1924-11e6-85bd-8ee2e2e1a85f.png)\n\nAfter\n![image](https://cloud.githubusercontent.com/assets/8969467/16218452/b9e89f08-3732-11e6-969d-a3a1796e7ad0.png)\n(please ignore the style differences - this is due to not having the css in my local copy)\n\nThis is still a bit weird. As discussed in SPARK-15237, I think the better approach is to separate out the DataFrame stats function instead of putting everything on one page. At least now it is clearer which description is on which function.\n\n## How was this patch tested?\n\nBuild doc\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\nAuthor: felixcheung <felixcheung_m@hotmail.com>\n\nCloses #13109 from felixcheung/rstatdoc.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/843a1eba8ec9d5a7beac0c74b54d24cb3c41b45a"},{"sha":"ce49bfc2550ba8f5a33235c7fc3b88201d63c276","author":{"name":"Xiangrui Meng","email":"e4720f5dc69a5967cefd679a46054c60c7e8d310@databricks.com"},"message":"Revert \"[SPARK-16086] [SQL] fix Python UDF without arguments (for 1.6)\"\n\nThis reverts commit a46553cbacf0e4012df89fe55385dec5beaa680a.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/ce49bfc2550ba8f5a33235c7fc3b88201d63c276"},{"sha":"37494a18e8d6e22113338523d6498e00ac9725ea","author":{"name":"Nick Pentreath","email":"01c7185e211fff3925ce587674cd3ea63724775a@za.ibm.com"},"message":"[SPARK-10258][DOC][ML] Add @Since annotations to ml.feature\n\nThis PR adds missing `Since` annotations to `ml.feature` package.\n\nCloses #8505.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Nick Pentreath <nickp@za.ibm.com>\n\nCloses #13641 from MLnick/add-since-annotations.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/37494a18e8d6e22113338523d6498e00ac9725ea"},{"sha":"a58f40239444d42adbc480ddde02cbb02a79bbe4","author":{"name":"Yuhao Yang","email":"718d4d5b3fde31fb7d3c3b83ad143ab39ee820cb@gmail.com"},"message":"[SPARK-16045][ML][DOC] Spark 2.0 ML.feature: doc update for stopwords and binarizer\n\n## What changes were proposed in this pull request?\n\njira: https://issues.apache.org/jira/browse/SPARK-16045\n2.0 Audit: Update document for StopWordsRemover and Binarizer.\n\n## How was this patch tested?\n\nmanual review for doc\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\nAuthor: Yuhao Yang <yuhao.yang@intel.com>\n\nCloses #13375 from hhbyyh/stopdoc.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/a58f40239444d42adbc480ddde02cbb02a79bbe4"},{"sha":"f3a768b7b96f00f33d2fe4e6c0bf4acf373ad4f4","author":{"name":"bomeng","email":"87283f0b743e3f373df8bb94602beccfc5bcbe89@us.ibm.com"},"message":"[SPARK-16084][SQL] Minor comments update for \"DESCRIBE\" table\n\n## What changes were proposed in this pull request?\n\n1. FORMATTED is actually supported, but partition is not supported;\n2. Remove parenthesis as it is not necessary just like anywhere else.\n\n## How was this patch tested?\n\nMinor issue. I do not think it needs a test case!\n\nAuthor: bomeng <bmeng@us.ibm.com>\n\nCloses #13791 from bomeng/SPARK-16084.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/f3a768b7b96f00f33d2fe4e6c0bf4acf373ad4f4"}]}
{"actor":{"display_login":"ueshin"}}