{"push_id":1287530910,"size":1000,"distinct_size":585,"ref":"refs/heads/imputer","head":"544a65c82a7d921bdff73998e8b350e11b51dcbe","before":"4e0743139796ac53df2554cfa53736b8035bae15","commits":[{"sha":"675a73715d3c8adb9d9a9dce5f76a2db5106790c","author":{"name":"Mortada Mehyar","email":"40cdf11c0f3d1304c3f3b5b64ad4d078be184305@gmail.com"},"message":"[DOCUMENTATION] fixed groupby aggregation example for pyspark\n\n## What changes were proposed in this pull request?\n\nfixing documentation for the groupby/agg example in python\n\n## How was this patch tested?\n\nthe existing example in the documentation dose not contain valid syntax (missing parenthesis) and is not using `Column` in the expression for `agg()`\n\nafter the fix here's how I tested it:\n\n```\nIn [1]: from pyspark.sql import Row\n\nIn [2]: import pyspark.sql.functions as func\n\nIn [3]: %cpaste\nPasting code; enter '--' alone on the line to stop or use Ctrl-D.\n:records = [{'age': 19, 'department': 1, 'expense': 100},\n: {'age': 20, 'department': 1, 'expense': 200},\n: {'age': 21, 'department': 2, 'expense': 300},\n: {'age': 22, 'department': 2, 'expense': 300},\n: {'age': 23, 'department': 3, 'expense': 300}]\n:--\n\nIn [4]: df = sqlContext.createDataFrame([Row(**d) for d in records])\n\nIn [5]: df.groupBy(\"department\").agg(df[\"department\"], func.max(\"age\"), func.sum(\"expense\")).show()\n\n+----------+----------+--------+------------+\n|department|department|max(age)|sum(expense)|\n+----------+----------+--------+------------+\n|         1|         1|      20|         300|\n|         2|         2|      22|         600|\n|         3|         3|      23|         300|\n+----------+----------+--------+------------+\n\nAuthor: Mortada Mehyar <mortada.mehyar@gmail.com>\n\nCloses #13587 from mortada/groupby_agg_doc_fix.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/675a73715d3c8adb9d9a9dce5f76a2db5106790c"},{"sha":"16ca32eace39c423224b0ec25922038fd45c501a","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-15823][PYSPARK][ML] Add @property for 'accuracy' in MulticlassMetrics\n\n## What changes were proposed in this pull request?\n`accuracy` should be decorated with `property` to keep step with other methods in `pyspark.MulticlassMetrics`, like `weightedPrecision`, `weightedRecall`, etc\n\n## How was this patch tested?\nmanual tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #13560 from zhengruifeng/add_accuracy_property.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/16ca32eace39c423224b0ec25922038fd45c501a"},{"sha":"cdd7f5a57a21d4a8f93456d149f65859c96190cf","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-15837][ML][PYSPARK] Word2vec python add maxsentence parameter\n\n## What changes were proposed in this pull request?\n\nWord2vec python add maxsentence parameter.\n\n## How was this patch tested?\n\nExisting test.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #13578 from WeichenXu123/word2vec_python_add_maxsentence.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/cdd7f5a57a21d4a8f93456d149f65859c96190cf"},{"sha":"abdb5d42c5802c8f60876aa1285c803d02881258","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-15812][SQ][STREAMING] Added support for sorting after streaming aggregation with complete mode\n\n## What changes were proposed in this pull request?\n\nWhen the output mode is complete, then the output of a streaming aggregation essentially will contain the complete aggregates every time. So this is not different from a batch dataset within an incremental execution. Other non-streaming operations should be supported on this dataset. In this PR, I am just adding support for sorting, as it is a common useful functionality. Support for other operations will come later.\n\n## How was this patch tested?\nAdditional unit tests.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #13549 from tdas/SPARK-15812.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/abdb5d42c5802c8f60876aa1285c803d02881258"},{"sha":"0ec279ffdf92853965e327a9f0f6956cacb7a23e","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-15753][SQL] Move Analyzer stuff to Analyzer from DataFrameWriter\n\n## What changes were proposed in this pull request?\n\nThis patch moves some codes in `DataFrameWriter.insertInto` that belongs to `Analyzer`.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #13496 from viirya/move-analyzer-stuff.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/0ec279ffdf92853965e327a9f0f6956cacb7a23e"},{"sha":"254bc8c34e70241508bdfc8ff42a65491f5280cd","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15866] Rename listAccumulator collectionAccumulator\n\n## What changes were proposed in this pull request?\nSparkContext.listAccumulator, by Spark's convention, makes it sound like \"list\" is a verb and the method should return a list of accumulators. This patch renames the method and the class collection accumulator.\n\n## How was this patch tested?\nUpdated test case to reflect the names.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13594 from rxin/SPARK-15866.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/254bc8c34e70241508bdfc8ff42a65491f5280cd"},{"sha":"7d7a0a5e0749909e97d90188707cc9220a1bb73a","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __str__ method similar to Scala API\n\n## What changes were proposed in this pull request?\nAdding __str__ to RFormula and model that will show the set formula param and resolved formula.  This is currently present in the Scala API, found missing in PySpark during Spark 2.0 coverage review.\n\n## How was this patch tested?\nrun pyspark-ml tests locally\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #13481 from BryanCutler/pyspark-ml-rformula_str-SPARK-15738.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/7d7a0a5e0749909e97d90188707cc9220a1bb73a"},{"sha":"2413fce9d6812a91eeffb4435c2b5b361d23214b","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-15743][SQL] Prevent saving with all-column partitioning\n\n## What changes were proposed in this pull request?\n\nWhen saving datasets on storage, `partitionBy` provides an easy way to construct the directory structure. However, if a user choose all columns as partition columns, some exceptions occurs.\n\n- **ORC with all column partitioning**: `AnalysisException` on **future read** due to schema inference failure.\n ```scala\nscala> spark.range(10).write.format(\"orc\").mode(\"overwrite\").partitionBy(\"id\").save(\"/tmp/data\")\n\nscala> spark.read.format(\"orc\").load(\"/tmp/data\").collect()\norg.apache.spark.sql.AnalysisException: Unable to infer schema for ORC at /tmp/data. It must be specified manually;\n```\n\n- **Parquet with all-column partitioning**: `InvalidSchemaException` on **write execution** due to Parquet limitation.\n ```scala\nscala> spark.range(100).write.format(\"parquet\").mode(\"overwrite\").partitionBy(\"id\").save(\"/tmp/data\")\n[Stage 0:>                                                          (0 + 8) / 8]16/06/02 16:51:17\nERROR Utils: Aborting task\norg.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not support empty group without leaves. Empty group: spark_schema\n... (lots of error messages)\n```\n\nAlthough some formats like JSON support all-column partitioning without any problem, it seems not a good idea to make lots of empty directories.\n\nThis PR prevents saving with all-column partitioning by consistently raising `AnalysisException` before executing save operation.\n\n## How was this patch tested?\n\nNewly added `PartitioningUtilsSuite`.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13486 from dongjoon-hyun/SPARK-15743.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/2413fce9d6812a91eeffb4435c2b5b361d23214b"},{"sha":"2c8f40cea113b597fbaf1cdd80a5b8bdd66155fb","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-15766][SPARKR] R should export is.nan\n\n## What changes were proposed in this pull request?\n\nWhen reviewing SPARK-15545, we found that is.nan is not exported, which should be exported.\n\nAdd it to the NAMESPACE.\n\n## How was this patch tested?\n\nManual tests.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #13508 from wangmiao1981/unused.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/2c8f40cea113b597fbaf1cdd80a5b8bdd66155fb"},{"sha":"5c16ad0d522e5124a6977533077afb7b38fc42a1","author":{"name":"Kay Ousterhout","email":"c3633890c4773690888d598adeeb420b7a01d083@gmail.com"},"message":"Revert [SPARK-14485][CORE] ignore task finished for executor lost\n\nThis reverts commit 695dbc816a6d70289abeb145cb62ff4e62b3f49b.\n\nThis change is being reverted because it hurts performance of some jobs, and\nonly helps in a narrow set of cases.  For more discussion, refer to the JIRA.\n\nAuthor: Kay Ousterhout <kayousterhout@gmail.com>\n\nCloses #13580 from kayousterhout/revert-SPARK-14485.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/5c16ad0d522e5124a6977533077afb7b38fc42a1"},{"sha":"fb219029dd1b8d2783c3e202361401048296595c","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-15871][SQL] Add `assertNotPartitioned` check in `DataFrameWriter`\n\n## What changes were proposed in this pull request?\n\nIt doesn't make sense to specify partitioning parameters, when we write data out from Datasets/DataFrames into `jdbc` tables or streaming `ForeachWriter`s.\n\nThis patch adds `assertNotPartitioned` check in `DataFrameWriter`.\n\n<table>\n<tr>\n\t<td align=\"center\"><strong>operation</strong></td>\n\t<td align=\"center\"><strong>should check not partitioned?</strong></td>\n</tr>\n<tr>\n\t<td align=\"center\">mode</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">outputMode</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">trigger</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">format</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">option/options</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">partitionBy</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">bucketBy</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">sortBy</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">save</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">queryName</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">startStream</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">foreach</td>\n\t<td align=\"center\">yes</td>\n</tr>\n<tr>\n\t<td align=\"center\">insertInto</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">saveAsTable</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">jdbc</td>\n\t<td align=\"center\">yes</td>\n</tr>\n<tr>\n\t<td align=\"center\">json</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">parquet</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">orc</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">text</td>\n\t<td align=\"center\"></td>\n</tr>\n<tr>\n\t<td align=\"center\">csv</td>\n\t<td align=\"center\"></td>\n</tr>\n</table>\n\n## How was this patch tested?\n\nNew dedicated tests.\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #13597 from lw-lin/add-assertNotPartitioned.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/fb219029dd1b8d2783c3e202361401048296595c"},{"sha":"667d4ea7b35f285954ea7cb719b7c80581e31f4d","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-6320][SQL] Move planLater method into GenericStrategy.\n\n## What changes were proposed in this pull request?\n\nThis PR moves `QueryPlanner.planLater()` method into `GenericStrategy` for extra strategies to be able to use `planLater` in its strategy.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #13147 from ueshin/issues/SPARK-6320.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/667d4ea7b35f285954ea7cb719b7c80581e31f4d"},{"sha":"865ec32dd997e63aea01a871d1c7b4947f43c111","author":{"name":"Sandeep Singh","email":"69043e619d79bcf4ae068e5b131d66c33e73727d@techaddict.me"},"message":"[MINOR][X][X] Replace all occurrences of None: Option with Option.empty\n\n## What changes were proposed in this pull request?\nReplace all occurrences of `None: Option[X]` with `Option.empty[X]`\n\n## How was this patch tested?\nExisiting Tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13591 from techaddict/minor-7.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/865ec32dd997e63aea01a871d1c7b4947f43c111"},{"sha":"026eb90644be7685971dacaabae67a293edd0133","author":{"name":"wangyang","email":"eba22d259fe8b5d85428661d5efb47a9dcdcf825@haizhi.com"},"message":"[SPARK-15875] Try to use Seq.isEmpty and Seq.nonEmpty instead of Seq.length == 0 and Seq.length > 0\n\n## What changes were proposed in this pull request?\n\nIn scala, immutable.List.length is an expensive operation so we should\navoid using Seq.length == 0 or Seq.lenth > 0, and use Seq.isEmpty and Seq.nonEmpty instead.\n\n## How was this patch tested?\nexisting tests\n\nAuthor: wangyang <wangyang@haizhi.com>\n\nCloses #13601 from yangw1234/isEmpty.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/026eb90644be7685971dacaabae67a293edd0133"},{"sha":"e05a2feebe928df691d5a8f42f22e088c6263dcf","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-15825] [SQL] Fix SMJ invalid results\n\n## What changes were proposed in this pull request?\nCode generated `SortMergeJoin` failed with wrong results when using structs as keys. This could (eventually) be traced back to the use of a wrong row reference when comparing structs.\n\n## How was this patch tested?\nTBD\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #13589 from hvanhovell/SPARK-15822.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/e05a2feebe928df691d5a8f42f22e088c6263dcf"},{"sha":"aec502d9114ad8e18bfbbd63f38780e076d326d1","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-15654] [SQL] fix non-splitable files for text based file formats\n\n## What changes were proposed in this pull request?\n\nCurrently, we always split the files when it's bigger than maxSplitBytes, but Hadoop LineRecordReader does not respect the splits for compressed files correctly, we should have a API for FileFormat to check whether the file could be splitted or not.\n\nThis PR is based on #13442, closes #13442\n\n## How was this patch tested?\n\nadd regression tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13531 from davies/fix_split.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/aec502d9114ad8e18bfbbd63f38780e076d326d1"},{"sha":"127a6678d7af6b5164a115be7c64525bb80001fe","author":{"name":"Sela","email":"a9fb1fba57482b5fd357a15ce2d7ec683714fb49@paypal.com"},"message":"[SPARK-15489][SQL] Dataset kryo encoder won't load custom user settings\n\n## What changes were proposed in this pull request?\n\nSerializer instantiation will consider existing SparkConf\n\n## How was this patch tested?\nmanual test with `ImmutableList` (Guava) and `kryo-serializers`'s `Immutable*Serializer` implementations.\n\nAdded Test Suite.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\nAuthor: Sela <ansela@paypal.com>\n\nCloses #13424 from amitsela/SPARK-15489.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/127a6678d7af6b5164a115be7c64525bb80001fe"},{"sha":"2022afe57dbf8cb0c9909399962c4a3649e0601c","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-15773][CORE][EXAMPLE] Avoid creating local variable `sc` in examples if possible\n\n## What changes were proposed in this pull request?\n\nInstead of using local variable `sc` like the following example, this PR uses `spark.sparkContext`. This makes examples more concise, and also fixes some misleading, i.e., creating SparkContext from SparkSession.\n```\n-    println(\"Creating SparkContext\")\n-    val sc = spark.sparkContext\n-\n     println(\"Writing local file to DFS\")\n     val dfsFilename = dfsDirPath + \"/dfs_read_write_test\"\n-    val fileRDD = sc.parallelize(fileContents)\n+    val fileRDD = spark.sparkContext.parallelize(fileContents)\n```\n\nThis will change 12 files (+30 lines, -52 lines).\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #13520 from dongjoon-hyun/SPARK-15773.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/2022afe57dbf8cb0c9909399962c4a3649e0601c"},{"sha":"54f758b5fc60ecb0da6b191939a72ef5829be38c","author":{"name":"Narine Kokhlikyan","email":"d599ff9888c8f1e63f53ba6c235c3984fad06ffb@gmail.com"},"message":"[SPARK-15884][SPARKR][SQL] Overriding stringArgs in MapPartitionsInR\n\n## What changes were proposed in this pull request?\n\nAs discussed in https://github.com/apache/spark/pull/12836\nwe need to override stringArgs method in MapPartitionsInR in order to avoid too large strings generated by \"stringArgs\" method based on the input arguments.\n\nIn this case exclude some of the input arguments: serialized R objects.\n\n## How was this patch tested?\nExisting test cases\n\nAuthor: Narine Kokhlikyan <narine.kokhlikyan@gmail.com>\n\nCloses #13610 from NarineK/dapply_MapPartitionsInR_stringArgs.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/54f758b5fc60ecb0da6b191939a72ef5829be38c"},{"sha":"bba5d7999f7b3ae9d816ea552ba9378fea1615a6","author":{"name":"Liang-Chi Hsieh","email":"36eec8bb19b3166395ab9bc07614f8325d08f990@tw.ibm.com"},"message":"[SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader\n\n## What changes were proposed in this pull request?\n\nThe base class `SpecificParquetRecordReaderBase` used for vectorized parquet reader will try to get pushed-down filters from the given configuration. This pushed-down filters are used for RowGroups-level filtering. However, we don't set up the filters to push down into the configuration. In other words, the filters are not actually pushed down to do RowGroups-level filtering. This patch is to fix this and tries to set up the filters for pushing down to configuration for the reader.\n\n## How was this patch tested?\nExisting tests should be passed.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #13371 from viirya/vectorized-reader-push-down-filter.","distinct":false,"url":"https://api.github.com/repos/hhbyyh/spark/commits/bba5d7999f7b3ae9d816ea552ba9378fea1615a6"}]}
{"actor":{"display_login":"hhbyyh"}}