{"push_id":1036684727,"size":137,"distinct_size":137,"ref":"refs/heads/master","head":"eef1d61a751c776c1b74d84a5e46604b299bfa44","before":"eacd9d8eda68260bbda7b0cd07410321dffaf428","commits":[{"sha":"d9e8f26d0334f393e3b02d7a3b607be54a2a5efe","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13924][SQL] officially support multi-insert\n\n## What changes were proposed in this pull request?\n\nThere is a feature of hive SQL called multi-insert. For example:\n```\nFROM src\nINSERT OVERWRITE TABLE dest1\nSELECT key + 1\nINSERT OVERWRITE TABLE dest2\nSELECT key WHERE key > 2\nINSERT OVERWRITE TABLE dest3\nSELECT col EXPLODE(arr) exp AS col\n...\n```\n\nWe partially support it currently, with some limitations: 1) WHERE can't reference columns produced by LATERAL VIEW. 2) It's not executed eagerly, i.e. `sql(\"...multi-insert clause...\")` won't take place right away like other commands, e.g. CREATE TABLE.\n\nThis PR removes these limitations and make us fully support multi-insert.\n\n## How was this patch tested?\n\nnew tests in `SQLQuerySuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11754 from cloud-fan/lateral-view.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/d9e8f26d0334f393e3b02d7a3b607be54a2a5efe"},{"sha":"d9670f84739b0840501b19b8cb0e851850edb8c1","author":{"email":"50c4b4c4c26f83ee4bbc45a85db1c3c80b5aecdb@intel.com","name":"Cheng Hao"},"message":"[SPARK-13894][SQL] SqlContext.range return type from DataFrame to DataSet\n\n## What changes were proposed in this pull request?\nhttps://issues.apache.org/jira/browse/SPARK-13894\nChange the return type of the `SQLContext.range` API from `DataFrame` to `Dataset`.\n\n## How was this patch tested?\nNo additional unit test required.\n\nAuthor: Cheng Hao <hao.cheng@intel.com>\n\nCloses #11730 from chenghao-intel/range.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/d9670f84739b0840501b19b8cb0e851850edb8c1"},{"sha":"91984978e7c86650d4cf523724b4a1aeaaecf260","author":{"email":"21ef94aec54fae1a36abbd011347596ae66d9959@foxmail.com","name":"Zheng RuiFeng"},"message":"[SPARK-13816][GRAPHX] Add parameter checks for algorithms in Graphx\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13816\n\n## What changes were proposed in this pull request?\n\nAdd parameter checks for algorithms in Graphx: Pregel,LabelPropagation,PageRank,SVDPlusPlus\n\n## How was this patch tested?\n\nmanual tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #11655 from zhengruifeng/graphx_param_check.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/91984978e7c86650d4cf523724b4a1aeaaecf260"},{"sha":"1d1de28a3c3c3a4bc37dc7565b9178a712df493a","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13827][SQL] Can't add subquery to an operator with same-name outputs while generate SQL string\n\n## What changes were proposed in this pull request?\n\nThis PR tries to solve a fundamental issue in the `SQLBuilder`. When we want to turn a logical plan into SQL string and put it after FROM clause, we need to wrap it with a sub-query. However, a logical plan is allowed to have same-name outputs with different qualifiers(e.g. the `Join` operator), and this kind of plan can't be put under a subquery as we will erase and assign a new qualifier to all outputs and make it impossible to distinguish same-name outputs.\n\nTo solve this problem, this PR renames all attributes with globally unique names(using exprId), so that we don't need qualifiers to resolve ambiguity anymore.\n\nFor example, `SELECT x.key, MAX(y.key) OVER () FROM t x JOIN t y`, we will parse this SQL to a Window operator and a Project operator, and add a sub-query between them. The generated SQL looks like:\n```\nSELECT sq_1.key, sq_1.max\nFROM (\n    SELECT sq_0.key, sq_0.key, MAX(sq_0.key) OVER () AS max\n    FROM (\n        SELECT x.key, y.key FROM t1 AS x JOIN t2 AS y\n    ) AS sq_0\n) AS sq_1\n```\nYou can see, the `key` columns become ambiguous after `sq_0`.\n\nAfter this PR, it will generate something like:\n```\nSELECT attr_30 AS key, attr_37 AS max\nFROM (\n    SELECT attr_30, attr_37\n    FROM (\n        SELECT attr_30, attr_35, MAX(attr_35) AS attr_37\n        FROM (\n            SELECT attr_30, attr_35 FROM\n                (SELECT key AS attr_30 FROM t1) AS sq_0\n            INNER JOIN\n                (SELECT key AS attr_35 FROM t1) AS sq_1\n        ) AS sq_2\n    ) AS sq_3\n) AS sq_4\n```\nThe outermost SELECT is used to turn the generated named to real names back, and the innermost SELECT is used to alias real columns to our generated names. Between them, there is no name ambiguity anymore.\n\n## How was this patch tested?\n\nexisting tests and new tests in LogicalPlanToSQLSuite.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11658 from cloud-fan/gensql.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/1d1de28a3c3c3a4bc37dc7565b9178a712df493a"},{"sha":"c4bd57602c0b14188d364bb475631bf473d25082","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12721][SQL] SQL Generation for Script Transformation\n\n#### What changes were proposed in this pull request?\n\nThis PR is to convert to SQL from analyzed logical plans containing operator `ScriptTransformation`.\n\nFor example, below is the SQL containing `Transform`\n```\nSELECT TRANSFORM (a, b, c, d) USING 'cat' FROM parquet_t2\n```\n\nIts logical plan is like\n```\nScriptTransformation [a#210L,b#211L,c#212L,d#213L], cat, [key#208,value#209], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim,\t)),List((field.delim,\t)),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),true)\n+- SubqueryAlias parquet_t2\n   +- Relation[a#210L,b#211L,c#212L,d#213L] ParquetRelation\n```\n\nThe generated SQL will be like\n```\nSELECT TRANSFORM (`parquet_t2`.`a`, `parquet_t2`.`b`, `parquet_t2`.`c`, `parquet_t2`.`d`) USING 'cat' AS (`key` string, `value` string) FROM `default`.`parquet_t2`\n```\n#### How was this patch tested?\n\nSeven test cases are added to `LogicalPlanToSQLSuite`.\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\nAuthor: xiaoli <lixiao1983@gmail.com>\nAuthor: Xiao Li <xiaoli@Xiaos-MacBook-Pro.local>\n\nCloses #11503 from gatorsmile/transformToSQL.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/c4bd57602c0b14188d364bb475631bf473d25082"},{"sha":"ae6c677c8a03174787be99af6238a5e1fbe4e389","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-13038][PYSPARK] Add load/save to pipeline\n\n## What changes were proposed in this pull request?\n\nJIRA issue: https://issues.apache.org/jira/browse/SPARK-13038\n\n1. Add load/save to PySpark Pipeline and PipelineModel\n\n2. Add `_transfer_stage_to_java()` and `_transfer_stage_from_java()` for `JavaWrapper`.\n\n## How was this patch tested?\n\nTest with doctest.\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #11683 from yinxusen/SPARK-13038-only.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/ae6c677c8a03174787be99af6238a5e1fbe4e389"},{"sha":"3f06eb72ca0c3e5779a702c7c677229e0c480751","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-13613][ML] Provide ignored tests to export test dataset into CSV format\n\n## What changes were proposed in this pull request?\nProvide ignored test cases to export the test dataset into CSV format in ```LinearRegressionSuite```, ```LogisticRegressionSuite```, ```AFTSurvivalRegressionSuite``` and ```GeneralizedLinearRegressionSuite```, so users can validate the training accuracy compared with R's glm, glmnet and survival package.\ncc mengxr\n## How was this patch tested?\nThe test suite is ignored, but I have enabled all these cases offline and it works as expected.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #11463 from yanboliang/spark-13613.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/3f06eb72ca0c3e5779a702c7c677229e0c480751"},{"sha":"6fc2b6541fd5ab73b289af5f7296fc602b5b4dce","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-11888][ML] Decision tree persistence in spark.ml\n\n### What changes were proposed in this pull request?\n\nMade these MLReadable and MLWritable: DecisionTreeClassifier, DecisionTreeClassificationModel, DecisionTreeRegressor, DecisionTreeRegressionModel\n* The shared implementation is in treeModels.scala\n* I use case classes to create a DataFrame to save, and I use the Dataset API to parse loaded files.\n\nOther changes:\n* Made CategoricalSplit.numCategories public (to use in persistence)\n* Fixed a bug in DefaultReadWriteTest.testEstimatorAndModelReadWrite, where it did not call the checkModelData function passed as an argument.  This caused an error in LDASuite, which I fixed.\n\n### How was this patch tested?\n\nPersistence is tested via unit tests.  For each algorithm, there are 2 non-trivial trees (depth 2).  One is built with continuous features, and one with categorical; this ensures that both types of splits are tested.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #11581 from jkbradley/dt-io.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/6fc2b6541fd5ab73b289af5f7296fc602b5b4dce"},{"sha":"85c42fda99973a0c35c743816a06ce9117bb1aad","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"[SPARK-13927][MLLIB] add row/column iterator to local matrices\n\n## What changes were proposed in this pull request?\n\nAdd row/column iterator to local matrices to simplify tasks like BlockMatrix => RowMatrix conversion. It handles dense and sparse matrices properly.\n\n## How was this patch tested?\n\nUnit tests on sparse and dense matrix.\n\ncc: dbtsai\n\nAuthor: Xiangrui Meng <meng@databricks.com>\n\nCloses #11757 from mengxr/SPARK-13927.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/85c42fda99973a0c35c743816a06ce9117bb1aad"},{"sha":"27e1f38851a8f28a28544b2021b3c5641d0ff3ab","author":{"email":"aa695cee2e341784407f771fa5b94d855e1c7dec@gmail.com","name":"GayathriMurali"},"message":"[SPARK-13034] PySpark ml.classification support export/import\n\n## What changes were proposed in this pull request?\n\nAdd export/import for all estimators and transformers(which have Scala implementation) under pyspark/ml/classification.py.\n\n## How was this patch tested?\n\n./python/run-tests\n./dev/lint-python\nUnit tests added to check persistence in Logistic Regression\n\nAuthor: GayathriMurali <gayathri.m.softie@gmail.com>\n\nCloses #11707 from GayathriMurali/SPARK-13034.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/27e1f38851a8f28a28544b2021b3c5641d0ff3ab"},{"sha":"4ce2d24e2a03966fde9a5be2d11395200f5dc4f6","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-13942][CORE][DOCS] Remove Shark-related docs for 2.x\n\n## What changes were proposed in this pull request?\n\n`Shark` was merged into `Spark SQL` since [July 2014](https://databricks.com/blog/2014/07/01/shark-spark-sql-hive-on-spark-and-the-future-of-sql-on-spark.html). The followings seem to be the only legacy. For Spark 2.x, we had better clean up those docs.\n\n**Migration Guide**\n```\n- ## Migration Guide for Shark Users\n- ...\n- ### Scheduling\n- ...\n- ### Reducer number\n- ...\n- ### Caching\n```\n\n## How was this patch tested?\n\nPass the Jenkins test.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #11770 from dongjoon-hyun/SPARK-13942.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/4ce2d24e2a03966fde9a5be2d11395200f5dc4f6"},{"sha":"b90c0206faeb0883fba1c79fe18aa72affb9988e","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-13922][SQL] Filter rows with null attributes in vectorized parquet reader\n\n# What changes were proposed in this pull request?\n\nIt's common for many SQL operators to not care about reading `null` values for correctness. Currently, this is achieved by performing `isNotNull` checks (for all relevant columns) on a per-row basis. Pushing these null filters in the vectorized parquet reader should bring considerable benefits (especially for cases when the underlying data doesn't contain any nulls or contains all nulls).\n\n## How was this patch tested?\n\n        Intel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n        String with Nulls Scan (0%):        Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n        -------------------------------------------------------------------------------------------\n        SQL Parquet Vectorized                   1229 / 1648          8.5         117.2       1.0X\n        PR Vectorized                             833 /  846         12.6          79.4       1.5X\n        PR Vectorized (Null Filtering)            732 /  782         14.3          69.8       1.7X\n\n        Intel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n        String with Nulls Scan (50%):       Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n        -------------------------------------------------------------------------------------------\n        SQL Parquet Vectorized                    995 / 1053         10.5          94.9       1.0X\n        PR Vectorized                             732 /  772         14.3          69.8       1.4X\n        PR Vectorized (Null Filtering)            725 /  790         14.5          69.1       1.4X\n\n        Intel(R) Core(TM) i7-4960HQ CPU  2.60GHz\n        String with Nulls Scan (95%):       Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n        -------------------------------------------------------------------------------------------\n        SQL Parquet Vectorized                    326 /  333         32.2          31.1       1.0X\n        PR Vectorized                             190 /  200         55.1          18.2       1.7X\n        PR Vectorized (Null Filtering)            168 /  172         62.2          16.1       1.9X\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #11749 from sameeragarwal/perf-testing.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/b90c0206faeb0883fba1c79fe18aa72affb9988e"},{"sha":"f96997ba244a14c26e85a2475415a762d0c0d0a8","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-13871][SQL] Support for inferring filters from data constraints\n\n## What changes were proposed in this pull request?\n\nThis PR generalizes the `NullFiltering` optimizer rule in catalyst to `InferFiltersFromConstraints` that can automatically infer all relevant filters based on an operator's constraints while making sure of 2 things:\n\n(a) no redundant filters are generated, and\n(b) filters that do not contribute to any further optimizations are not generated.\n\n## How was this patch tested?\n\nExtended all tests in `InferFiltersFromConstraintsSuite` (that were initially based on `NullFilteringSuite` to test filter inference in `Filter` and `Join` operators.\n\nIn particular the 2 tests ( `single inner join with pre-existing filters: filter out values on either side` and `multiple inner joins: filter out values on all sides on equi-join keys` attempts to highlight/test the real potential of this rule for join optimization.\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #11665 from sameeragarwal/infer-filters.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/f96997ba244a14c26e85a2475415a762d0c0d0a8"},{"sha":"77ba3021c12dc63cb7d831f964f901e0474acd96","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"[SPARK-13869][SQL] Remove redundant conditions while combining filters\n\n## What changes were proposed in this pull request?\n\n**[I'll link it to the JIRA once ASF JIRA is back online]**\n\nThis PR modifies the existing `CombineFilters` rule to remove redundant conditions while combining individual filter predicates. For instance, queries of the form `table.where('a === 1 && 'b === 1).where('a === 1 && 'c === 1)` will now be optimized to ` table.where('a === 1 && 'b === 1 && 'c === 1)` (instead of ` table.where('a === 1 && 'a === 1 && 'b === 1 && 'c === 1)`)\n\n## How was this patch tested?\n\nUnit test in `FilterPushdownSuite`\n\nAuthor: Sameer Agarwal <sameer@databricks.com>\n\nCloses #11670 from sameeragarwal/combine-filters.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/77ba3021c12dc63cb7d831f964f901e0474acd96"},{"sha":"d4d84936fb82bee91f4b04608de9f75c293ccc9e","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-11011][SQL] Narrow type of UDT serialization\n\n## What changes were proposed in this pull request?\n\nNarrow down the parameter type of `UserDefinedType#serialize()`. Currently, the parameter type is `Any`, however it would logically make more sense to narrow it down to the type of the actual user defined type.\n\n## How was this patch tested?\n\nExisting tests were successfully run on local machine.\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #11379 from jodersky/SPARK-11011-udt-types.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/d4d84936fb82bee91f4b04608de9f75c293ccc9e"},{"sha":"92b70576eabf8ff94ac476e2b3c66f8b3d28e79e","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-13761][ML] Deprecate validateParams\n\n## What changes were proposed in this pull request?\n\nDeprecate validateParams() method here: https://github.com/apache/spark/blob/035d3acdf3c1be5b309a861d5c5beb803b946b5e/mllib/src/main/scala/org/apache/spark/ml/param/params.scala#L553\nMove all functionality in overridden methods to transformSchema().\nCheck docs to make sure they indicate complex Param interaction checks should be done in transformSchema.\n\n## How was this patch tested?\n\nunit tests\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #11620 from hhbyyh/depreValid.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/92b70576eabf8ff94ac476e2b3c66f8b3d28e79e"},{"sha":"ca9ef86c84ee84263f437a979017898f4bed0feb","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-13923][SQL] Implement SessionCatalog\n\n## What changes were proposed in this pull request?\n\nAs part of the effort to merge `SQLContext` and `HiveContext`, this patch implements an internal catalog called `SessionCatalog` that handles temporary functions and tables and delegates metastore operations to `ExternalCatalog`. Currently, this is still dead code, but in the future it will be part of `SessionState` and will replace `o.a.s.sql.catalyst.analysis.Catalog`.\n\nA recent patch #11573 parses Hive commands ourselves in Spark, but still passes the entire query text to Hive. In a future patch, we will use `SessionCatalog` to implement the parsed commands.\n\n## How was this patch tested?\n\n800+ lines of tests in `SessionCatalogSuite`.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #11750 from andrewor14/temp-catalog.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/ca9ef86c84ee84263f437a979017898f4bed0feb"},{"sha":"917f4000b418ee0997551d4dfabb369c4e9243a9","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-13719][SQL] Parse JSON rows having an array type and a struct type in the same fieild\n\n## What changes were proposed in this pull request?\n\nThis https://github.com/apache/spark/pull/2400 added the support to parse JSON rows wrapped with an array. However, this throws an exception when the given data contains array data and struct data in the same field as below:\n\n```json\n{\"a\": {\"b\": 1}}\n{\"a\": []}\n```\n\nand the schema is given as below:\n\n```scala\nval schema =\n  StructType(\n    StructField(\"a\", StructType(\n      StructField(\"b\", StringType) :: Nil\n    )) :: Nil)\n```\n\n- **Before**\n\n```scala\nsqlContext.read.schema(schema).json(path).show()\n```\n\n```scala\nException in thread \"main\" org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 0.0 failed 4 times, most recent failure: Lost task 7.3 in stage 0.0 (TID 10, 192.168.1.170): java.lang.ClassCastException: org.apache.spark.sql.types.GenericArrayData cannot be cast to org.apache.spark.sql.catalyst.InternalRow\n\tat org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getStruct(rows.scala:50)\n\tat org.apache.spark.sql.catalyst.expressions.GenericMutableRow.getStruct(rows.scala:247)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown Source)\n...\n```\n\n- **After**\n\n```scala\nsqlContext.read.schema(schema).json(path).show()\n```\n\n```bash\n+----+\n|   a|\n+----+\n| [1]|\n|null|\n+----+\n```\n\nFor other data types, in this case it converts the given values are `null` but only this case emits an exception.\n\nThis PR makes the support for wrapped rows applied only at the top level.\n\n## How was this patch tested?\n\nUnit tests were used and `./dev/run_tests` for code style tests.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #11752 from HyukjinKwon/SPARK-3308-follow-up.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/917f4000b418ee0997551d4dfabb369c4e9243a9"},{"sha":"c100d31ddc6db9c03b7a65a20a7dd56dcdc18baf","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13873] [SQL] Avoid copy of UnsafeRow when there is no join in whole stage codegen\n\n## What changes were proposed in this pull request?\n\nWe need to copy the UnsafeRow since a Join could produce multiple rows from single input rows. We could avoid that if there is no join (or the join will not produce multiple rows) inside WholeStageCodegen.\n\nUpdated the benchmark for `collect`, we could see 20-30% speedup.\n\n## How was this patch tested?\n\nexisting unit tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11740 from davies/avoid_copy2.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/c100d31ddc6db9c03b7a65a20a7dd56dcdc18baf"},{"sha":"7eef2463ade693f8e87ecd913a5adf03b69ac14e","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-13118][SQL] Expression encoding for optional synthetic classes\n\n## What changes were proposed in this pull request?\n\nFix expression generation for optional types.\nStandard Java reflection causes issues when dealing with synthetic Scala objects (things that do not map to Java and thus contain a dollar sign in their name). This patch introduces Scala reflection in such cases.\n\nThis patch also adds a regression test for Dataset's handling of classes defined in package objects (which was the initial purpose of this PR).\n\n## How was this patch tested?\nA new test in ExpressionEncoderSuite that tests optional inner classes and a regression test for Dataset's handling of package objects.\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #11708 from jodersky/SPARK-13118-package-objects.","distinct":true,"url":"https://api.github.com/repos/prabhu1984/spark/commits/7eef2463ade693f8e87ecd913a5adf03b69ac14e"}]}