{"push_id":1049224037,"size":13,"distinct_size":13,"ref":"refs/heads/function","head":"ae359fb60dcbb34514487553ac3c7fb24de3d4c0","before":"3718e613498cbf9a996f52bc3f215f1051f2ae51","commits":[{"sha":"a3e293542a6e7df9bcc7d9bbd22b3c93a81bcc38","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Disable StateStoreSuite.maintenance","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/a3e293542a6e7df9bcc7d9bbd22b3c93a81bcc38"},{"sha":"f705037617d55bb479ec60bcb1e55c736224be94","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14338][SQL] Improve `SimplifyConditionals` rule to handle `null` in IF/CASEWHEN\n\n## What changes were proposed in this pull request?\n\nCurrently, `SimplifyConditionals` handles `true` and `false` to optimize branches. This PR improves `SimplifyConditionals` to take advantage of `null` conditions for `if` and `CaseWhen` expressions, too.\n\n**Before**\n```\nscala> sql(\"SELECT IF(null, 1, 0)\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [if (null) 1 else 0 AS (IF(CAST(NULL AS BOOLEAN), 1, 0))#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\nscala> sql(\"select case when cast(null as boolean) then 1 else 2 end\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [CASE WHEN null THEN 1 ELSE 2 END AS CASE WHEN CAST(NULL AS BOOLEAN) THEN 1 ELSE 2 END#14]\n:     +- INPUT\n+- Scan OneRowRelation[]\n```\n\n**After**\n```\nscala> sql(\"SELECT IF(null, 1, 0)\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [0 AS (IF(CAST(NULL AS BOOLEAN), 1, 0))#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\nscala> sql(\"select case when cast(null as boolean) then 1 else 2 end\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [2 AS CASE WHEN CAST(NULL AS BOOLEAN) THEN 1 ELSE 2 END#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\n```\n\n**Hive**\n```\nhive> select if(null,1,2);\nOK\n2\nhive> select case when cast(null as boolean) then 1 else 2 end;\nOK\n2\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new extended test cases).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12122 from dongjoon-hyun/SPARK-14338.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/f705037617d55bb479ec60bcb1e55c736224be94"},{"sha":"4a6e78abd9d5edc4a5092738dff0006bbe202a89","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[MINOR][DOCS] Use multi-line JavaDoc comments in Scala code.\n\n## What changes were proposed in this pull request?\n\nThis PR aims to fix all Scala-Style multiline comments into Java-Style multiline comments in Scala codes.\n(All comment-only changes over 77 files: +786 lines, −747 lines)\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12130 from dongjoon-hyun/use_multiine_javadoc_comments.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/4a6e78abd9d5edc4a5092738dff0006bbe202a89"},{"sha":"03d130f9734be66e8aefc4ffaa207ee13e837629","author":{"email":"a6ba1154cbf361d4882390aa08f5a86dcc774822@gmail.com","name":"Liwei Lin"},"message":"[SPARK-14342][CORE][DOCS][TESTS] Remove straggler references to Tachyon\n\n## What changes were proposed in this pull request?\n\nStraggler references to Tachyon were removed:\n- for docs, `tachyon` has been generalized as `off-heap memory`;\n- for Mesos test suits, the key-value `tachyon:true`/`tachyon:false` has been changed to `os:centos`/`os:ubuntu`, since `os` is an example constrain used by the [Mesos official docs](http://mesos.apache.org/documentation/attributes-resources/).\n\n## How was this patch tested?\n\nExisting test suites.\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #12129 from lw-lin/tachyon-cleanup.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/03d130f9734be66e8aefc4ffaa207ee13e837629"},{"sha":"1cf70183423b938ec064925b20fd4a5b9e355991","author":{"email":"651d9e2ab320d6d21a117141758c2477389f223d@fb.com","name":"Sital Kedia"},"message":"[SPARK-14056] Appends s3 specific configurations and spark.hadoop con…\n\n## What changes were proposed in this pull request?\n\nAppends s3 specific configurations and spark.hadoop configurations to hive configuration.\n\n## How was this patch tested?\n\nTested by running a job on cluster.\n\n…figurations to hive configuration.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #11876 from sitalkedia/hiveConf.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/1cf70183423b938ec064925b20fd4a5b9e355991"},{"sha":"c2f25b1a148eeb1791ea7018b14b3a665c13212a","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13996] [SQL] Add more not null attributes for Filter codegen\n\n## What changes were proposed in this pull request?\nJIRA: https://issues.apache.org/jira/browse/SPARK-13996\n\nFilter codegen finds the attributes not null by checking IsNotNull(a) expression with a condition if child.output.contains(a). However, the current approach to checking it is not comprehensive. We can improve it.\n\nE.g., for this plan:\n\n    val rdd = sqlContext.sparkContext.makeRDD(Seq(Row(1, \"1\"), Row(null, \"1\"), Row(2, \"2\")))\n    val schema = new StructType().add(\"k\", IntegerType).add(\"v\", StringType)\n    val smallDF = sqlContext.createDataFrame(rdd, schema)\n    val df = smallDF.filter(\"isnotnull(k + 1)\")\n\nThe code snippet generated without this patch:\n\n    /* 031 */   protected void processNext() throws java.io.IOException {\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\n    /* 033 */\n    /* 034 */     /*** PRODUCE: INPUT */\n    /* 035 */\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\n    /* 039 */       /* input[0, int] */\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\n    /* 042 */\n    /* 043 */       /* isnotnull((input[0, int] + 1)) */\n    /* 044 */       /* (input[0, int] + 1) */\n    /* 045 */       boolean filter_isNull3 = true;\n    /* 046 */       int filter_value3 = -1;\n    /* 047 */\n    /* 048 */       if (!filter_isNull) {\n    /* 049 */         filter_isNull3 = false; // resultCode could change nullability.\n    /* 050 */         filter_value3 = filter_value + 1;\n    /* 051 */\n    /* 052 */       }\n    /* 053 */       if (!(!(filter_isNull3))) continue;\n    /* 054 */\n    /* 055 */       filter_metricValue.add(1);\n\nWith this patch:\n\n    /* 031 */   protected void processNext() throws java.io.IOException {\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\n    /* 033 */\n    /* 034 */     /*** PRODUCE: INPUT */\n    /* 035 */\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\n    /* 039 */       /* input[0, int] */\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\n    /* 042 */\n    /* 043 */       if (filter_isNull) continue;\n    /* 044 */\n    /* 045 */       filter_metricValue.add(1);\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #11810 from viirya/add-more-not-null-attrs.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/c2f25b1a148eeb1791ea7018b14b3a665c13212a"},{"sha":"7be46205083fc688249ee619ac7758904f7aa55d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Fix Scala 2.10 compilation","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/7be46205083fc688249ee619ac7758904f7aa55d"},{"sha":"2262a93358c2f6d4cfb73645c4ebc963c5640ec8","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-14231] [SQL] JSON data source infers floating-point values as a double when they do not fit in a decimal\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-14231\n\nCurrently, JSON data source supports to infer `DecimalType` for big numbers and `floatAsBigDecimal` option which reads floating-point values as `DecimalType`.\n\nBut there are few restrictions in Spark `DecimalType` below:\n\n1. The precision cannot be bigger than 38.\n2. scale cannot be bigger than precision.\n\nCurrently, both restrictions are not being handled.\n\nThis PR handles the cases by inferring them as `DoubleType`. Also, the option name was changed from `floatAsBigDecimal` to `prefersDecimal` as suggested [here](https://issues.apache.org/jira/browse/SPARK-14231?focusedCommentId=15215579&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15215579).\n\nSo, the codes below:\n\n```scala\ndef doubleRecords: RDD[String] =\n  sqlContext.sparkContext.parallelize(\n    s\"\"\"{\"a\": 1${\"0\" * 38}, \"b\": 0.01}\"\"\" ::\n    s\"\"\"{\"a\": 2${\"0\" * 38}, \"b\": 0.02}\"\"\" :: Nil)\n\nval jsonDF = sqlContext.read\n  .option(\"prefersDecimal\", \"true\")\n  .json(doubleRecords)\njsonDF.printSchema()\n```\n\nproduces below:\n\n- **Before**\n\n```scala\norg.apache.spark.sql.AnalysisException: Decimal scale (2) cannot be greater than precision (1).;\n\tat org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:44)\n\tat org.apache.spark.sql.execution.datasources.json.InferSchema$.org$apache$spark$sql$execution$datasources$json$InferSchema$$inferField(InferSchema.scala:144)\n\tat org.apache.spark.sql.execution.datasources.json.InferSchema$.org$apache$spark$sql$execution$datasources$json$InferSchema$$inferField(InferSchema.scala:108)\n\tat\n...\n```\n\n- **After**\n\n```scala\nroot\n |-- a: double (nullable = true)\n |-- b: double (nullable = true)\n```\n\n## How was this patch tested?\n\nUnit tests were used and `./dev/run_tests` for coding style tests.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #12030 from HyukjinKwon/SPARK-14231.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/2262a93358c2f6d4cfb73645c4ebc963c5640ec8"},{"sha":"1f0c5dcebba1f9d1149043a496e0175f78252bae","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14350][SQL] EXPLAIN output should be in a single cell\n\n## What changes were proposed in this pull request?\n\nEXPLAIN output should be in a single cell.\n\n**Before**\n```\nscala> sql(\"explain select 1\").collect()\nres0: Array[org.apache.spark.sql.Row] = Array([== Physical Plan ==], [WholeStageCodegen], [:  +- Project [1 AS 1#1]], [:     +- INPUT], [+- Scan OneRowRelation[]])\n```\n\n**After**\n```\nscala> sql(\"explain select 1\").collect()\nres1: Array[org.apache.spark.sql.Row] =\nArray([== Physical Plan ==\nWholeStageCodegen\n:  +- Project [1 AS 1#4]\n:     +- INPUT\n+- Scan OneRowRelation[]])\n```\nOr,\n```\nscala> sql(\"explain select 1\").head\nres1: org.apache.spark.sql.Row =\n[== Physical Plan ==\nWholeStageCodegen\n:  +- Project [1 AS 1#5]\n:     +- INPUT\n+- Scan OneRowRelation[]]\n```\n\nPlease note that `Spark-shell(Scala-shell)` trims long string output. So, you may need to use `println` to get full strings.\n```\nscala> println(sql(\"explain codegen select 'a' as a group by 1\").head)\n[Found 2 WholeStageCodegen subtrees.\n== Subtree 1 / 2 ==\nWholeStageCodegen\n...\n/* 059 */   }\n/* 060 */ }\n\n]\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests. (Testcases are updated.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12137 from dongjoon-hyun/SPARK-14350.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/1f0c5dcebba1f9d1149043a496e0175f78252bae"},{"sha":"c238cd07448f94bbceb661daad90b6a6d597a846","author":{"email":"1a1ab5abfb65135a4f49eb66e79d9e570e28807d@us.ibm.com","name":"bomeng"},"message":"[SPARK-14341][SQL] Throw exception on unsupported create / drop macro ddl\n\n## What changes were proposed in this pull request?\n\nWe throw an AnalysisException that looks like this:\n\n```\nscala> sqlContext.sql(\"CREATE TEMPORARY MACRO SIGMOID (x DOUBLE) 1.0 / (1.0 + EXP(-x))\")\norg.apache.spark.sql.catalyst.parser.ParseException:\nUnsupported SQL statement\n== SQL ==\nCREATE TEMPORARY MACRO SIGMOID (x DOUBLE) 1.0 / (1.0 + EXP(-x))\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.nativeCommand(ParseDriver.scala:66)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:56)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:86)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)\n  at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:198)\n  at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:749)\n  ... 48 elided\n\n```\n\n## How was this patch tested?\n\nAdd test cases in HiveQuerySuite.scala\n\nAuthor: bomeng <bmeng@us.ibm.com>\n\nCloses #12125 from bomeng/SPARK-14341.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/c238cd07448f94bbceb661daad90b6a6d597a846"},{"sha":"ac6dfc5c94a07128b64003852f74105fbb37bd83","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"Remove withGenerator.","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/ac6dfc5c94a07128b64003852f74105fbb37bd83"},{"sha":"64e2ee3eabb514b80941a4db886633f857d4355b","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"Cleanup","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/64e2ee3eabb514b80941a4db886633f857d4355b"},{"sha":"ae359fb60dcbb34514487553ac3c7fb24de3d4c0","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"Merge remote-tracking branch 'upstream/master' into function","distinct":true,"url":"https://api.github.com/repos/yhuai/spark/commits/ae359fb60dcbb34514487553ac3c7fb24de3d4c0"}]}