{"push_id":1219576634,"size":24,"distinct_size":24,"ref":"refs/heads/master","head":"8a8d26f1e27db5c2228307b1c3609b4713b9d0db","before":"1221ce04029154778ccb5453e348f6d116092cc5","commits":[{"sha":"daace6014216b996bcc8937f1fdcea732b6910ca","author":{"name":"Brian Cho","email":"868169ee102579a79a0df470bdf82d6b6b4fb664@fb.com"},"message":"[SPARK-5581][CORE] When writing sorted map output file, avoid open / …\n\n…close between each partition\n\n## What changes were proposed in this pull request?\n\nReplace commitAndClose with separate commit and close to avoid opening and closing\nthe file between partitions.\n\n## How was this patch tested?\n\nRun existing unit tests, add a few unit tests regarding reverts.\n\nObserved a ~20% reduction in total time in tasks on stages with shuffle\nwrites to many partitions.\n\nJoshRosen\n\nAuthor: Brian Cho <bcho@fb.com>\n\nCloses #13382 from dafrista/separatecommit-master.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/daace6014216b996bcc8937f1fdcea732b6910ca"},{"sha":"468a3c3ac5d039f21613f9237c7bdef9b92f5fea","author":{"name":"Qifan Pu","email":"4624f8175b6bf095c6fd5ae8d365ad06117abe3f@gmail.com"},"message":"[SPARK-16699][SQL] Fix performance bug in hash aggregate on long string keys\n\nIn the following code in `VectorizedHashMapGenerator.scala`:\n```\n    def hashBytes(b: String): String = {\n      val hash = ctx.freshName(\"hash\")\n      s\"\"\"\n         |int $result = 0;\n         |for (int i = 0; i < $b.length; i++) {\n         |  ${genComputeHash(ctx, s\"$b[i]\", ByteType, hash)}\n         |  $result = ($result ^ (0x9e3779b9)) + $hash + ($result << 6) + ($result >>> 2);\n         |}\n       \"\"\".stripMargin\n    }\n\n```\nwhen b=input.getBytes(), the current 2.0 code results in getBytes() being called n times, n being length of input. getBytes() involves memory copy is thus expensive and causes a performance degradation.\nFix is to evaluate getBytes() before the for loop.\n\nPerformance bug, no additional test added.\n\nAuthor: Qifan Pu <qifan.pu@gmail.com>\n\nCloses #14337 from ooq/SPARK-16699.\n\n(cherry picked from commit d226dce12babcd9f30db033417b2b9ce79f44312)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/468a3c3ac5d039f21613f9237c7bdef9b92f5fea"},{"sha":"68b4020d0c0d4f063facfbf4639ef4251dcfda8b","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-16648][SQL] Make ignoreNullsExpr a child expression of First and Last\n\n## What changes were proposed in this pull request?\n\nDefault `TreeNode.withNewChildren` implementation doesn't work for `Last` and when both constructor arguments are the same, e.g.:\n\n```sql\nLAST_VALUE(FALSE) -- The 2nd argument defaults to FALSE\nLAST_VALUE(FALSE, FALSE)\nLAST_VALUE(TRUE, TRUE)\n```\n\nThis is because although `Last` is a unary expression, both of its constructor arguments, `child` and `ignoreNullsExpr`, are `Expression`s. When they have the same value, `TreeNode.withNewChildren` treats both of them as child nodes by mistake. `First` is also affected by this issue in exactly the same way.\n\nThis PR fixes this issue by making `ignoreNullsExpr` a child expression of `First` and `Last`.\n\n## How was this patch tested?\n\nNew test case added in `WindowQuerySuite`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #14295 from liancheng/spark-16648-last-value.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/68b4020d0c0d4f063facfbf4639ef4251dcfda8b"},{"sha":"7ffd99ec5f267730734431097cbb700ad074bebe","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16674][SQL] Avoid per-record type dispatch in JDBC when reading\n\n## What changes were proposed in this pull request?\n\nCurrently, `JDBCRDD.compute` is doing type dispatch for each row to read appropriate values.\nIt might not have to be done like this because the schema is already kept in `JDBCRDD`.\n\nSo, appropriate converters can be created first according to the schema, and then apply them to each row.\n\n## How was this patch tested?\n\nExisting tests should cover this.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14313 from HyukjinKwon/SPARK-16674.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/7ffd99ec5f267730734431097cbb700ad074bebe"},{"sha":"d27d362ebae0c4a5cc6c99f13ef20049214dd4f9","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16660][SQL] CreateViewCommand should not take CatalogTable\n\n## What changes were proposed in this pull request?\n\n`CreateViewCommand` only needs some information of a `CatalogTable`, but not all of them. We have some tricks(e.g. we need to check the table type is `VIEW`, we need to make `CatalogColumn.dataType` nullable) to allow it to take a `CatalogTable`.\nThis PR cleans it up and only pass in necessary information to `CreateViewCommand`.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14297 from cloud-fan/minor2.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/d27d362ebae0c4a5cc6c99f13ef20049214dd4f9"},{"sha":"64529b186a1c33740067cc7639d630bc5b9ae6e8","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16691][SQL] move BucketSpec to catalyst module and use it in CatalogTable\n\n## What changes were proposed in this pull request?\n\nIt's weird that we have `BucketSpec` to abstract bucket info, but don't use it in `CatalogTable`. This PR moves `BucketSpec` into catalyst module.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14331 from cloud-fan/check.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/64529b186a1c33740067cc7639d630bc5b9ae6e8"},{"sha":"d6a52176ade92853f37167ad27631977dc79bc76","author":{"name":"Sameer Agarwal","email":"a70383dff9f522db5c95b6d04d9bfe45099f5e1d@cs.berkeley.edu"},"message":"[SPARK-16668][TEST] Test parquet reader for row groups containing both dictionary and plain encoded pages\n\n## What changes were proposed in this pull request?\n\nThis patch adds an explicit test for [SPARK-14217] by setting the parquet dictionary and page size the generated parquet file spans across 3 pages (within a single row group) where the first page is dictionary encoded and the remaining two are plain encoded.\n\n## How was this patch tested?\n\n1. ParquetEncodingSuite\n2. Also manually tested that this test fails without https://github.com/apache/spark/pull/12279\n\nAuthor: Sameer Agarwal <sameerag@cs.berkeley.edu>\n\nCloses #14304 from sameeragarwal/hybrid-encoding-test.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/d6a52176ade92853f37167ad27631977dc79bc76"},{"sha":"79826f3c7936ee27457d030c7115d5cac69befd7","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16698][SQL] Field names having dots should be allowed for datasources based on FileFormat\n\n## What changes were proposed in this pull request?\n\nIt seems this is a regression assuming from https://issues.apache.org/jira/browse/SPARK-16698.\n\nField name having dots throws an exception. For example the codes below:\n\n```scala\nval path = \"/tmp/path\"\nval json =\"\"\" {\"a.b\":\"data\"}\"\"\"\nspark.sparkContext\n  .parallelize(json :: Nil)\n  .saveAsTextFile(path)\nspark.read.json(path).collect()\n```\n\nthrows an exception as below:\n\n```\nUnable to resolve a.b given [a.b];\norg.apache.spark.sql.AnalysisException: Unable to resolve a.b given [a.b];\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1$$anonfun$apply$5.apply(LogicalPlan.scala:134)\n\tat org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1$$anonfun$apply$5.apply(LogicalPlan.scala:134)\n\tat scala.Option.getOrElse(Option.scala:121)\n```\n\nThis problem was introduced in https://github.com/apache/spark/commit/17eec0a71ba8713c559d641e3f43a1be726b037c#diff-27c76f96a7b2733ecfd6f46a1716e153R121\n\nWhen extracting the data columns, it does not count that it can contains dots in field names. Actually, it seems the fields name are not expected as quoted when defining schema. So, It not have to consider whether this is wrapped with quotes because the actual schema (inferred or user-given schema) would not have the quotes for fields.\n\nFor example, this throws an exception. (**Loading JSON from RDD is fine**)\n\n```scala\nval json =\"\"\" {\"a.b\":\"data\"}\"\"\"\nval rdd = spark.sparkContext.parallelize(json :: Nil)\nspark.read.schema(StructType(Seq(StructField(\"`a.b`\", StringType, true))))\n  .json(rdd).select(\"`a.b`\").printSchema()\n```\n\nas below:\n\n```\ncannot resolve '```a.b```' given input columns: [`a.b`];\norg.apache.spark.sql.AnalysisException: cannot resolve '```a.b```' given input columns: [`a.b`];\n\tat org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)\n```\n\n## How was this patch tested?\n\nUnit tests in `FileSourceStrategySuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14339 from HyukjinKwon/SPARK-16698-regression.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/79826f3c7936ee27457d030c7115d5cac69befd7"},{"sha":"7ea6d282b925819ddb3874a67b3c9da8cc41f131","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-16703][SQL] Remove extra whitespace in SQL generation for window functions\n\n## What changes were proposed in this pull request?\n\nThis PR fixes a minor formatting issue of `WindowSpecDefinition.sql` when no partitioning expressions are present.\n\nBefore:\n\n```sql\n( ORDER BY `a` ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)\n```\n\nAfter:\n\n```sql\n(ORDER BY `a` ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)\n```\n\n## How was this patch tested?\n\nNew test case added in `ExpressionSQLBuilderSuite`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #14334 from liancheng/window-spec-sql-format.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/7ea6d282b925819ddb3874a67b3c9da8cc41f131"},{"sha":"b73defdd790cb823a4f9958ca89cec06fd198051","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR][DOCS] fix broken url in doc\n\n## What changes were proposed in this pull request?\n\nFix broken url, also,\n\nsparkR.session.stop doc page should have it in the header, instead of saying \"sparkR.stop\"\n![image](https://cloud.githubusercontent.com/assets/8969467/17080129/26d41308-50d9-11e6-8967-79d6c920313f.png)\n\nData type section is in the middle of a list of gapply/gapplyCollect subsections:\n![image](https://cloud.githubusercontent.com/assets/8969467/17080122/f992d00a-50d8-11e6-8f2c-fd5786213920.png)\n\n## How was this patch tested?\n\nmanual test\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #14329 from felixcheung/rdoclinkfix.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/b73defdd790cb823a4f9958ca89cec06fd198051"},{"sha":"ad3708e78377d631e3d586548c961f4748322bf0","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16653][ML][OPTIMIZER] update ANN convergence tolerance param default to 1e-6\n\n## What changes were proposed in this pull request?\n\nreplace ANN convergence tolerance param default\nfrom 1e-4 to 1e-6\n\nso that it will be the same with other algorithms in MLLib which use LBFGS as optimizer.\n\n## How was this patch tested?\n\nExisting Test.\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14286 from WeichenXu123/update_ann_tol.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/ad3708e78377d631e3d586548c961f4748322bf0"},{"sha":"dd784a8822497ad0631208d56325c4d74ab9e036","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16685] Remove audit-release scripts.\n\n## What changes were proposed in this pull request?\nThis patch removes dev/audit-release. It was initially created to do basic release auditing. They have been unused by for the last one year+.\n\n## How was this patch tested?\nN/A\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14342 from rxin/SPARK-16685.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/dd784a8822497ad0631208d56325c4d74ab9e036"},{"sha":"978cd5f125eb5a410bad2e60bf8385b11cf1b978","author":{"name":"Philipp Hoffmann","email":"3b8e10480754ba93523db5dc4626a5f001e54bb7@philipphoffmann.de"},"message":"[SPARK-15271][MESOS] Allow force pulling executor docker images\n\n## What changes were proposed in this pull request?\n\nMesos agents by default will not pull docker images which are cached\nlocally already. In order to run Spark executors from mutable tags like\n`:latest` this commit introduces a Spark setting\n`spark.mesos.executor.docker.forcePullImage`. Setting this flag to\ntrue will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous\nimplementation and Mesos' default\nbehaviour).\n\n## How was this patch tested?\n\nI ran a sample application including this change on a Mesos cluster and verified the correct behaviour for both, with and without, force pulling the executor image. As expected the image is being force pulled if the flag is set.\n\nAuthor: Philipp Hoffmann <mail@philipphoffmann.de>\n\nCloses #13051 from philipphoffmann/force-pull-image.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/978cd5f125eb5a410bad2e60bf8385b11cf1b978"},{"sha":"3b6e1d094e153599e158331b10d33d74a667be5a","author":{"name":"Shuai Lin","email":"7a96e9fba54b1a616c0c679715dfd11c0089fe85@gmail.com"},"message":"[SPARK-16485][DOC][ML] Fixed several inline formatting in ml features doc\n\n## What changes were proposed in this pull request?\n\nFixed several inline formatting in ml features doc.\n\nBefore:\n\n<img width=\"475\" alt=\"screen shot 2016-07-14 at 12 24 57 pm\" src=\"https://cloud.githubusercontent.com/assets/717363/16827974/1e1b6e04-49be-11e6-8aa9-4a0cb6cd3b4e.png\">\n\nAfter:\n\n<img width=\"404\" alt=\"screen shot 2016-07-14 at 12 25 48 pm\" src=\"https://cloud.githubusercontent.com/assets/717363/16827976/2576510a-49be-11e6-96dd-92a1fa464d36.png\">\n\n## How was this patch tested?\n\nGenetate the docs locally by `SKIP_API=1 jekyll build` and view it in the browser.\n\nAuthor: Shuai Lin <linshuai2012@gmail.com>\n\nCloses #14194 from lins05/fix-docs-formatting.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/3b6e1d094e153599e158331b10d33d74a667be5a"},{"sha":"fc17121d592acbd7405135cd576bafc5c574650e","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"Revert \"[SPARK-15271][MESOS] Allow force pulling executor docker images\"\n\nThis reverts commit 978cd5f125eb5a410bad2e60bf8385b11cf1b978.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/fc17121d592acbd7405135cd576bafc5c574650e"},{"sha":"cda4603de340d533c49feac1b244ddfd291f9bcf","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SQL][DOC] Fix a default name for parquet compression\n\n## What changes were proposed in this pull request?\nThis pr is to fix a wrong description for parquet default compression.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #14351 from maropu/FixParquetDoc.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/cda4603de340d533c49feac1b244ddfd291f9bcf"},{"sha":"f5ea7fe53974a7e8cbfc222b9a6f47669b53ccfd","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-16166][CORE] Also take off-heap memory usage into consideration in log and webui display\n\n## What changes were proposed in this pull request?\n\nCurrently in the log and UI display, only on-heap storage memory is calculated and displayed,\n\n```\n16/06/27 13:41:52 INFO MemoryStore: Block rdd_5_0 stored as values in memory (estimated size 17.8 KB, free 665.9 MB)\n```\n<img width=\"1232\" alt=\"untitled\" src=\"https://cloud.githubusercontent.com/assets/850797/16369960/53fb614e-3c6e-11e6-8fa3-7ffe65abcb49.png\">\n\nWith [SPARK-13992](https://issues.apache.org/jira/browse/SPARK-13992) off-heap memory is supported for data persistence, so here change to also take off-heap storage memory into consideration.\n\n## How was this patch tested?\n\nUnit test and local verification.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #13920 from jerryshao/SPARK-16166.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/f5ea7fe53974a7e8cbfc222b9a6f47669b53ccfd"},{"sha":"12f490b5c85cdee26d47eb70ad1a1edd00504f21","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-16715][TESTS] Fix a potential ExprId conflict for SubexpressionEliminationSuite.\"Semantic equals and hash\"\n\n## What changes were proposed in this pull request?\n\nSubexpressionEliminationSuite.\"Semantic equals and hash\" assumes the default AttributeReference's exprId wont' be \"ExprId(1)\". However, that depends on when this test runs. It may happen to use \"ExprId(1)\".\n\nThis PR detects the conflict and makes sure we create a different ExprId when the conflict happens.\n\n## How was this patch tested?\n\nJenkins unit tests.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #14350 from zsxwing/SPARK-16715.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/12f490b5c85cdee26d47eb70ad1a1edd00504f21"},{"sha":"c979c8bba02bc89cb9ad81b212f085a8a5490a07","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-14131][STREAMING] SQL Improved fix for avoiding potential deadlocks in HDFSMetadataLog\n\n## What changes were proposed in this pull request?\nCurrent fix for deadlock disables interrupts in the StreamExecution which getting offsets for all sources, and when writing to any metadata log, to avoid potential deadlocks in HDFSMetadataLog(see JIRA for more details). However, disabling interrupts can have unintended consequences in other sources. So I am making the fix more narrow, by disabling interrupt it only in the HDFSMetadataLog. This is a narrower fix for something risky like disabling interrupt.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #14292 from tdas/SPARK-14131.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/c979c8bba02bc89cb9ad81b212f085a8a5490a07"},{"sha":"db36e1e75d69d63b76312e85ae3a6c95cebbe65e","author":{"name":"Tao Lin","email":"12272d0502edbcb768ca9955d422d10260579899@gmail.com"},"message":"[SPARK-15590][WEBUI] Paginate Job Table in Jobs tab\n\n## What changes were proposed in this pull request?\n\nThis patch adds pagination support for the Job Tables in the Jobs tab. Pagination is provided for all of the three Job Tables (active, completed, and failed). Interactions (jumping, sorting, and setting page size) for paged tables are also included.\n\nThe diff didn't keep track of some lines based on the original ones. The function `makeRow`of the original `AllJobsPage.scala` is reused. They are separated at the beginning of the function `jobRow` (L427-439) and the function `row`(L594-618) in the new `AllJobsPage.scala`.\n\n## How was this patch tested?\n\nTested manually by using checking the Web UI after completing and failing hundreds of jobs.\nGenerate completed jobs by:\n```scala\nval d = sc.parallelize(Array(1,2,3,4,5))\nfor(i <- 1 to 255){ var b = d.collect() }\n```\nGenerate failed jobs by calling the following code multiple times:\n```scala\nvar b = d.map(_/0).collect()\n```\nInteractions like jumping, sorting, and setting page size are all tested.\n\nThis shows the pagination for completed jobs:\n![paginate success jobs](https://cloud.githubusercontent.com/assets/5558370/15986498/efa12ef6-303b-11e6-8b1d-c3382aeb9ad0.png)\n\nThis shows the sorting works in job tables:\n![sorting](https://cloud.githubusercontent.com/assets/5558370/15986539/98c8a81a-303c-11e6-86f2-8d2bc7924ee9.png)\n\nThis shows the pagination for failed jobs and the effect of jumping and setting page size:\n![paginate failed jobs](https://cloud.githubusercontent.com/assets/5558370/15986556/d8c1323e-303c-11e6-8e4b-7bdb030ea42b.png)\n\nAuthor: Tao Lin <nblintao@gmail.com>\n\nCloses #13620 from nblintao/dev.","distinct":true,"url":"https://api.github.com/repos/HyukjinKwon/spark/commits/db36e1e75d69d63b76312e85ae3a6c95cebbe65e"}]}
{"actor":{"display_login":"HyukjinKwon"}}