{"push_id":979896776,"size":62,"distinct_size":18,"ref":"refs/heads/master","head":"19dc69de795eb08f3bab4988ad88732bf8ca7bae","before":"5cf20598cec4e60b53c0e40dc4243f436396e7fc","commits":[{"sha":"39cc620e9c9bf3722992da8d39a928ba638ff21d","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-13163][WEB UI] Column width on new History Server DataTables not getting set correctly\n\nThe column width for the new DataTables now adjusts for the current page rather than being hard-coded for the entire table's data.\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #11057 from ajbozarth/spark13163.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/39cc620e9c9bf3722992da8d39a928ba638ff21d"},{"sha":"4b80026f077edb25a2a4563abaf326240e89d4af","author":{"email":"02e10e341a2fea4868ba58fdfcfe400dcd4d1dd0@yahoo-inc.com","name":"zhuol"},"message":"[SPARK-13126] fix the right margin of history page.\n\nThe right margin of the history page is little bit off. A simple fix for that issue.\n\nAuthor: zhuol <zhuol@yahoo-inc.com>\n\nCloses #11029 from zhuoliu/13126.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/4b80026f077edb25a2a4563abaf326240e89d4af"},{"sha":"ce3bdaeeff718b7b5809eed15f7e4b5188e9fc7c","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[HOTFIX] Fix Scala 2.10 build break in TakeOrderedAndProjectSuite.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/ce3bdaeeff718b7b5809eed15f7e4b5188e9fc7c"},{"sha":"5947fa8fa1f95d8fc1537c1e37bc16bae8fe7988","author":{"email":"e58867706a26c17cde71f0f9f5e010ffb994f9e8@gmail.com","name":"Takeshi YAMAMURO"},"message":"[SPARK-13057][SQL] Add benchmark codes and the performance results for implemented compression schemes for InMemoryRelation\n\nThis pr adds benchmark codes for in-memory cache compression to make future developments and discussions more smooth.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #10965 from maropu/ImproveColumnarCache.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/5947fa8fa1f95d8fc1537c1e37bc16bae8fe7988"},{"sha":"29c547303f886b96b74b411ac70f0fa81113f086","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-12414][CORE] Remove closure serializer\n\nRemove spark.closure.serializer option and use JavaSerializer always\n\nCC andrewor14 rxin I see there's a discussion in the JIRA but just thought I'd offer this for a look at what the change would be.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #11150 from srowen/SPARK-12414.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/29c547303f886b96b74b411ac70f0fa81113f086"},{"sha":"0902e20288366db6270f3a444e66114b1b63a3e2","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-13146][SQL] Management API for continuous queries\n\n### Management API for Continuous Queries\n\n**API for getting status of each query**\n- Whether active or not\n- Unique name of each query\n- Status of the sources and sinks\n- Exceptions\n\n**API for managing each query**\n- Immediately stop an active query\n- Waiting for a query to be terminated, correctly or with error\n\n**API for managing multiple queries**\n- Listing all active queries\n- Getting an active query by name\n- Waiting for any one of the active queries to be terminated\n\n**API for listening to query life cycle events**\n- ContinuousQueryListener API for query start, progress and termination events.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #11030 from tdas/streaming-df-management-api.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/0902e20288366db6270f3a444e66114b1b63a3e2"},{"sha":"719973b05ef6d6b9fbb83d76aebac6454ae84fad","author":{"email":"a4c2e0cc8ad292da17d16a102d8c116e1b23db77@databricks.com","name":"raela"},"message":"[SPARK-13274] Fix Aggregator Links on GroupedDataset Scala API\n\nUpdate Aggregator links to point to #org.apache.spark.sql.expressions.Aggregator\n\nAuthor: raela <raela@databricks.com>\n\nCloses #11158 from raelawang/master.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/719973b05ef6d6b9fbb83d76aebac6454ae84fad"},{"sha":"663cc400f3b927633e47df07eea409da0e9ae70e","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-12725][SQL] Resolving Name Conflicts in SQL Generation and Name Ambiguity Caused by Internally Generated Expressions\n\nSome analysis rules generate aliases or auxiliary attribute references with the same name but different expression IDs. For example, `ResolveAggregateFunctions` introduces `havingCondition` and `aggOrder`, and `DistinctAggregationRewriter` introduces `gid`.\n\nThis is OK for normal query execution since these attribute references get expression IDs. However, it's troublesome when converting resolved query plans back to SQL query strings since expression IDs are erased.\n\nHere's an example Spark 1.6.0 snippet for illustration:\n```scala\nsqlContext.range(10).select('id as 'a, 'id as 'b).registerTempTable(\"t\")\nsqlContext.sql(\"SELECT SUM(a) FROM t GROUP BY a, b ORDER BY COUNT(a), COUNT(b)\").explain(true)\n```\nThe above code produces the following resolved plan:\n```\n== Analyzed Logical Plan ==\n_c0: bigint\nProject [_c0#101L]\n+- Sort [aggOrder#102L ASC,aggOrder#103L ASC], true\n   +- Aggregate [a#47L,b#48L], [(sum(a#47L),mode=Complete,isDistinct=false) AS _c0#101L,(count(a#47L),mode=Complete,isDistinct=false) AS aggOrder#102L,(count(b#48L),mode=Complete,isDistinct=false) AS aggOrder#103L]\n      +- Subquery t\n         +- Project [id#46L AS a#47L,id#46L AS b#48L]\n            +- LogicalRDD [id#46L], MapPartitionsRDD[44] at range at <console>:26\n```\nHere we can see that both aggregate expressions in `ORDER BY` are extracted into an `Aggregate` operator, and both of them are named `aggOrder` with different expression IDs.\n\nThe solution is to automatically add the expression IDs into the attribute name for the Alias and AttributeReferences that are generated by Analyzer in SQL Generation.\n\nIn this PR, it also resolves another issue. Users could use the same name as the internally generated names. The duplicate names should not cause name ambiguity. When resolving the column, Catalyst should not pick the column that is internally generated.\n\nCould you review the solution? marmbrus liancheng\n\nI did not set the newly added flag for all the alias and attribute reference generated by Analyzers. Please let me know if I should do it? Thank you!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11050 from gatorsmile/namingConflicts.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/663cc400f3b927633e47df07eea409da0e9ae70e"},{"sha":"0f09f0226983cdc409ef504dff48395787dc844f","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-13205][SQL] SQL Generation Support for Self Join\n\nThis PR addresses two issues:\n  - Self join does not work in SQL Generation\n  - When creating new instances for `LogicalRelation`, `metastoreTableIdentifier` is lost.\n\nliancheng Could you please review the code changes? Thank you!\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11084 from gatorsmile/selfJoinInSQLGen.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/0f09f0226983cdc409ef504dff48395787dc844f"},{"sha":"b5761d150b66ee0ae5f1be897d9d7a1abb039884","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12706] [SQL] grouping() and grouping_id()\n\nGrouping() returns a column is aggregated or not, grouping_id() returns the aggregation levels.\n\ngrouping()/grouping_id() could be used with window function, but does not work in having/sort clause, will be fixed by another PR.\n\nThe GROUPING__ID/grouping_id() in Hive is wrong (according to docs), we also did it wrongly, this PR change that to match the behavior in most databases (also the docs of Hive).\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10677 from davies/grouping.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/b5761d150b66ee0ae5f1be897d9d7a1abb039884"},{"sha":"8f744fe3d931c2380613b8e5bafa1bb1fd292839","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13234] [SQL] remove duplicated SQL metrics\n\nFor lots of SQL operators, we have metrics for both of input and output, the number of input rows should be exactly the number of output rows of child, we could only have metrics for output rows.\n\nAfter we improved the performance using whole stage codegen, the overhead of SQL metrics are not trivial anymore, we should avoid that if it's not necessary.\n\nThis PR remove all the SQL metrics for number of input rows, add SQL metric of number of output rows for all LeafNode. All remove the SQL metrics from those operators that have the same number of rows from input and output (for example, Projection, we may don't need that).\n\nThe new SQL UI will looks like:\n\n![metrics](https://cloud.githubusercontent.com/assets/40902/12965227/63614e5e-d009-11e5-88b3-84fea04f9c20.png)\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11163 from davies/remove_metrics.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/8f744fe3d931c2380613b8e5bafa1bb1fd292839"},{"sha":"1842c55d89ae99a610a955ce61633a9084e000f2","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-13276] Catch bad characters at the end of a Table Identifier/Expression string\n\nThe parser currently parses the following strings without a hitch:\n* Table Identifier:\n  * `a.b.c` should fail, but results in the following table identifier `a.b`\n  * `table!#` should fail, but results in the following table identifier `table`\n* Expression\n  * `1+2 r+e` should fail, but results in the following expression `1 + 2`\n\nThis PR fixes this by adding terminated rules for both expression parsing and table identifier parsing.\n\ncc cloud-fan (we discussed this in https://github.com/apache/spark/pull/10649) jayadevanmurali (this causes your PR https://github.com/apache/spark/pull/11051 to fail)\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #11159 from hvanhovell/SPARK-13276.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/1842c55d89ae99a610a955ce61633a9084e000f2"},{"sha":"e88bff12795a6134e2e7204996b603e948380e18","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-13235][SQL] Removed an Extra Distinct from the Plan when Using Union in SQL\n\nCurrently, the parser added two `Distinct` operators in the plan if we are using `Union` or `Union Distinct` in the SQL. This PR is to remove the extra `Distinct` from the plan.\n\nFor example, before the fix, the following query has a plan with two `Distinct`\n```scala\nsql(\"select * from t0 union select * from t0\").explain(true)\n```\n```\n== Parsed Logical Plan ==\n'Project [unresolvedalias(*,None)]\n+- 'Subquery u_2\n   +- 'Distinct\n      +- 'Project [unresolvedalias(*,None)]\n         +- 'Subquery u_1\n            +- 'Distinct\n               +- 'Union\n                  :- 'Project [unresolvedalias(*,None)]\n                  :  +- 'UnresolvedRelation `t0`, None\n                  +- 'Project [unresolvedalias(*,None)]\n                     +- 'UnresolvedRelation `t0`, None\n\n== Analyzed Logical Plan ==\nid: bigint\nProject [id#16L]\n+- Subquery u_2\n   +- Distinct\n      +- Project [id#16L]\n         +- Subquery u_1\n            +- Distinct\n               +- Union\n                  :- Project [id#16L]\n                  :  +- Subquery t0\n                  :     +- Relation[id#16L] ParquetRelation\n                  +- Project [id#16L]\n                     +- Subquery t0\n                        +- Relation[id#16L] ParquetRelation\n\n== Optimized Logical Plan ==\nAggregate [id#16L], [id#16L]\n+- Aggregate [id#16L], [id#16L]\n   +- Union\n      :- Project [id#16L]\n      :  +- Relation[id#16L] ParquetRelation\n      +- Project [id#16L]\n         +- Relation[id#16L] ParquetRelation\n```\nAfter the fix, the plan is changed without the extra `Distinct` as follows:\n```\n== Parsed Logical Plan ==\n'Project [unresolvedalias(*,None)]\n+- 'Subquery u_1\n   +- 'Distinct\n      +- 'Union\n         :- 'Project [unresolvedalias(*,None)]\n         :  +- 'UnresolvedRelation `t0`, None\n         +- 'Project [unresolvedalias(*,None)]\n           +- 'UnresolvedRelation `t0`, None\n\n== Analyzed Logical Plan ==\nid: bigint\nProject [id#17L]\n+- Subquery u_1\n   +- Distinct\n      +- Union\n        :- Project [id#16L]\n        :  +- Subquery t0\n        :     +- Relation[id#16L] ParquetRelation\n        +- Project [id#16L]\n          +- Subquery t0\n          +- Relation[id#16L] ParquetRelation\n\n== Optimized Logical Plan ==\nAggregate [id#17L], [id#17L]\n+- Union\n  :- Project [id#16L]\n  :  +- Relation[id#16L] ParquetRelation\n  +- Project [id#16L]\n    +- Relation[id#16L] ParquetRelation\n```\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #11120 from gatorsmile/unionDistinct.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/e88bff12795a6134e2e7204996b603e948380e18"},{"sha":"18bcbbdd84e80222d1d29530831c6d68d02e7593","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-13270][SQL] Remove extra new lines in whole stage codegen and include pipeline plan in comments.\n\nAuthor: Nong Li <nong@databricks.com>\n\nCloses #11155 from nongli/spark-13270.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/18bcbbdd84e80222d1d29530831c6d68d02e7593"},{"sha":"c2f21d88981789fe3366f2c4040445aeff5bf083","author":{"email":"93406da1fa8ebf4406b5d741093e2ccfc7f09e95@nttdata.co.jp","name":"Sasaki Toru"},"message":"[SPARK-13264][DOC] Removed multi-byte characters in spark-env.sh.template\n\nIn spark-env.sh.template, there are multi-byte characters, this PR will remove it.\n\nAuthor: Sasaki Toru <sasakitoa@nttdata.co.jp>\n\nCloses #11149 from sasakitoa/remove_multibyte_in_sparkenv.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/c2f21d88981789fe3366f2c4040445aeff5bf083"},{"sha":"f9ae99fee13681e436fde9899b6a189746348ba1","author":{"email":"6bb95bb4a898fecc0c804e7561dfd22e9e79dfbc@gmail.com","name":"Junyang"},"message":"[SPARK-13074][CORE] Add JavaSparkContext. getPersistentRDDs method\n\nThe \"getPersistentRDDs()\" is a useful API of SparkContext to get cached RDDs. However, the JavaSparkContext does not have this API.\n\nAdd a simple getPersistentRDDs() to get java.util.Map<Integer, JavaRDD> for Java users.\n\nAuthor: Junyang <fly.shenjy@gmail.com>\n\nCloses #10978 from flyjy/master.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/f9ae99fee13681e436fde9899b6a189746348ba1"},{"sha":"13c17cbb0530d52b9a08d5197017c96501d99e8c","author":{"email":"4cdd7d7856bb210504d2f96aa5d4e36d0cc9334c@us.ibm.com","name":"Alex Bozarth"},"message":"[SPARK-13124][WEB UI] Fixed CSS and JS issues caused by addition of JQuery DataTables\n\nMade sure the old tables continue to use the old css and the new DataTables use the new css. Also fixed it so the Safari Web Inspector doesn't throw errors when on the new DataTables pages.\n\nAuthor: Alex Bozarth <ajbozart@us.ibm.com>\n\nCloses #11038 from ajbozarth/spark13124.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/13c17cbb0530d52b9a08d5197017c96501d99e8c"},{"sha":"219a74a7c2d3b858224c4738190ccc92d7cbf06d","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[STREAMING][TEST] Fix flaky streaming.FailureSuite\n\nUnder some corner cases, the test suite failed to shutdown the SparkContext causing cascaded failures. This fix does two things\n- Makes sure no SparkContext is active after every test\n- Makes sure StreamingContext is always shutdown (prevents leaking of StreamingContexts as well, just in case)\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #11166 from tdas/fix-failuresuite.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/219a74a7c2d3b858224c4738190ccc92d7cbf06d"},{"sha":"e31c80737b7f4d8baa02230788e3963433cb3ef9","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13277][SQL] ANTLR ignores other rule using the USING keyword\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-13277\n\nThere is an ANTLR warning during compilation:\n\n    warning(200): org/apache/spark/sql/catalyst/parser/SparkSqlParser.g:938:7:\n    Decision can match input such as \"KW_USING Identifier\" using multiple alternatives: 2, 3\n\n    As a result, alternative(s) 3 were disabled for that input\n\nThis patch is to fix it.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #11168 from viirya/fix-parser-using.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/e31c80737b7f4d8baa02230788e3963433cb3ef9"},{"sha":"0d50a22084eea91d4efb0a3ed3fa59b8d9680795","author":{"email":"afd4e7c6358d74b219992bc91704a9760efe52ac@tcs.com","name":"jayadevanmurali"},"message":"[SPARK-12982][SQL] Add table name validation in temp table registration\n\nAdd the table name validation at the temp table creation\n\nAuthor: jayadevanmurali <jayadevan.m@tcs.com>\n\nCloses #11051 from jayadevanmurali/branch-0.2-SPARK-12982.","distinct":false,"url":"https://api.github.com/repos/andrewor14/spark/commits/0d50a22084eea91d4efb0a3ed3fa59b8d9680795"}]}