{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13271","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13271/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13271/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13271/events","html_url":"https://github.com/apache/spark/pull/13271","id":156401215,"number":13271,"title":"[SPARK-15495][SQL][WIP] Improve the explain output","user":{"login":"clockfly","id":2595532,"avatar_url":"https://avatars.githubusercontent.com/u/2595532?v=3","gravatar_id":"","url":"https://api.github.com/users/clockfly","html_url":"https://github.com/clockfly","followers_url":"https://api.github.com/users/clockfly/followers","following_url":"https://api.github.com/users/clockfly/following{/other_user}","gists_url":"https://api.github.com/users/clockfly/gists{/gist_id}","starred_url":"https://api.github.com/users/clockfly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clockfly/subscriptions","organizations_url":"https://api.github.com/users/clockfly/orgs","repos_url":"https://api.github.com/users/clockfly/repos","events_url":"https://api.github.com/users/clockfly/events{/privacy}","received_events_url":"https://api.github.com/users/clockfly/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":34,"created_at":"2016-05-24T01:28:40Z","updated_at":"2016-05-27T21:41:53Z","closed_at":"2016-05-27T21:10:06Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13271","html_url":"https://github.com/apache/spark/pull/13271","diff_url":"https://github.com/apache/spark/pull/13271.diff","patch_url":"https://github.com/apache/spark/pull/13271.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR adds a verbose flag to control the explain output of Aggregator operator. \r\n\r\n1.  **Make the Aggregation output simple:**\r\n\r\n    Before change:\r\n\r\n    ```\r\n    scala> Seq((1,2,3)).toDF(\"a\", \"b\", \"c\").createTempView(\"df1\") \r\n    scala> spark.sql(\"select count(a), count(c), b from df1 group by b\").explain()\r\n    == Physical Plan ==\r\n    *TungstenAggregate(key=[b#8], functions=[(count(1),mode=Final,isDistinct=false),(count(1),mode=Final,isDistinct=false)], output=[count(a)#63L,count(c)#64L,b#8])\r\n    +- Exchange hashpartitioning(b#8, 200), None\r\n    +- *TungstenAggregate(key=[b#8], functions=[(count(1),mode=Partial,isDistinct=false),(count(1),mode=Partial,isDistinct=false)], output=[b#8,count#67L,count#68L])\r\n        +- LocalTableScan [b#8], [[2]]\r\n    ```\r\n\r\n    After change:\r\n\r\n    ```\r\n    scala> spark.sql(\"select count(a), count(c), b from df1 group by b\").explain()\r\n    == Physical Plan ==\r\n    *Aggregate(key=[b#8], functions=[count(1)(Final),count(1)(Final)])\r\n    +- Exchange hashpartitioning(b#8, 200)\r\n    +- *Aggregate(key=[b#8], functions=[count(1)(Partial),count(1)(Partial)])\r\n        +- LocalTableScan [b#8: int], [[2]]\r\n    ```\r\n    \r\n    For explain(true), the output is:\r\n    \r\n    ```\r\n    ...\r\n    == Physical Plan ==\r\n    *Aggregate(key=[b#8], functions=[count(1)(Final),count(1)(Final)], output=[count(a)#54L,count(c)#55L,b#8])\r\n    +- Exchange hashpartitioning(b#8, 200)\r\n    +- *Aggregate(key=[b#8], functions=[count(1)(Partial),count(1)(Partial)], output=[b#8,count#58L,count#59L])\r\n        +- LocalTableScan [b#8: int], [[2]]\r\n    ```\r\n    \r\n2. **To override the verbosity setting, the operator can define a function like this:**\r\n \r\n    ```\r\n    private[sql]\r\n    def simpleStringImpl(verbose: Boolean): String = {\r\n      if (verbose) \"verbose output\"\r\n    } else {\r\n      \"less verbose\"\r\n    }\r\n    ```\r\n\r\n## How was this patch tested?\r\n\r\nManual test and existing UT."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/222259604","html_url":"https://github.com/apache/spark/pull/13271#issuecomment-222259604","issue_url":"https://api.github.com/repos/apache/spark/issues/13271","id":222259604,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-05-27T21:41:52Z","updated_at":"2016-05-27T21:41:52Z","body":"**[Test build #59519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59519/consoleFull)** for PR 13271 at commit [`eabde70`](https://github.com/apache/spark/commit/eabde70f8d3d513535af08e02ea58a4fbf2fba79).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}