{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20647","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20647/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20647/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20647/events","html_url":"https://github.com/apache/spark/pull/20647","id":298852525,"number":20647,"title":"[SPARK-23303][SQL] improve the explain result for data source v2 relations","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-02-21T06:06:56Z","updated_at":"2018-02-27T08:22:53Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20647","html_url":"https://github.com/apache/spark/pull/20647","diff_url":"https://github.com/apache/spark/pull/20647.diff","patch_url":"https://github.com/apache/spark/pull/20647.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThe proposed explain format:\r\n**[streaming header] [RelationV2/ScanV2] [data source name] [output] [pushed filters] [options]**\r\n\r\n**streaming header**: if it's a streaming relation, put a \"Streaming\" at the beginning.\r\n**RelationV2/ScanV2**: if it's a logical plan, put a \"RelationV2\", else, put a \"ScanV2\"\r\n**data source name**: the simple class name of the data source implementation\r\n**output**: a string of the plan output attributes\r\n**pushed filters**: a string of all the filters that have been pushed to this data source\r\n**options**: all the options to create the data source reader.\r\n\r\nThe current explain result for data source v2 relation is unreadable:\r\n```\r\n== Parsed Logical Plan ==\r\n'Filter ('i > 6)\r\n+- AnalysisBarrier\r\n      +- Project [j#1]\r\n         +- DataSourceV2Relation [i#0, j#1], org.apache.spark.sql.sources.v2.AdvancedDataSourceV2$Reader@3b415940\r\n\r\n== Analyzed Logical Plan ==\r\nj: int\r\nProject [j#1]\r\n+- Filter (i#0 > 6)\r\n   +- Project [j#1, i#0]\r\n      +- DataSourceV2Relation [i#0, j#1], org.apache.spark.sql.sources.v2.AdvancedDataSourceV2$Reader@3b415940\r\n\r\n== Optimized Logical Plan ==\r\nProject [j#1]\r\n+- Filter isnotnull(i#0)\r\n   +- DataSourceV2Relation [i#0, j#1], org.apache.spark.sql.sources.v2.AdvancedDataSourceV2$Reader@3b415940\r\n\r\n== Physical Plan ==\r\n*(1) Project [j#1]\r\n+- *(1) Filter isnotnull(i#0)\r\n   +- *(1) DataSourceV2Scan [i#0, j#1], org.apache.spark.sql.sources.v2.AdvancedDataSourceV2$Reader@3b415940\r\n```\r\n\r\nafter this PR\r\n```\r\n== Parsed Logical Plan ==\r\n'Project [unresolvedalias('j, None)]\r\n+- AnalysisBarrier\r\n      +- RelationV2 AdvancedDataSourceV2[i#0, j#1]\r\n\r\n== Analyzed Logical Plan ==\r\nj: int\r\nProject [j#1]\r\n+- RelationV2 AdvancedDataSourceV2[i#0, j#1]\r\n\r\n== Optimized Logical Plan ==\r\nRelationV2 AdvancedDataSourceV2[j#1]\r\n\r\n== Physical Plan ==\r\n*(1) ScanV2 AdvancedDataSourceV2[j#1]\r\n```\r\n-------\r\n```\r\n== Analyzed Logical Plan ==\r\ni: int, j: int\r\nFilter (i#88 > 3)\r\n+- RelationV2 JavaAdvancedDataSourceV2[i#88, j#89]\r\n\r\n== Optimized Logical Plan ==\r\nFilter isnotnull(i#88)\r\n+- RelationV2 JavaAdvancedDataSourceV2[i#88, j#89] (Pushed Filters: [GreaterThan(i,3)])\r\n\r\n== Physical Plan ==\r\n*(1) Filter isnotnull(i#88)\r\n+- *(1) ScanV2 JavaAdvancedDataSourceV2[i#88, j#89] (Pushed Filters: [GreaterThan(i,3)])\r\n```\r\n\r\nan example for streaming query\r\n```\r\n== Parsed Logical Plan ==\r\nAggregate [value#6], [value#6, count(1) AS count(1)#11L]\r\n+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true, false) AS value#6]\r\n   +- MapElements <function1>, class java.lang.String, [StructField(value,StringType,true)], obj#5: java.lang.String\r\n      +- DeserializeToObject cast(value#25 as string).toString, obj#4: java.lang.String\r\n         +- Streaming RelationV2 MemoryStreamDataSource[value#25]\r\n\r\n== Analyzed Logical Plan ==\r\nvalue: string, count(1): bigint\r\nAggregate [value#6], [value#6, count(1) AS count(1)#11L]\r\n+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true, false) AS value#6]\r\n   +- MapElements <function1>, class java.lang.String, [StructField(value,StringType,true)], obj#5: java.lang.String\r\n      +- DeserializeToObject cast(value#25 as string).toString, obj#4: java.lang.String\r\n         +- Streaming RelationV2 MemoryStreamDataSource[value#25]\r\n\r\n== Optimized Logical Plan ==\r\nAggregate [value#6], [value#6, count(1) AS count(1)#11L]\r\n+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true, false) AS value#6]\r\n   +- MapElements <function1>, class java.lang.String, [StructField(value,StringType,true)], obj#5: java.lang.String\r\n      +- DeserializeToObject value#25.toString, obj#4: java.lang.String\r\n         +- Streaming RelationV2 MemoryStreamDataSource[value#25]\r\n\r\n== Physical Plan ==\r\n*(4) HashAggregate(keys=[value#6], functions=[count(1)], output=[value#6, count(1)#11L])\r\n+- StateStoreSave [value#6], state info [ checkpoint = *********(redacted)/cloud/dev/spark/target/tmp/temporary-549f264b-2531-4fcb-a52f-433c77347c12/state, runId = f84d9da9-2f8c-45c1-9ea1-70791be684de, opId = 0, ver = 0, numPartitions = 5], Complete, 0\r\n   +- *(3) HashAggregate(keys=[value#6], functions=[merge_count(1)], output=[value#6, count#16L])\r\n      +- StateStoreRestore [value#6], state info [ checkpoint = *********(redacted)/cloud/dev/spark/target/tmp/temporary-549f264b-2531-4fcb-a52f-433c77347c12/state, runId = f84d9da9-2f8c-45c1-9ea1-70791be684de, opId = 0, ver = 0, numPartitions = 5]\r\n         +- *(2) HashAggregate(keys=[value#6], functions=[merge_count(1)], output=[value#6, count#16L])\r\n            +- Exchange hashpartitioning(value#6, 5)\r\n               +- *(1) HashAggregate(keys=[value#6], functions=[partial_count(1)], output=[value#6, count#16L])\r\n                  +- *(1) SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true, false) AS value#6]\r\n                     +- *(1) MapElements <function1>, obj#5: java.lang.String\r\n                        +- *(1) DeserializeToObject value#25.toString, obj#4: java.lang.String\r\n                           +- *(1) ScanV2 MemoryStreamDataSource[value#25]\r\n```\r\n## How was this patch tested?\r\n\r\nN/A"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/368783932","html_url":"https://github.com/apache/spark/pull/20647#issuecomment-368783932","issue_url":"https://api.github.com/repos/apache/spark/issues/20647","id":368783932,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-02-27T08:22:53Z","updated_at":"2018-02-27T08:22:53Z","author_association":"NONE","body":"**[Test build #87710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87710/testReport)** for PR 20647 at commit [`8c5b934`](https://github.com/apache/spark/commit/8c5b934c98485154f711d975864479761f01b481)."}}
{"actor":{"display_login":"SparkQA"}}