{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13815","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13815/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13815/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13815/events","html_url":"https://github.com/apache/spark/pull/13815","id":161533902,"number":13815,"title":"[SPARK-15443][SQL]Fix 'explain' for streaming Dataset","user":{"login":"zsxwing","id":1000778,"avatar_url":"https://avatars.githubusercontent.com/u/1000778?v=3","gravatar_id":"","url":"https://api.github.com/users/zsxwing","html_url":"https://github.com/zsxwing","followers_url":"https://api.github.com/users/zsxwing/followers","following_url":"https://api.github.com/users/zsxwing/following{/other_user}","gists_url":"https://api.github.com/users/zsxwing/gists{/gist_id}","starred_url":"https://api.github.com/users/zsxwing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zsxwing/subscriptions","organizations_url":"https://api.github.com/users/zsxwing/orgs","repos_url":"https://api.github.com/users/zsxwing/repos","events_url":"https://api.github.com/users/zsxwing/events{/privacy}","received_events_url":"https://api.github.com/users/zsxwing/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-06-21T20:39:07Z","updated_at":"2016-06-23T23:03:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13815","html_url":"https://github.com/apache/spark/pull/13815","diff_url":"https://github.com/apache/spark/pull/13815.diff","patch_url":"https://github.com/apache/spark/pull/13815.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n- Fix the `explain` command for streaming Dataset/DataFrame. E.g.,\r\n```\r\n== Parsed Logical Plan ==\r\n'SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- 'MapElements <function1>, obj#6: java.lang.String\r\n   +- 'DeserializeToObject unresolveddeserializer(createexternalrow(getcolumnbyordinal(0, StringType).toString, StructField(value,StringType,true))), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- StreamingRelation FileSource[/Users/zsx/stream], [value#0]\r\n\r\n== Analyzed Logical Plan ==\r\nvalue: string\r\nSerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- MapElements <function1>, obj#6: java.lang.String\r\n   +- DeserializeToObject createexternalrow(value#0.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- StreamingRelation FileSource[/Users/zsx/stream], [value#0]\r\n\r\n== Optimized Logical Plan ==\r\nSerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- MapElements <function1>, obj#6: java.lang.String\r\n   +- DeserializeToObject createexternalrow(value#0.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- StreamingRelation FileSource[/Users/zsx/stream], [value#0]\r\n\r\n== Physical Plan ==\r\n*SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- *MapElements <function1>, obj#6: java.lang.String\r\n   +- *DeserializeToObject createexternalrow(value#0.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- *Filter <function1>.apply\r\n         +- StreamingRelation FileSource[/Users/zsx/stream], [value#0]\r\n```\r\n\r\n- Add `StreamingQuery.explain` to display the last execution plan. E.g.,\r\n```\r\n== Parsed Logical Plan ==\r\nSerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- MapElements <function1>, obj#6: java.lang.String\r\n   +- DeserializeToObject createexternalrow(value#12.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- Relation[value#12] text\r\n\r\n== Analyzed Logical Plan ==\r\nvalue: string\r\nSerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- MapElements <function1>, obj#6: java.lang.String\r\n   +- DeserializeToObject createexternalrow(value#12.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- Relation[value#12] text\r\n\r\n== Optimized Logical Plan ==\r\nSerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- MapElements <function1>, obj#6: java.lang.String\r\n   +- DeserializeToObject createexternalrow(value#12.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- Filter <function1>.apply\r\n         +- Relation[value#12] text\r\n\r\n== Physical Plan ==\r\n*SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, java.lang.String, true], true) AS value#7]\r\n+- *MapElements <function1>, obj#6: java.lang.String\r\n   +- *DeserializeToObject createexternalrow(value#12.toString, StructField(value,StringType,true)), obj#5: org.apache.spark.sql.Row\r\n      +- *Filter <function1>.apply\r\n         +- *Scan text [value#12] Format: org.apache.spark.sql.execution.datasources.text.TextFileFormat@1836ab91, InputPaths: file:/Users/zsx/stream/a.txt, file:/Users/zsx/stream/b.txt, file:/Users/zsx/stream/c.txt, PushedFilters: [], ReadSchema: struct<value:string>\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nThe added unit tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/228211392","html_url":"https://github.com/apache/spark/pull/13815#issuecomment-228211392","issue_url":"https://api.github.com/repos/apache/spark/issues/13815","id":228211392,"user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T23:03:02Z","updated_at":"2016-06-23T23:03:02Z","body":"LGTM. Merging to master and branch 2.0."}}
{"actor":{"display_login":"yhuai"}}