{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21319","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21319/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21319/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21319/events","html_url":"https://github.com/apache/spark/pull/21319","id":322724797,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg3NzU3ODEz","number":21319,"title":"[SPARK-24267][SQL] explicitly keep DataSourceReader in DataSourceV2Relation","user":{"login":"cloud-fan","id":3182036,"node_id":"MDQ6VXNlcjMxODIwMzY=","avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-05-14T08:50:05Z","updated_at":"2018-06-12T01:03:14Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21319","html_url":"https://github.com/apache/spark/pull/21319","diff_url":"https://github.com/apache/spark/pull/21319.diff","patch_url":"https://github.com/apache/spark/pull/21319.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nTo workaround the issue that Spark only have statistics in logical plan, we need to create the `DataSourceReader` at logical phase so that we can report statistics. Currently this created `DataSourceReader` is kept as a `lazy val` in `DataSourceV2Relation`. If we think about how `lazy val` is implemented in Scala, we actually keep a `DataSourceReader` instance in `DataSourceV2Relation`, and exclude it when defining equality of `DataSourceV2Relation`.\r\n\r\nThis works, but have 2 problems:\r\n1. after the pushdown rule, if `DataSourceV2Relation` get transformed and return a new copy, we will re-do the pushdown and re-create the `DataSourceReader`. This is not a problem now but we may encounter such cases in the future.\r\n2. It's more intutive to put operator pushdown logic in the rule. When adding pushdown support for streaming relations, It's easier to share the pushdown code for batch and streaming data sources, if the pushdown logic is in the rule. Otherwise we may need to duplicate the pushdown logic in `StreamingDataSourceV2Relation`.\r\n\r\nThis PR proposes to implement the `lazy val` by ourselves: keep `DataSourceReader` as an optional parameter in the constructor of `DataSourceV2Relation`, exclude it in the equality definition but include it when copying.\r\n\r\nMoving statistics to physical plans would be best, but it's a huge topic and won't be covered in this PR.\r\n## How was this patch tested?\r\n\r\nexisting tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/396432909","html_url":"https://github.com/apache/spark/pull/21319#issuecomment-396432909","issue_url":"https://api.github.com/repos/apache/spark/issues/21319","id":396432909,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjQzMjkwOQ==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-06-12T01:03:14Z","updated_at":"2018-06-12T01:03:14Z","author_association":"NONE","body":"**[Test build #91680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91680/testReport)** for PR 21319 at commit [`91fdedc`](https://github.com/apache/spark/commit/91fdedc4d91a7abde5f6b64dbfcf354b67d89a48)."}}
{"actor":{"display_login":"SparkQA"}}