{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22402","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22402/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22402/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22402/events","html_url":"https://github.com/apache/spark/pull/22402","id":359432010,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE0OTIyOTgw","number":22402,"title":"[SPARK-25414][SS] The numInputRows metrics can be incorrect for streaming self-join","user":{"login":"cloud-fan","id":3182036,"node_id":"MDQ6VXNlcjMxODIwMzY=","avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-09-12T11:12:18Z","updated_at":"2018-09-13T01:28:08Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22402","html_url":"https://github.com/apache/spark/pull/22402","diff_url":"https://github.com/apache/spark/pull/22402.diff","patch_url":"https://github.com/apache/spark/pull/22402.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFor self-join/self-union, Spark will produce a physical plan which has multiple `DataSourceV2ScanExec` instances referring to the same `ReadSupport` instance. In this case, we should not double count the numInputRows metrics.\r\n\r\nActually we already have 2 test cases to verify the behavior:\r\n1. `StreamingQuerySuite.input row calculation with same V2 source used twice in self-join`\r\n2. `KafkaMicroBatchSourceSuiteBase.ensure stream-stream self-join generates only one offset in log and correct metrics`.\r\n\r\nHowever, the first test is wrong written and exposes the bug. The bug is hidden in the second test, because exchange reuse is triggered and the plan only has one `DataSourceV2ScanExec` instance.\r\n\r\nNote that, some people may think this bug is caused by https://github.com/apache/spark/pull/22380 . It's obviously not because I didn't change any test in #22380. Actually we are aware of this bug and even mentioned it in the comment, but forgot to handle it:\r\n```\r\n...\r\n// 3. Multiple DataSourceV2ScanExec instance may refer to the same source (can happen with\r\n//    self-unions or self-joins). Add up the number of rows for each unique source.\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nThe updated test case"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/420852190","html_url":"https://github.com/apache/spark/pull/22402#issuecomment-420852190","issue_url":"https://api.github.com/repos/apache/spark/issues/22402","id":420852190,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDg1MjE5MA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-09-13T01:28:08Z","updated_at":"2018-09-13T01:28:08Z","author_association":"NONE","body":"**[Test build #96016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96016/testReport)** for PR 22402 at commit [`8649347`](https://github.com/apache/spark/commit/86493470d0ae7da14ccad7edd0facd73032896c9)."}}
{"actor":{"display_login":"SparkQA"}}