{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21886","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21886/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21886/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21886/events","html_url":"https://github.com/apache/spark/pull/21886","id":344945997,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0MjIwNTkw","number":21886,"title":"[SPARK-21274][SQL] Implement INTERSECT ALL clause","user":{"login":"dilipbiswal","id":14225158,"node_id":"MDQ6VXNlcjE0MjI1MTU4","avatar_url":"https://avatars0.githubusercontent.com/u/14225158?v=4","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-07-26T17:58:03Z","updated_at":"2018-07-27T01:52:24Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21886","html_url":"https://github.com/apache/spark/pull/21886","diff_url":"https://github.com/apache/spark/pull/21886.diff","patch_url":"https://github.com/apache/spark/pull/21886.patch"},"body":"## What changes were proposed in this pull request?\r\nImplements INTERSECT ALL clause through query rewrites using existing operators in Spark.  Please refer to [Link](https://drive.google.com/open?id=1nyW0T0b_ajUduQoPgZLAsyHK8s3_dko3ulQuxaLpUXE) for the design.\r\n\r\nInput Query\r\n``` SQL\r\nSELECT c1 FROM ut1 INTERSECT ALL SELECT c1 FROM ut2\r\n```\r\nRewritten Query\r\n```SQL\r\n   SELECT c1\r\n    FROM (\r\n         SELECT replicate_row(min_count, c1) AS (min_count, c1)\r\n         FROM (\r\n              SELECT c1,\r\n                     vcol1_cnt,\r\n                     vcol2_cnt,\r\n                     IF (vcol1_cnt > vcol2_cnt, vcol2_cnt, vcol1_cnt) AS min_count\r\n              FROM (\r\n                   SELECT   c1, count(vcol1) as vcol1_cnt, count(vcol2) as vcol2_cnt\r\n                   FROM (\r\n                        SELECT c1, true as vcol1, null as vcol2 FROM ut1\r\n                        UNION ALL\r\n                        SELECT c1, null as vcol1, true as vcol2 FROM ut2\r\n                        ) AS union_all\r\n                   GROUP BY c1\r\n                   HAVING vcol1_cnt >= 1 AND vcol2_cnt >= 1\r\n                  )\r\n              )\r\n          )\r\n```\r\n\r\n## How was this patch tested?\r\nAdded test cases in SQLQueryTestSuite, DataFrameSuite, SetOperationSuite"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/408287229","html_url":"https://github.com/apache/spark/pull/21886#issuecomment-408287229","issue_url":"https://api.github.com/repos/apache/spark/issues/21886","id":408287229,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODI4NzIyOQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-07-27T01:52:24Z","updated_at":"2018-07-27T01:52:24Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1391/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}