{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12855","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12855/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12855/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12855/events","html_url":"https://github.com/apache/spark/pull/12855","id":152681366,"number":12855,"title":"[SPARK-10216][SQL] Avoid creating empty files during overwriting with group by query","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2016-05-03T01:29:16Z","updated_at":"2016-05-17T04:26:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12855","html_url":"https://github.com/apache/spark/pull/12855","diff_url":"https://github.com/apache/spark/pull/12855.diff","patch_url":"https://github.com/apache/spark/pull/12855.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently, `INSERT INTO` with `GROUP BY` query tries to make at least 200 files (default value of `spark.sql.shuffle.partition`), which results in lots of empty files.\r\n\r\nThis PR makes it avoid creating empty files during overwriting into Hive table and in internal data sources  with group by query.\r\n\r\nThis checks whether the given partition has data in it or not and creates/writes file only when it actually has data.\r\n\r\n## How was this patch tested?\r\n\r\nUnittests in `InsertIntoHiveTableSuite` and `HadoopFsRelationTest`. \r\n\r\nCloses #8411"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/219616350","html_url":"https://github.com/apache/spark/pull/12855#issuecomment-219616350","issue_url":"https://api.github.com/repos/apache/spark/issues/12855","id":219616350,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-05-17T04:26:52Z","updated_at":"2016-05-17T04:26:52Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58662/\nTest FAILed."}}