{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21004","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21004/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21004/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21004/events","html_url":"https://github.com/apache/spark/pull/21004","id":312330132,"number":21004,"title":"[SPARK-23896][SQL]Improve PartitioningAwareFileIndex","user":{"login":"gengliangwang","id":1097932,"avatar_url":"https://avatars0.githubusercontent.com/u/1097932?v=4","gravatar_id":"","url":"https://api.github.com/users/gengliangwang","html_url":"https://github.com/gengliangwang","followers_url":"https://api.github.com/users/gengliangwang/followers","following_url":"https://api.github.com/users/gengliangwang/following{/other_user}","gists_url":"https://api.github.com/users/gengliangwang/gists{/gist_id}","starred_url":"https://api.github.com/users/gengliangwang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gengliangwang/subscriptions","organizations_url":"https://api.github.com/users/gengliangwang/orgs","repos_url":"https://api.github.com/users/gengliangwang/repos","events_url":"https://api.github.com/users/gengliangwang/events{/privacy}","received_events_url":"https://api.github.com/users/gengliangwang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2018-04-08T18:56:22Z","updated_at":"2018-04-12T15:33:54Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21004","html_url":"https://github.com/apache/spark/pull/21004","diff_url":"https://github.com/apache/spark/pull/21004.diff","patch_url":"https://github.com/apache/spark/pull/21004.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently `PartitioningAwareFileIndex` accepts an optional parameter `userPartitionSchema`. If provided, it will combine the inferred partition schema with the parameter.\r\n\r\nHowever,\r\n1. to get `userPartitionSchema`, we need to  combine inferred partition schema with `userSpecifiedSchema`\r\n2. to get the inferred partition schema, we have to create a temporary file index.\r\n\r\nOnly after that, a final version of `PartitioningAwareFileIndex` can be created.\r\n\r\nThis can be improved by passing `userSpecifiedSchema` to `PartitioningAwareFileIndex`.\r\n\r\nWith the improvement, we can reduce redundant code and avoid parsing the file partition twice. \r\n## How was this patch tested?\r\nUnit test\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/380847917","html_url":"https://github.com/apache/spark/pull/21004#issuecomment-380847917","issue_url":"https://api.github.com/repos/apache/spark/issues/21004","id":380847917,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-04-12T15:33:54Z","updated_at":"2018-04-12T15:33:54Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}