{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22676","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22676/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22676/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22676/events","html_url":"https://github.com/apache/spark/pull/22676","id":368083883,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIxMzQwMzM0","number":22676,"title":"[SPARK-25684][SQL] Organize header related codes in CSV datasource","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-10-09T07:55:43Z","updated_at":"2018-10-09T07:58:18Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22676","html_url":"https://github.com/apache/spark/pull/22676","diff_url":"https://github.com/apache/spark/pull/22676.diff","patch_url":"https://github.com/apache/spark/pull/22676.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n1. Move `CSVDataSource.makeSafeHeader` to `CSVUtils.makeSafeHeader` (as is).\r\n  *Rationale:*\r\n    - Historically and at the first place of refactoring (which I did), I intended to put all CSV specific handling (like options), filtering, extracting header, etc.\r\n    - See `JsonDataSource`. Now `CSVDataSource` is quite consistent with `JsonDataSource`. Since CSV's code path is quite complicated, we might better match them as possible as we can.\r\n\r\n2. Move `CSVDataSource.checkHeaderColumnNames` to `CSVHeaderChecker.checkHeaderColumnNames` (as is).\r\n  *Rationale:*\r\n      - Similar reasons above with 1.\r\n\r\n3. Put `enforceSchema` logics into `CSVHeaderChecker`.\r\n  *Rationale:*\r\n      - The checking header and column pruning stuff were added (per https://github.com/apache/spark/pull/20894 and https://github.com/apache/spark/pull/21296) but some of codes such as https://github.com/apache/spark/pull/21296 are duplicated\r\n      - Also, checking header code is basically here and there. We better put them in a single place, which is quite error-prone. See (https://github.com/apache/spark/pull/22656).\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests should cover this."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/428097408","html_url":"https://github.com/apache/spark/pull/22676#issuecomment-428097408","issue_url":"https://api.github.com/repos/apache/spark/issues/22676","id":428097408,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODA5NzQwOA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-10-09T07:58:18Z","updated_at":"2018-10-09T07:58:18Z","author_association":"NONE","body":"**[Test build #97147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97147/testReport)** for PR 22676 at commit [`5690668`](https://github.com/apache/spark/commit/56906680ab7d5d63be04bac2c3a19bb52baa3025)."}}
{"actor":{"display_login":"SparkQA"}}