{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13988","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13988/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13988/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13988/events","html_url":"https://github.com/apache/spark/pull/13988","id":163075840,"number":13988,"title":"[SPARK-16101][SQL] Refactoring CSV data source to be consistent with JSON data source","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2016-06-30T03:54:29Z","updated_at":"2016-10-08T10:42:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13988","html_url":"https://github.com/apache/spark/pull/13988","diff_url":"https://github.com/apache/spark/pull/13988.diff","patch_url":"https://github.com/apache/spark/pull/13988.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR refactors CSV data source to be consistent with JSON data source.\r\n\r\nThis PR removes classes `CSVParser` and introduces new classes `UnivocityParser`, `UnivocityGenerator` and `CSVUtils` to be consistent with JSON data source (`JacksonParser`, `JacksonGenerator` and `JacksonUtils`). Also, `CSVRelation` is merged with `CSVFileFormat` just like `JSONFileFormat`.\r\n\r\nThis is a rough design:\r\n\r\n```\r\n             UnivocityGenerator - (writing logics in CsvParser)\r\n           ↗\r\nCsvParser\r\n           ↘\r\n             UnivocityParser - (parsing logics in CsvParser and CSVRelation)\r\n           ↗\r\nCSVRelation\r\n           ↘\r\n             CSVFileFormat - (CSVOutputWriterFactory and CsvOutputWriter in CSVRelation)\r\n\r\n             CSVUtils - (common logics across CSV data source, e.g., filtering the first line).\r\n```\r\n\r\n- This PR makes the methods in classes have consistent arguments with JSON ones.\r\n\r\n  - `UnivocityGenerator` and `JacksonGenerator` \r\n\r\n    ```scala\r\n    private[csv] class UnivocityGenerator(\r\n        schema: StructType,\r\n        writer: Writer,\r\n        options: CSVOptions = new CSVOptions(Map.empty[String, String])) {\r\n    ...\r\n\r\n    def write ...\r\n    def close ...\r\n    def flush ...\r\n    ```\r\n\r\n    ```scala\r\n    private[sql] class JacksonGenerator(\r\n       schema: StructType,\r\n       writer: Writer,\r\n       options: JSONOptions = new JSONOptions(Map.empty[String, String])) {\r\n    ...\r\n\r\n    def write ...\r\n    def close ...\r\n    def flush ...\r\n   ```\r\n\r\n  - `UnivocityParser` resembles the old version of `JacksonParser`.\r\n\r\n  - `csv.CSVInferSchema` and `json.InferSchema`\r\n\r\n    ```scala\r\n    csv.CSVInferSchema.infer(csv: RDD[String], options: CSVOptions)\r\n\r\n    json.InferSchema.infer(json: RDD[String], ... , configOptions: JSONOptions)\r\n    ```\r\n\r\n- This PR also makes the classes put in together in a consistent manner with JSON.\r\n\r\n  - `CsvFileFormat`\r\n    ```scala\r\n    CsvFileFormat\r\n    ...\r\n    CsvOutputWriter\r\n    ```\r\n\r\n  - `JsonFileFormat`\r\n\r\n    ```scala\r\n    JsonFileFormat\r\n    JsonOutputWriter\r\n    ```\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests should cover this.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/252417732","html_url":"https://github.com/apache/spark/pull/13988#issuecomment-252417732","issue_url":"https://api.github.com/repos/apache/spark/issues/13988","id":252417732,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-08T10:42:50Z","updated_at":"2016-10-08T10:42:50Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66572/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}