{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13912","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13912/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13912/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13912/events","html_url":"https://github.com/apache/spark/pull/13912","id":162314680,"number":13912,"title":"[SPARK-16216][SQL] CSV data source does not write date and timestamp correctly","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-06-26T06:18:05Z","updated_at":"2016-06-26T06:36:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13912","html_url":"https://github.com/apache/spark/pull/13912","diff_url":"https://github.com/apache/spark/pull/13912.diff","patch_url":"https://github.com/apache/spark/pull/13912.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR corrects CSV data sources in order to write `DateType` and `TimestampType` correctly just like JSON data source.\r\n\r\nCurrently this writes the data as below:\r\n\r\n```\r\n+----------------+\r\n|            date|\r\n+----------------+\r\n|1440637200000000|\r\n|1414459800000000|\r\n|1454040000000000|\r\n+----------------+\r\n```\r\n\r\nThis PR make this write dates and timestamps in a formatted string as below:\r\n\r\n- **TimestampType**\r\n  - With `dateFormat` option (e.g. `dd/MM/yyyy HH:mm`)\r\n    ```\r\n    +----------------+\r\n    |            date|\r\n    +----------------+\r\n    |2015/08/26 18:00|\r\n    |2014/10/27 18:30|\r\n    |2016/01/28 20:00|\r\n    +----------------+\r\n    ```\r\n\r\n  - Without `dateFormat` option\r\n    ```\r\n    +---------------------+\r\n    |                 date|\r\n    +---------------------+\r\n    |2015-08-26 18:00:00.0|\r\n    |2014-10-27 18:30:00.0|\r\n    |2016-01-28 20:00:00.0|\r\n    +---------------------+\r\n    ```\r\n\r\n- **DateType**\r\n  - With `dateFormat` option (e.g. `yyyy/MM/dd`)\r\n    ```\r\n    +----------+\r\n    |      date|\r\n    +----------+\r\n    |2015/08/26|\r\n    |2014/10/27|\r\n    |2016/01/28|\r\n    +----------+\r\n    ```\r\n\r\n  - Without `dateFormat` option\r\n    ```\r\n    +----------+\r\n    |      date|\r\n    +----------+\r\n    |2015-08-26|\r\n    |2014-10-27|\r\n    |2016-01-28|\r\n    +----------+\r\n    ```\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests in `CsvSuite.scala`.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/228586981","html_url":"https://github.com/apache/spark/pull/13912#issuecomment-228586981","issue_url":"https://api.github.com/repos/apache/spark/issues/13912","id":228586981,"user":{"login":"srowen","id":822522,"avatar_url":"https://avatars.githubusercontent.com/u/822522?v=3","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"created_at":"2016-06-26T06:36:20Z","updated_at":"2016-06-26T06:36:20Z","body":"This has a big problem though -- the times are now wrong, because there is no timezone.\r\nWhy would we write formatted dates in a data file though? it's for human consumption. Timestamp is an appropriate way to render a point in time."}}
{"actor":{"display_login":"srowen"}}