{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20208","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20208/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20208/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20208/events","html_url":"https://github.com/apache/spark/pull/20208","id":287187776,"node_id":"MDExOlB1bGxSZXF1ZXN0MTYxOTQ1MTQ1","number":20208,"title":"[SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources","user":{"login":"dongjoon-hyun","id":9700541,"node_id":"MDQ6VXNlcjk3MDA1NDE=","avatar_url":"https://avatars2.githubusercontent.com/u/9700541?v=4","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":68,"created_at":"2018-01-09T18:35:43Z","updated_at":"2018-07-12T21:08:23Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20208","html_url":"https://github.com/apache/spark/pull/20208","diff_url":"https://github.com/apache/spark/pull/20208.diff","patch_url":"https://github.com/apache/spark/pull/20208.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThe reader schema is said to be evolved (or projected) when it changed after the data is written. The followings are already supported in file-based data sources. Note that partition columns are not maintained in files. In this PR, `column` means `non-partition column`.\r\n\r\n   1. Add a column\r\n   2. Hide a column\r\n   3. Change a column position \r\n   4. Change a column type (upcast)\r\n\r\nThis issue aims to guarantee users a backward-compatible read-schema test coverage on file-based data sources and to prevent future regressions by *adding read schema tests explicitly*.\r\n\r\nHere, we consider safe changes without data loss. For example, data type change should be from small types to larger types like `int`-to-`long`, not vice versa.\r\n\r\nAs of today, in the master branch, file-based data sources have the following coverage.\r\n\r\nFile Format | Coverage  | Note\r\n----------- | ---------- | ------------------------------------------------\r\nTEXT          | N/A            | Schema consists of a single string column.\r\nCSV            | 1, 2, 4        |\r\nJSON          | 1, 2, 3, 4    |\r\nORC            | 1, 2, 3, 4    | Native vectorized ORC reader has the widest coverage among ORC formats.\r\nPARQUET   | 1, 2, 3        |\r\n\r\n\r\n## How was this patch tested?\r\n\r\nPass the Jenkins with newly added test suites."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/404651358","html_url":"https://github.com/apache/spark/pull/20208#issuecomment-404651358","issue_url":"https://api.github.com/repos/apache/spark/issues/20208","id":404651358,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDY1MTM1OA==","user":{"login":"gatorsmile","id":11567269,"node_id":"MDQ6VXNlcjExNTY3MjY5","avatar_url":"https://avatars1.githubusercontent.com/u/11567269?v=4","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"created_at":"2018-07-12T21:08:23Z","updated_at":"2018-07-12T21:08:23Z","author_association":"MEMBER","body":"@dongjoon-hyun This PR is to improve the test coverage. LGTM. \r\n\r\nWhen the schema do not match with the schemas of underlying data source, the current error messages might be weird. This is a common issue, I think. Please submit a separate PR to improve the error handling in these cases? "}}
{"actor":{"display_login":"gatorsmile"}}