{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14365","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14365/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14365/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14365/events","html_url":"https://github.com/apache/spark/pull/14365","id":167550400,"number":14365,"title":"[SPARK-16628][SQL] Translate file-based relation schema when file schema is inconsistent with catalog schema","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2016-07-26T08:36:49Z","updated_at":"2016-12-08T10:09:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14365","html_url":"https://github.com/apache/spark/pull/14365","diff_url":"https://github.com/apache/spark/pull/14365.diff","patch_url":"https://github.com/apache/spark/pull/14365.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWe will convert Metastore Orc tables (represented by `MetastoreRelation`) to datasource tables (represented by `HadoopFsRelation`) if `spark.sql.hive.convertMetastoreOrc` is enabled for better performance.\r\n\r\nHowever, due to a Hive issue, an Orc table created by Hive does not store column name correctly in the Orc files. For these Orc tables, the converted relation has wrong schema.\r\n\r\nTo fix this, we assume the metastore schema `dataSchema` can match to `physicalSchema` by each column disregarding the column names. If not, we throw an exception that suggests users to disable the conversion of Hive Orc tables.\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/265702784","html_url":"https://github.com/apache/spark/pull/14365#issuecomment-265702784","issue_url":"https://api.github.com/repos/apache/spark/issues/14365","id":265702784,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T10:09:16Z","updated_at":"2016-12-08T10:09:16Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}