{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14155","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14155/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14155/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14155/events","html_url":"https://github.com/apache/spark/pull/14155","id":165090204,"number":14155,"title":"[SPARK-16498][SQL] move hive hack for data source table into HiveExternalCatalog","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":33,"created_at":"2016-07-12T14:11:12Z","updated_at":"2016-08-20T05:02:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14155","html_url":"https://github.com/apache/spark/pull/14155","diff_url":"https://github.com/apache/spark/pull/14155.diff","patch_url":"https://github.com/apache/spark/pull/14155.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSpark SQL doesn't have its own meta store yet, and use hive's currently. However, hive's meta store has some limitations(e.g. columns can't be too many, not case-preserving, bad decimal type support, etc.), so we have some hacks to successfully store data source table metadata into hive meta store, i.e. put all the information in table properties.\r\n\r\nThis PR moves these hacks to `HiveExternalCatalog`, tries to isolate hive specific logic in one place.\r\n\r\nchanges overview:\r\n\r\n1.  **before this PR**: we need to put metadata(schema, partition columns, etc.) of data source tables to table properties before saving it to external catalog, even the external catalog doesn't use hive metastore(e.g. `InMemoryCatalog`)\r\n**after this PR**: the table properties tricks are only in `HiveExternalCatalog`, the caller side doesn't need to take care of it anymore.\r\n\r\n2. **before this PR**: because the table properties tricks are done outside of external catalog, so we also need to revert these tricks when we read the table metadata from external catalog and use it. e.g. in `DescribeTableCommand` we will read schema and partition columns from table properties.\r\n**after this PR**: The table metadata read from external catalog is exactly the same with what we saved to it.\r\n\r\nbonus: now we can create data source table using `SessionCatalog`, if schema is specified.\r\nbreaks: `schemaStringLengthThreshold` is not configurable anymore. `hive.default.rcfile.serde` is not configurable anymore.\r\n\r\n## How was this patch tested?\r\n\r\nexisting tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/241179951","html_url":"https://github.com/apache/spark/pull/14155#issuecomment-241179951","issue_url":"https://api.github.com/repos/apache/spark/issues/14155","id":241179951,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-08-20T05:02:38Z","updated_at":"2016-08-20T05:02:38Z","body":"**[Test build #64122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64122/consoleFull)** for PR 14155 at commit [`38b838a`](https://github.com/apache/spark/commit/38b838a9d27d5e11bad5f5e7040fe2d6d2e56216)."}}
{"actor":{"display_login":"SparkQA"}}