{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11189","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11189/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11189/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11189/events","html_url":"https://github.com/apache/spark/pull/11189","id":133358396,"number":11189,"title":"[SPARK-13080] [SQL] [WIP] Implement new Catalog API using Hive","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2016-02-12T22:26:00Z","updated_at":"2016-02-18T00:07:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11189","html_url":"https://github.com/apache/spark/pull/11189","diff_url":"https://github.com/apache/spark/pull/11189.diff","patch_url":"https://github.com/apache/spark/pull/11189.patch"},"body":"This is a step towards merging `SQLContext` and `HiveContext`. A new internal `Catalog` API was introduced in #10982 and extended in #11069. This patch introduces an implementation of this API using `HiveClient`, an existing interface to Hive. It also extends `HiveClient` with additional calls to Hive that are needed to complete the catalog implementation.\r\n\r\n**Where should I start reviewing?** The new catalog introduced is `HiveCatalog`. This class is relatively simple because it just calls `HiveClientImpl`, where most of the new logic is. I would *not* start with `HiveClient`, `HiveQl`, or `HiveMetastoreCatalog`, which are modified mainly because of a refactor.\r\n\r\n**Why is this patch so big?** I had to refactor `HiveClient` to remove an intermediate representation of databases, tables, partitions etc. After this refactor `CatalogTable` convert directly to and from `HiveTable` (etc.). Otherwise we would have to first convert `CatalogTable` to the intermediate representation and then convert that to `HiveTable`, which is messy.\r\n\r\n**The new class hierarchy is as follows:**\r\n```\r\norg.apache.spark.sql.catalyst.catalog.Catalog\r\n  - org.apache.spark.sql.catalyst.catalog.InMemoryCatalog\r\n  - org.apache.spark.sql.hive.HiveCatalog\r\n```\r\nNote that, as of this patch, none of these classes are currently used anywhere yet. This will come in the future before the Spark 2.0 release.\r\n\r\nWIP pending tests and potential cleanups."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/185468966","html_url":"https://github.com/apache/spark/pull/11189#issuecomment-185468966","issue_url":"https://api.github.com/repos/apache/spark/issues/11189","id":185468966,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-02-18T00:07:43Z","updated_at":"2016-02-18T00:07:43Z","body":"Merged build finished. Test FAILed."}}