{"action":"created","comment":{"url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/comments/233077034","pull_request_review_id":174412023,"id":233077034,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzMzA3NzAzNA==","diff_hunk":"@@ -171,26 +95,9 @@ def drop_relation(self, relation, model_name=None):\n \n         https://docs.aws.amazon.com/redshift/latest/dg/r_DROP_TABLE.html\n         \"\"\"\n-        global drop_lock\n-\n-        to_return = None\n-\n-        with drop_lock:\n-\n-            connection = self.get_connection(model_name)\n-\n-            if connection.transaction_open:\n-                self.commit(connection)\n-\n-            self.begin(connection.name)\n-\n-            to_return = super(RedshiftAdapter, self).drop_relation(\n-                relation, model_name)\n-\n-            self.commit(connection)\n-            self.begin(connection.name)\n-\n-            return to_return\n+        with self.connections.fresh_transaction(model_name):","path":"dbt/adapters/redshift/impl.py","position":146,"original_position":146,"commit_id":"9bee0190d2adbac8b20b4e30d59bcec8e40710ae","original_commit_id":"9bee0190d2adbac8b20b4e30d59bcec8e40710ae","user":{"login":"beckjake","id":452302,"node_id":"MDQ6VXNlcjQ1MjMwMg==","avatar_url":"https://avatars2.githubusercontent.com/u/452302?v=4","gravatar_id":"","url":"https://api.github.com/users/beckjake","html_url":"https://github.com/beckjake","followers_url":"https://api.github.com/users/beckjake/followers","following_url":"https://api.github.com/users/beckjake/following{/other_user}","gists_url":"https://api.github.com/users/beckjake/gists{/gist_id}","starred_url":"https://api.github.com/users/beckjake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beckjake/subscriptions","organizations_url":"https://api.github.com/users/beckjake/orgs","repos_url":"https://api.github.com/users/beckjake/repos","events_url":"https://api.github.com/users/beckjake/events{/privacy}","received_events_url":"https://api.github.com/users/beckjake/received_events","type":"User","site_admin":false},"body":"This is functionally the same, I just extracted the transaction management into a context manager. It's not getting a new connection unless the named connection never existed. That context manager just triggers a (maybe commit) + begin, yields, and triggers a commit + begin again. Hence `fresh_transaction`.\r\n\r\nI saw #697 and decided I'm not qualified on that front.","created_at":"2018-11-13T15:11:18Z","updated_at":"2018-11-13T15:11:18Z","html_url":"https://github.com/fishtown-analytics/dbt/pull/1109#discussion_r233077034","pull_request_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/comments/233077034"},"html":{"href":"https://github.com/fishtown-analytics/dbt/pull/1109#discussion_r233077034"},"pull_request":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109"}},"in_reply_to_id":232886607},"pull_request":{"url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109","id":228418444,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI4NDE4NDQ0","html_url":"https://github.com/fishtown-analytics/dbt/pull/1109","diff_url":"https://github.com/fishtown-analytics/dbt/pull/1109.diff","patch_url":"https://github.com/fishtown-analytics/dbt/pull/1109.patch","issue_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/1109","number":1109,"state":"open","locked":false,"title":"Feature/adapter refactor (#962, #963, #965, #1035, #1090)","user":{"login":"beckjake","id":452302,"node_id":"MDQ6VXNlcjQ1MjMwMg==","avatar_url":"https://avatars2.githubusercontent.com/u/452302?v=4","gravatar_id":"","url":"https://api.github.com/users/beckjake","html_url":"https://github.com/beckjake","followers_url":"https://api.github.com/users/beckjake/followers","following_url":"https://api.github.com/users/beckjake/following{/other_user}","gists_url":"https://api.github.com/users/beckjake/gists{/gist_id}","starred_url":"https://api.github.com/users/beckjake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beckjake/subscriptions","organizations_url":"https://api.github.com/users/beckjake/orgs","repos_url":"https://api.github.com/users/beckjake/repos","events_url":"https://api.github.com/users/beckjake/events{/privacy}","received_events_url":"https://api.github.com/users/beckjake/received_events","type":"User","site_admin":false},"body":"fixes #963, fixes #1035, fixes #962, fixes #965, fixes #1090\r\n\r\nThis merges PRs #1064 and #1091 into, I have combined them as a single description below.\r\n\r\n### User-facing changes:\r\n - Remove deprecated methods\r\n - Connections are now managed on a per-adapter-instance basis. As adapters are generally singletons, that should be no different than before. This means that adapters have full control over their connection management,  which seems right.\r\n - All adapters are now optional, and installers have to explicitly opt-in with the [feature] syntax. \r\n - Adapter packages load lazily, when the configuration's credentials object is parsed.\r\n\r\n### Internal changes\r\n - Created a more consistent class hierarchy where bigquery adapter does not derive from postgres, snowflake and postgres have a common ancestor, etc\r\n - Defined what methods subclasses must override, use python's abstract classes to enforce them\r\n - Gave the DBWrapper its list of methods to expose via a decorator + metaclass model because I broke those arrays like 20 times during development and wow, what a pain to maintain with subclassing.\r\n - Split adapters into two components: A connection manager that doesn't understand relations/the manifest/anything about dbt, and the \"real\" adapter that does.\r\n- `dbt.adapters.factory.ADAPTER_CLASSES` now starts off empty\r\n-  When the chosen target references an adapter `typename`, dbt imports `dbt.adapters.typename` as `pkg`, registers `pkg.Adapter` for the type, and returns `pkg.Credentials` for validation. So packages need to expose their adapters/credentials under those names.\r\n- Accordingly, contracts are validated at target parsing time, which occurs after regular configuration parsing\r\n- Converting dbt internal schema representations into bigquery schema representations now happens in the bigquery adapter itself, purely for dependency reasons.\r\n- adapter-specific Connection contracts are now part of the adapter packages themselves, instead of part of dbt.contracts.connection\r\n\r\n### Test changes\r\nI changed the tox environments (and so also the CI tests) to use the new `-e .[...]` syntax. Things seem to work.\r\n\r\n### User impact\r\n\r\nTo get the existing dbt installation behavior, either continue to use `-r requirements.txt` or use `-e .[postgres,snowflake,redshift,bigquery]`. When this is on pypi, users will have to `pip install dbt[postgres,snowflake,redshift,bigquery]` to get the same behavior, which is unfortunate. Python does not really provide an opt-out mechanism for dependencies, only opt-in, and has no concept of a \"default\" features set.\r\n\r\nIf that is scary from a backwards-compatibility perspective, one gross idea I had was to make this something like `dbt-core`, and make a `dbt` package that depends upon `dbt-core[postgres,snowflake,redshift,bigquery]`.\r\n\r\n#### Assorted thoughts\r\n\r\nWe also may want to add a deprecation warning to 0.12.x for `get_columns_in_table`, and then we could change this branch to remove it entirely. I've replaced it with `get_columns_in_relation`, which takes a `Relation`, and `get_columns_in_table` builds one and passes it in (like `drop`/`rename`/`truncate` did).\r\n\r\nIt wouldn't be too hard to just assign one connection per thread, as it is connections and threads are related (connection count is directly related to thread count) but disjoint (good luck mapping connections to threads or vice-versa). We could force that, by making the thread pool responsible for connections, so each thread gets one connection at start time. Then we would never have to track the existence of connections, just their state.\r\n\r\nOne thing this PR does not solve is the adapter-specific macros issue. Those are still included in all dbt runs all the time, and exist regardless of `[feature]` settings.\r\n\r\nThe connection management does leave me with this vague feeling that I should be able to do something like this:\r\n\r\n```\r\nwith adapter.transaction() as conn:\r\n   # connection created or reopened, begin called on __enter__\r\n   conn.execute(...) # or add_query(), whatever\r\n# commit/rollback + close called as appropriate on __exit__\r\n# connection is now back in the pool, in a closed state\r\n```\r\n\r\nAs it is I'm never really confident that all connections have been closed properly.","created_at":"2018-11-05T16:47:17Z","updated_at":"2018-11-13T15:11:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9715e942ba493ea0f36ed2741f4181490634fa2f","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109/commits","review_comments_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109/comments","review_comment_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/comments{/number}","comments_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/1109/comments","statuses_url":"https://api.github.com/repos/fishtown-analytics/dbt/statuses/9bee0190d2adbac8b20b4e30d59bcec8e40710ae","head":{"label":"fishtown-analytics:feature/adapter-refactor","ref":"feature/adapter-refactor","sha":"9bee0190d2adbac8b20b4e30d59bcec8e40710ae","user":{"login":"fishtown-analytics","id":18339788,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MzM5Nzg4","avatar_url":"https://avatars1.githubusercontent.com/u/18339788?v=4","gravatar_id":"","url":"https://api.github.com/users/fishtown-analytics","html_url":"https://github.com/fishtown-analytics","followers_url":"https://api.github.com/users/fishtown-analytics/followers","following_url":"https://api.github.com/users/fishtown-analytics/following{/other_user}","gists_url":"https://api.github.com/users/fishtown-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/fishtown-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fishtown-analytics/subscriptions","organizations_url":"https://api.github.com/users/fishtown-analytics/orgs","repos_url":"https://api.github.com/users/fishtown-analytics/repos","events_url":"https://api.github.com/users/fishtown-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/fishtown-analytics/received_events","type":"Organization","site_admin":false},"repo":{"id":53548867,"node_id":"MDEwOlJlcG9zaXRvcnk1MzU0ODg2Nw==","name":"dbt","full_name":"fishtown-analytics/dbt","private":false,"owner":{"login":"fishtown-analytics","id":18339788,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MzM5Nzg4","avatar_url":"https://avatars1.githubusercontent.com/u/18339788?v=4","gravatar_id":"","url":"https://api.github.com/users/fishtown-analytics","html_url":"https://github.com/fishtown-analytics","followers_url":"https://api.github.com/users/fishtown-analytics/followers","following_url":"https://api.github.com/users/fishtown-analytics/following{/other_user}","gists_url":"https://api.github.com/users/fishtown-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/fishtown-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fishtown-analytics/subscriptions","organizations_url":"https://api.github.com/users/fishtown-analytics/orgs","repos_url":"https://api.github.com/users/fishtown-analytics/repos","events_url":"https://api.github.com/users/fishtown-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/fishtown-analytics/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/fishtown-analytics/dbt","description":"dbt (data build tool) is a command line tool that enables data analysts and engineers to transform data in their warehouse more effectively.","fork":false,"url":"https://api.github.com/repos/fishtown-analytics/dbt","forks_url":"https://api.github.com/repos/fishtown-analytics/dbt/forks","keys_url":"https://api.github.com/repos/fishtown-analytics/dbt/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fishtown-analytics/dbt/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fishtown-analytics/dbt/teams","hooks_url":"https://api.github.com/repos/fishtown-analytics/dbt/hooks","issue_events_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/events{/number}","events_url":"https://api.github.com/repos/fishtown-analytics/dbt/events","assignees_url":"https://api.github.com/repos/fishtown-analytics/dbt/assignees{/user}","branches_url":"https://api.github.com/repos/fishtown-analytics/dbt/branches{/branch}","tags_url":"https://api.github.com/repos/fishtown-analytics/dbt/tags","blobs_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/refs{/sha}","trees_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fishtown-analytics/dbt/statuses/{sha}","languages_url":"https://api.github.com/repos/fishtown-analytics/dbt/languages","stargazers_url":"https://api.github.com/repos/fishtown-analytics/dbt/stargazers","contributors_url":"https://api.github.com/repos/fishtown-analytics/dbt/contributors","subscribers_url":"https://api.github.com/repos/fishtown-analytics/dbt/subscribers","subscription_url":"https://api.github.com/repos/fishtown-analytics/dbt/subscription","commits_url":"https://api.github.com/repos/fishtown-analytics/dbt/commits{/sha}","git_commits_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/commits{/sha}","comments_url":"https://api.github.com/repos/fishtown-analytics/dbt/comments{/number}","issue_comment_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/comments{/number}","contents_url":"https://api.github.com/repos/fishtown-analytics/dbt/contents/{+path}","compare_url":"https://api.github.com/repos/fishtown-analytics/dbt/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fishtown-analytics/dbt/merges","archive_url":"https://api.github.com/repos/fishtown-analytics/dbt/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fishtown-analytics/dbt/downloads","issues_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues{/number}","pulls_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls{/number}","milestones_url":"https://api.github.com/repos/fishtown-analytics/dbt/milestones{/number}","notifications_url":"https://api.github.com/repos/fishtown-analytics/dbt/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fishtown-analytics/dbt/labels{/name}","releases_url":"https://api.github.com/repos/fishtown-analytics/dbt/releases{/id}","deployments_url":"https://api.github.com/repos/fishtown-analytics/dbt/deployments","created_at":"2016-03-10T02:38:00Z","updated_at":"2018-11-12T20:50:04Z","pushed_at":"2018-11-13T15:02:31Z","git_url":"git://github.com/fishtown-analytics/dbt.git","ssh_url":"git@github.com:fishtown-analytics/dbt.git","clone_url":"https://github.com/fishtown-analytics/dbt.git","svn_url":"https://github.com/fishtown-analytics/dbt","homepage":"https://www.getdbt.com/","size":5419,"stargazers_count":477,"watchers_count":477,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":73,"mirror_url":null,"archived":false,"open_issues_count":141,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":73,"open_issues":141,"watchers":477,"default_branch":"dev/guion-bluford"}},"base":{"label":"fishtown-analytics:dev/stephen-girard","ref":"dev/stephen-girard","sha":"e716db7983790cd9acbbc6e66d123e76f665fca4","user":{"login":"fishtown-analytics","id":18339788,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MzM5Nzg4","avatar_url":"https://avatars1.githubusercontent.com/u/18339788?v=4","gravatar_id":"","url":"https://api.github.com/users/fishtown-analytics","html_url":"https://github.com/fishtown-analytics","followers_url":"https://api.github.com/users/fishtown-analytics/followers","following_url":"https://api.github.com/users/fishtown-analytics/following{/other_user}","gists_url":"https://api.github.com/users/fishtown-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/fishtown-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fishtown-analytics/subscriptions","organizations_url":"https://api.github.com/users/fishtown-analytics/orgs","repos_url":"https://api.github.com/users/fishtown-analytics/repos","events_url":"https://api.github.com/users/fishtown-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/fishtown-analytics/received_events","type":"Organization","site_admin":false},"repo":{"id":53548867,"node_id":"MDEwOlJlcG9zaXRvcnk1MzU0ODg2Nw==","name":"dbt","full_name":"fishtown-analytics/dbt","private":false,"owner":{"login":"fishtown-analytics","id":18339788,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE4MzM5Nzg4","avatar_url":"https://avatars1.githubusercontent.com/u/18339788?v=4","gravatar_id":"","url":"https://api.github.com/users/fishtown-analytics","html_url":"https://github.com/fishtown-analytics","followers_url":"https://api.github.com/users/fishtown-analytics/followers","following_url":"https://api.github.com/users/fishtown-analytics/following{/other_user}","gists_url":"https://api.github.com/users/fishtown-analytics/gists{/gist_id}","starred_url":"https://api.github.com/users/fishtown-analytics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fishtown-analytics/subscriptions","organizations_url":"https://api.github.com/users/fishtown-analytics/orgs","repos_url":"https://api.github.com/users/fishtown-analytics/repos","events_url":"https://api.github.com/users/fishtown-analytics/events{/privacy}","received_events_url":"https://api.github.com/users/fishtown-analytics/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/fishtown-analytics/dbt","description":"dbt (data build tool) is a command line tool that enables data analysts and engineers to transform data in their warehouse more effectively.","fork":false,"url":"https://api.github.com/repos/fishtown-analytics/dbt","forks_url":"https://api.github.com/repos/fishtown-analytics/dbt/forks","keys_url":"https://api.github.com/repos/fishtown-analytics/dbt/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fishtown-analytics/dbt/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fishtown-analytics/dbt/teams","hooks_url":"https://api.github.com/repos/fishtown-analytics/dbt/hooks","issue_events_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/events{/number}","events_url":"https://api.github.com/repos/fishtown-analytics/dbt/events","assignees_url":"https://api.github.com/repos/fishtown-analytics/dbt/assignees{/user}","branches_url":"https://api.github.com/repos/fishtown-analytics/dbt/branches{/branch}","tags_url":"https://api.github.com/repos/fishtown-analytics/dbt/tags","blobs_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/refs{/sha}","trees_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fishtown-analytics/dbt/statuses/{sha}","languages_url":"https://api.github.com/repos/fishtown-analytics/dbt/languages","stargazers_url":"https://api.github.com/repos/fishtown-analytics/dbt/stargazers","contributors_url":"https://api.github.com/repos/fishtown-analytics/dbt/contributors","subscribers_url":"https://api.github.com/repos/fishtown-analytics/dbt/subscribers","subscription_url":"https://api.github.com/repos/fishtown-analytics/dbt/subscription","commits_url":"https://api.github.com/repos/fishtown-analytics/dbt/commits{/sha}","git_commits_url":"https://api.github.com/repos/fishtown-analytics/dbt/git/commits{/sha}","comments_url":"https://api.github.com/repos/fishtown-analytics/dbt/comments{/number}","issue_comment_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues/comments{/number}","contents_url":"https://api.github.com/repos/fishtown-analytics/dbt/contents/{+path}","compare_url":"https://api.github.com/repos/fishtown-analytics/dbt/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fishtown-analytics/dbt/merges","archive_url":"https://api.github.com/repos/fishtown-analytics/dbt/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fishtown-analytics/dbt/downloads","issues_url":"https://api.github.com/repos/fishtown-analytics/dbt/issues{/number}","pulls_url":"https://api.github.com/repos/fishtown-analytics/dbt/pulls{/number}","milestones_url":"https://api.github.com/repos/fishtown-analytics/dbt/milestones{/number}","notifications_url":"https://api.github.com/repos/fishtown-analytics/dbt/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fishtown-analytics/dbt/labels{/name}","releases_url":"https://api.github.com/repos/fishtown-analytics/dbt/releases{/id}","deployments_url":"https://api.github.com/repos/fishtown-analytics/dbt/deployments","created_at":"2016-03-10T02:38:00Z","updated_at":"2018-11-12T20:50:04Z","pushed_at":"2018-11-13T15:02:31Z","git_url":"git://github.com/fishtown-analytics/dbt.git","ssh_url":"git@github.com:fishtown-analytics/dbt.git","clone_url":"https://github.com/fishtown-analytics/dbt.git","svn_url":"https://github.com/fishtown-analytics/dbt","homepage":"https://www.getdbt.com/","size":5419,"stargazers_count":477,"watchers_count":477,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":73,"mirror_url":null,"archived":false,"open_issues_count":141,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":73,"open_issues":141,"watchers":477,"default_branch":"dev/guion-bluford"}},"_links":{"self":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109"},"html":{"href":"https://github.com/fishtown-analytics/dbt/pull/1109"},"issue":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/issues/1109"},"comments":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/issues/1109/comments"},"review_comments":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109/comments"},"review_comment":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/pulls/1109/commits"},"statuses":{"href":"https://api.github.com/repos/fishtown-analytics/dbt/statuses/9bee0190d2adbac8b20b4e30d59bcec8e40710ae"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"beckjake"}}