{"action":"opened","issue":{"url":"https://api.github.com/repos/analyst-collective/dbt/issues/42","repository_url":"https://api.github.com/repos/analyst-collective/dbt","labels_url":"https://api.github.com/repos/analyst-collective/dbt/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/analyst-collective/dbt/issues/42/comments","events_url":"https://api.github.com/repos/analyst-collective/dbt/issues/42/events","html_url":"https://github.com/analyst-collective/dbt/issues/42","id":159190351,"number":42,"title":"dbt should have the ability to seed tables from csvs","user":{"login":"jthandy","id":2341841,"avatar_url":"https://avatars.githubusercontent.com/u/2341841?v=3","gravatar_id":"","url":"https://api.github.com/users/jthandy","html_url":"https://github.com/jthandy","followers_url":"https://api.github.com/users/jthandy/followers","following_url":"https://api.github.com/users/jthandy/following{/other_user}","gists_url":"https://api.github.com/users/jthandy/gists{/gist_id}","starred_url":"https://api.github.com/users/jthandy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jthandy/subscriptions","organizations_url":"https://api.github.com/users/jthandy/orgs","repos_url":"https://api.github.com/users/jthandy/repos","events_url":"https://api.github.com/users/jthandy/events{/privacy}","received_events_url":"https://api.github.com/users/jthandy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-08T15:15:35Z","updated_at":"2016-06-08T15:15:35Z","closed_at":null,"body":"there are a bunch of types of data that are needed around the edges of any specific analysis. mapping one entity to another, tables for exceptions to ignore, etc. all of this data needs to be in the warehouse somewhere so that it can be incorporated into analytic queries.\r\n\r\ncurrently this process is being facilitated via super-long union all queries that encode the data within the structure of a view. this is not a good solution for performance and maintenance reasons; it's also just weird/awkward. instead, there should be CSVs that are checked into a /data directory in the dbt folder that get loaded by dbt. \r\n\r\npotentially the command should be `dbt seed`.\r\n\r\nother thoughts: \r\n- there needs to be a file that defines the schema for each table; otherwise it would have to be inferred from the csv and that's more work than we need to do right now.\r\n- it's far easier to truncate and reload these tables than drop and rebuild tables because it doesn't require cascade-dropping all of the dependencies on the table. we probably need a separate `dbt seed --create-tables` vs. just `dbt seed`, where simply calling seed truncates and reloads whereas `--create-tables` drops any existing tables and recreates from the definition files.\r\n- loading should probably be done via insert statements. while a redshift COPY command would be more performant, it also would force us to go outside of SQLAlchemy and would not be compatible across multiple eventual endpoints. performance is not a concern because we the purpose of this function is to load very small numbers of rows (single-digit thousands should be considered high for this use case)."}}
{"actor":{"display_login":"jthandy"}}