{"action":"opened","issue":{"url":"https://api.github.com/repos/18F/openFEC/issues/1515","repository_url":"https://api.github.com/repos/18F/openFEC","labels_url":"https://api.github.com/repos/18F/openFEC/issues/1515/labels{/name}","comments_url":"https://api.github.com/repos/18F/openFEC/issues/1515/comments","events_url":"https://api.github.com/repos/18F/openFEC/issues/1515/events","html_url":"https://github.com/18F/openFEC/issues/1515","id":133456730,"number":1515,"title":"Explicitly track dependencies between data views","user":{"login":"jmcarp","id":1633460,"avatar_url":"https://avatars.githubusercontent.com/u/1633460?v=3","gravatar_id":"","url":"https://api.github.com/users/jmcarp","html_url":"https://github.com/jmcarp","followers_url":"https://api.github.com/users/jmcarp/followers","following_url":"https://api.github.com/users/jmcarp/following{/other_user}","gists_url":"https://api.github.com/users/jmcarp/gists{/gist_id}","starred_url":"https://api.github.com/users/jmcarp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmcarp/subscriptions","organizations_url":"https://api.github.com/users/jmcarp/orgs","repos_url":"https://api.github.com/users/jmcarp/repos","events_url":"https://api.github.com/users/jmcarp/events{/privacy}","received_events_url":"https://api.github.com/users/jmcarp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-13T18:28:43Z","updated_at":"2016-02-13T18:28:43Z","closed_at":null,"body":"Some of our data views are built directly from the raw FEC tables, but others depend on a chain of our own views. For example, the candidate full-text search view (ofec_candidate_fulltext_mv) depends on the candidate detail view (ofec_candidate_detail_mv). If the views are created in the wrong order, the process will crash. If the views are updated in the wrong order, some views may be stale.\r\n\r\nAt the moment, we're using the very brittle approach of running sql files in alphabetical order by file name. Instead, I propose that we explicitly declare dependencies between data import and refresh steps and ensure that all steps run in an appropriate order. Airbnb's airflow library is designed to solve exactly this kind of problem and might be a good candidate--or if that's overkill, we can build a directed graph of view dependencies and run steps in topologically sorted order."}}