{"action":"created","issue":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1197","repository_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data","labels_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1197/labels{/name}","comments_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1197/comments","events_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1197/events","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/pull/1197","id":130809288,"number":1197,"title":"Introducing Raw Data tracking and Command Logs","user":{"login":"gordonje","id":4304548,"avatar_url":"https://avatars.githubusercontent.com/u/4304548?v=3","gravatar_id":"","url":"https://api.github.com/users/gordonje","html_url":"https://github.com/gordonje","followers_url":"https://api.github.com/users/gordonje/followers","following_url":"https://api.github.com/users/gordonje/following{/other_user}","gists_url":"https://api.github.com/users/gordonje/gists{/gist_id}","starred_url":"https://api.github.com/users/gordonje/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gordonje/subscriptions","organizations_url":"https://api.github.com/users/gordonje/orgs","repos_url":"https://api.github.com/users/gordonje/repos","events_url":"https://api.github.com/users/gordonje/events{/privacy}","received_events_url":"https://api.github.com/users/gordonje/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2016-02-02T20:35:01Z","updated_at":"2016-02-05T17:33:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/pulls/1197","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/pull/1197","diff_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/pull/1197.diff","patch_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/pull/1197.patch"},"body":"A lot of interrelated changes in this PR, mostly inspired by #1189:\r\n\r\n*    Introducing a scheme for tracking (in the app's database) release datetimes and sizes of CAL-ACCESS as RawDataVersions and the RawDataFiles included in each version's zip file (in the DATA/ directory)\r\n*    For each RawDataFile, store the number of records and columns at each stage of transformation: download, clean and load. More specifically:\r\n    *    `downloadcalaccessrawdata`'s prep() method inserts the RawDataFile records;\r\n    *    `cleancalaccessrawfile` looks up the RawDataFile record and sets the download_records_count and download_column_count values (since this is where we were already calculating these values)\r\n    *    `loadcalaccessrawfile` looks up the RawDataFile record and sets the clean_records_count and clean_column_count values (again, these values were already being calculated in this command)\r\n    *    `verifycalaccessrawfile` looks up the RawDataFile and sets the load_records_count and load_column_count values (again...already calculating this in this command)\r\n*    Introducing a scheme for logging start and finish datetimes for each execution of the `updatecalaccessrawdata`, `downloadcalaccessrawdata`, `cleancalaccessrawfile` and `loadcalaccessrawfile` commands\r\n    *    Commands called by other commands (e.g., update usually calls the other three) refer back to their caller, which I believe lays the foundation for smarter resuming, that is, picking up where the user left off in the middle of cleaning or loading raw files (started discussing this in one of @bcipolli's PRs, #1153)\r\n*    These new tracking models are defined in their own models .py file and do not inherit from CalAccessBaseModel, because they don't need those methods, and they are filtered out of the get_model_list() function, which is frequently used to iterate over the models that describe the actual raw data\r\n*    Introducing a new command: `reportcalaccessrawdata`, which iterates over the data models, calling `verifycalaccessrawfile` for each. It then calculates total and percentages for cleaned records and loaded records, and prints these to the console, and saves a .CSV file of each files stats to the docs/ directory.\r\n    *    Would like to do a little more work on this command before the next release. Specifically, I want to set up a template that will include other useful summary stats and will save that output, maybe also in the docs folder.\r\n\r\nCaveats:\r\n*    Executions of `updatecalaccessrawdata` where the `test-data` option is invoked will not be tracked because it isn't clear which version to assign to the tracking records. And probably we don't want these being mixed in with live data\r\n*    Relatedly, if there isn't any RawDataVersion logged in the db, the `cleancalaccessrawfile`, `loadcalaccessrawfile` and `verifycalaccessrawfile` commands will not attempt to create CalAccessCommandLogs or update RawDataFile records. This was the solution to a late-stage issue exposed by our automated tests, which create new databases and work exclusively with test data. So once again, there's no obvious RawDataVersion to assign to the tracking records.\r\n    *    This seemed a better alternative than adding a `test-data` to these three commands (similar to the one for `updatecalaccessrawdata`). I could be convinced otherwise.\r\n    *    A more ideal solution might come out future work around archiving older RawDataVersions so that users could specify which CAL-ACCESS version they want to load, verify or whatever. Maybe this could happen as part digging further into dat.\r\n\r\nOther notes:\r\n*    I also added a start_datetime and duration() method to the CalAccessCommand base class for convenient printing of durations of commands. This might want to be more tightly integrated with the tracking records\r\n*    I moved the get_download_metadata() along with the url attribute method from `downloadcalaccessrawdata` up to CalAccessCommand because at least `updatecalaccessrawdata` needs to check on the currently available raw data version, and other command's might want this as well.\r\n*    I added a command_name attribute to CalAccessCommand in order to facilitate creation of CalAccessCommandLog records without hard-coding strings for the names of the commands. I was a little surprised that something like this wasn't built-in. If someone has a more straight-forward approach, I'm all ears\r\n*    This code could definitely be more DRY than it is. It seems to me now we have two distinct kinds of commands: Those that act on one data file / model versus those that act on the entire database. We might consider sub-classing CalAccessCommand to make the distinctions more concrete. The specific need driving this for me was a desire to have a single place where we define methods for starting and stopping the CalAccessCommand log records, it isn't clear how to do that when those logs require file name value for some commands, but not others."},"comment":{"url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/comments/180457318","html_url":"https://github.com/california-civic-data-coalition/django-calaccess-raw-data/pull/1197#issuecomment-180457318","issue_url":"https://api.github.com/repos/california-civic-data-coalition/django-calaccess-raw-data/issues/1197","id":180457318,"user":{"login":"palewire","id":9993,"avatar_url":"https://avatars.githubusercontent.com/u/9993?v=3","gravatar_id":"","url":"https://api.github.com/users/palewire","html_url":"https://github.com/palewire","followers_url":"https://api.github.com/users/palewire/followers","following_url":"https://api.github.com/users/palewire/following{/other_user}","gists_url":"https://api.github.com/users/palewire/gists{/gist_id}","starred_url":"https://api.github.com/users/palewire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palewire/subscriptions","organizations_url":"https://api.github.com/users/palewire/orgs","repos_url":"https://api.github.com/users/palewire/repos","events_url":"https://api.github.com/users/palewire/events{/privacy}","received_events_url":"https://api.github.com/users/palewire/received_events","type":"User","site_admin":false},"created_at":"2016-02-05T17:33:53Z","updated_at":"2016-02-05T17:33:53Z","body":"I've been out of town and haven't been reviewing this code very closely until today.\r\n\r\nLooking at the code now, I see it aiming to support the ability to load the raw tables into separate databases with tracking tables synchronized across databases via a series of ``get_or_copy_`` methods, like [these](https://github.com/california-civic-data-coalition/django-calaccess-raw-data/blob/coverage-reports/calaccess_raw/management/commands/__init__.py#L83-L154). \r\n\r\nI wonder if that bit of complexity is necessary. What is the use case of this kind of thing and why isn't a higher-level switch without all that good enough?"}}