{"action":"created","comment":{"url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments/186008854","pull_request_review_id":117514961,"id":186008854,"diff_hunk":"@@ -104,3 +104,6 @@ venv.bak/\n \n # mypy\n .mypy_cache/\n+\n+# Local development\n+local.yaml","path":".gitignore","position":null,"original_position":6,"commit_id":"bfc9bc8ebda9b54f79ee1341a86b227d3b91fb67","original_commit_id":"1db9f1c3182cd8eaeeac1a4725f72d3894304eca","user":{"login":"jessex","id":431895,"avatar_url":"https://avatars1.githubusercontent.com/u/431895?v=4","gravatar_id":"","url":"https://api.github.com/users/jessex","html_url":"https://github.com/jessex","followers_url":"https://api.github.com/users/jessex/followers","following_url":"https://api.github.com/users/jessex/following{/other_user}","gists_url":"https://api.github.com/users/jessex/gists{/gist_id}","starred_url":"https://api.github.com/users/jessex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jessex/subscriptions","organizations_url":"https://api.github.com/users/jessex/orgs","repos_url":"https://api.github.com/users/jessex/repos","events_url":"https://api.github.com/users/jessex/events{/privacy}","received_events_url":"https://api.github.com/users/jessex/received_events","type":"User","site_admin":false},"body":"Check out #73 for a proposal along these lines.","created_at":"2018-05-04T07:44:31Z","updated_at":"2018-05-04T07:44:31Z","html_url":"https://github.com/Recidiviz/recidiviz-data/pull/46#discussion_r186008854","pull_request_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments/186008854"},"html":{"href":"https://github.com/Recidiviz/recidiviz-data/pull/46#discussion_r186008854"},"pull_request":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46"}},"in_reply_to_id":183368540},"pull_request":{"url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46","id":167521738,"html_url":"https://github.com/Recidiviz/recidiviz-data/pull/46","diff_url":"https://github.com/Recidiviz/recidiviz-data/pull/46.diff","patch_url":"https://github.com/Recidiviz/recidiviz-data/pull/46.patch","issue_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/46","number":46,"state":"closed","locked":false,"title":"Snapshot Scraping","user":{"login":"cawarren","id":59526,"avatar_url":"https://avatars2.githubusercontent.com/u/59526?v=4","gravatar_id":"","url":"https://api.github.com/users/cawarren","html_url":"https://github.com/cawarren","followers_url":"https://api.github.com/users/cawarren/followers","following_url":"https://api.github.com/users/cawarren/following{/other_user}","gists_url":"https://api.github.com/users/cawarren/gists{/gist_id}","starred_url":"https://api.github.com/users/cawarren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cawarren/subscriptions","organizations_url":"https://api.github.com/users/cawarren/orgs","repos_url":"https://api.github.com/users/cawarren/repos","events_url":"https://api.github.com/users/cawarren/events{/privacy}","received_events_url":"https://api.github.com/users/cawarren/received_events","type":"User","site_admin":false},"body":"This PR:\r\n- Adds a second type of scraping activity, *snapshot scraping.* Snapshot scraping targets only those records relevant to the timeframe we look at for recidivism calculations, and creates snapshots if there are any new changes to these records.\r\n- Adds all mutable `Record` and `UsNyRecord` fields to the `InmateSnapshot` and `UsNyInmateSnapshot` models for more detailed tracking of changes to inmate records\r\n- Migrates current `InmateFacilitySnapshot` entities into `UsNyInmateSnapshot` entities\r\n- Migrates the `last_release_date` and `last_release_type` fields from `UsNyRecord` to `latest_release_date` and `latest_release_type` fields in the superclass `Record`, as these are needed more generally for snapshot scrapes\r\n- Fixes issue 28\r\n\r\nChanges to this PR, prior to merge:\r\n- Remove test database request handler (provided as aid to testing changes)\r\n\r\nFollow-ups to this PR:\r\n- Remove data migration request handler and `deferred` builtin support\r\n- Remove `InmateFacilitySnapshot` model \r\n- Add snapshot scraping cron jobs at a reasonable frequency, and a time of day that won't conflict with background scraping.\r\n\r\n**Overview of changes**\r\n\r\n**Scraper control** has been consolidated into `queue_control.py`, centralizing the start/stop/resumption of scrapers. The `recidiviz.py` file that used to contain start logic for scrapers no longer exists.\r\n\r\n**Snapshot Scraping** For newly started scrapes, the `queue_control` module generates a list of targets - names for background scrapes, and inmates for the snapshot scrapes (we focus on inmates instead of records, because we want to find any new records we don't yet know about for the inmates who have records relevant to calculator). For snapshot scrapes, we also generate an 'ignore list' - a set of records that the scraper shouldn't bother scraping, because they're known and too old to be relevant to our metrics.\r\n\r\nFor either scrape type, the resulting target lists are dumped into the datastore as `DocketItem`s for that particular scrape type. The scraper logic now picks up an arbitrary `DocketItem` for its scrape_type, scrapes that item, then deletes the processed `DocketItem` and selects a new item from the docket. The scrape is complete when no `DocketItems` for the selected scrape type remain.\r\n\r\n**Migration** The `data_migration` module supports two migration types: `record` and `snapshot`. See the *This PR* section above for the specific changes each migration type makes. \r\n\r\nThe migration tool is destructive - it handles both migration and deletion of the data that's been migrated. It supports a `test_only` URL flag, which will limit the number of entities migrated to 5 and migrate non-destructively.\r\n\r\nExample queries are included in comments at the top of the `data_migration` module.\r\n\r\n**Fake data population** The `populate_test_db` module is provided for simpler population of environment variables and fake data, for testing both the data migration and the updated snapshot and background scrape behavior.\r\n\r\nExample queries for different combinations of fake data population are provided in comments at the top of the module.","created_at":"2018-02-06T20:29:47Z","updated_at":"2018-05-04T07:44:31Z","closed_at":"2018-04-30T03:57:39Z","merged_at":"2018-04-30T03:57:39Z","merge_commit_sha":"308b3318bb15e7c1efc27ed8a21d40d7bc7f54ec","assignee":{"login":"cawarren","id":59526,"avatar_url":"https://avatars2.githubusercontent.com/u/59526?v=4","gravatar_id":"","url":"https://api.github.com/users/cawarren","html_url":"https://github.com/cawarren","followers_url":"https://api.github.com/users/cawarren/followers","following_url":"https://api.github.com/users/cawarren/following{/other_user}","gists_url":"https://api.github.com/users/cawarren/gists{/gist_id}","starred_url":"https://api.github.com/users/cawarren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cawarren/subscriptions","organizations_url":"https://api.github.com/users/cawarren/orgs","repos_url":"https://api.github.com/users/cawarren/repos","events_url":"https://api.github.com/users/cawarren/events{/privacy}","received_events_url":"https://api.github.com/users/cawarren/received_events","type":"User","site_admin":false},"assignees":[{"login":"cawarren","id":59526,"avatar_url":"https://avatars2.githubusercontent.com/u/59526?v=4","gravatar_id":"","url":"https://api.github.com/users/cawarren","html_url":"https://github.com/cawarren","followers_url":"https://api.github.com/users/cawarren/followers","following_url":"https://api.github.com/users/cawarren/following{/other_user}","gists_url":"https://api.github.com/users/cawarren/gists{/gist_id}","starred_url":"https://api.github.com/users/cawarren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cawarren/subscriptions","organizations_url":"https://api.github.com/users/cawarren/orgs","repos_url":"https://api.github.com/users/cawarren/repos","events_url":"https://api.github.com/users/cawarren/events{/privacy}","received_events_url":"https://api.github.com/users/cawarren/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46/commits","review_comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46/comments","review_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/46/comments","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/bfc9bc8ebda9b54f79ee1341a86b227d3b91fb67","head":{"label":"Recidiviz:ny-rescrape","ref":"ny-rescrape","sha":"bfc9bc8ebda9b54f79ee1341a86b227d3b91fb67","user":{"login":"Recidiviz","id":35868122,"avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"repo":{"id":108306686,"name":"recidiviz-data","full_name":"Recidiviz/recidiviz-data","owner":{"login":"Recidiviz","id":35868122,"avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Recidiviz/recidiviz-data","description":"Ingesting prison and jail data from various data sources and calculating criminal justice metrics","fork":false,"url":"https://api.github.com/repos/Recidiviz/recidiviz-data","forks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/forks","keys_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/teams","hooks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/hooks","issue_events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/events{/number}","events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/events","assignees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/assignees{/user}","branches_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/branches{/branch}","tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/tags","blobs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/{sha}","languages_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/languages","stargazers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/stargazers","contributors_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contributors","subscribers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscribers","subscription_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscription","commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contents/{+path}","compare_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/merges","archive_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/downloads","issues_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues{/number}","pulls_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls{/number}","milestones_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/milestones{/number}","notifications_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/labels{/name}","releases_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/releases{/id}","deployments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/deployments","created_at":"2017-10-25T18:04:59Z","updated_at":"2018-05-04T05:47:50Z","pushed_at":"2018-05-04T07:41:13Z","git_url":"git://github.com/Recidiviz/recidiviz-data.git","ssh_url":"git@github.com:Recidiviz/recidiviz-data.git","clone_url":"https://github.com/Recidiviz/recidiviz-data.git","svn_url":"https://github.com/Recidiviz/recidiviz-data","homepage":"","size":2325,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":30,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0"},"forks":2,"open_issues":30,"watchers":2,"default_branch":"master"}},"base":{"label":"Recidiviz:master","ref":"master","sha":"51ae8fcea35dbb396c1546eaf20e1aebca11ba33","user":{"login":"Recidiviz","id":35868122,"avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"repo":{"id":108306686,"name":"recidiviz-data","full_name":"Recidiviz/recidiviz-data","owner":{"login":"Recidiviz","id":35868122,"avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Recidiviz/recidiviz-data","description":"Ingesting prison and jail data from various data sources and calculating criminal justice metrics","fork":false,"url":"https://api.github.com/repos/Recidiviz/recidiviz-data","forks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/forks","keys_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/teams","hooks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/hooks","issue_events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/events{/number}","events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/events","assignees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/assignees{/user}","branches_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/branches{/branch}","tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/tags","blobs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/{sha}","languages_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/languages","stargazers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/stargazers","contributors_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contributors","subscribers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscribers","subscription_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscription","commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contents/{+path}","compare_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/merges","archive_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/downloads","issues_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues{/number}","pulls_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls{/number}","milestones_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/milestones{/number}","notifications_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/labels{/name}","releases_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/releases{/id}","deployments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/deployments","created_at":"2017-10-25T18:04:59Z","updated_at":"2018-05-04T05:47:50Z","pushed_at":"2018-05-04T07:41:13Z","git_url":"git://github.com/Recidiviz/recidiviz-data.git","ssh_url":"git@github.com:Recidiviz/recidiviz-data.git","clone_url":"https://github.com/Recidiviz/recidiviz-data.git","svn_url":"https://github.com/Recidiviz/recidiviz-data","homepage":"","size":2325,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":30,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0"},"forks":2,"open_issues":30,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46"},"html":{"href":"https://github.com/Recidiviz/recidiviz-data/pull/46"},"issue":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/46"},"comments":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/46/comments"},"review_comments":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46/comments"},"review_comment":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/46/commits"},"statuses":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/bfc9bc8ebda9b54f79ee1341a86b227d3b91fb67"}},"author_association":"OWNER"}}
{"actor":{"display_login":"jessex"}}