{"action":"created","comment":{"url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments/244524572","pull_request_review_id":188428900,"id":244524572,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0NDUyNDU3Mg==","diff_hunk":"@@ -0,0 +1,148 @@\n+\n+# Data Extractor\n+\n+The Data Extractor is a tool designed to make the extraction of data from a website much simpler.  At a high level the data extractor takes a webpage, and attempts to extract values from provided keys so the user doesn't need to worry about walking HTML trees or writing css selectors to get the data they need.  It will smartly find all of the keys the user lists and maps them to the correct fields in the IngestInfo object.  The result is a populated and properly construct object containing the information that exists on a page.\n+\n+## How To Use\n+\n+### The Yaml File\n+\n+The first step is to create a yaml file which contains all of the keys that you care about, and how they map to the underlying ingestion object fields.  To see a listing of all of the possible fields and their descriptions, see [IngestInfo](https://github.com/Recidiviz/recidiviz-data/tree/master/recidiviz/ingest/models/README.md).  \n+\n+#### key_mappings\n+\n+The first piece to fill in is the key_mappings.  Specifically this is data on a page that is listed as single key value pairs on a web page.  That is one value per key.  An example is the following:\n+\n+<div align=\"center\"><img src=\"https://i.imgur.com/oHVTVcU.png\" border=0></div>\n+\n+The resulting key_mappings in the yaml is:\n+\n+```yaml\n+key_mappings:\n+  Inmate No: person.person_id\n+  Gender: person.gender\n+  BirthDate: person.birthdate\n+  Age: person.age\n+  Race: person.race\n+  Committed By: booking.hold\n+  Booking Date-Time: booking.admission_date\n+  \"Booking #\": booking.booking_id\n+```\n+\n+Note that in this case, this particular site has fields for custody status and release date-time, but in fact they are never populated.  We want to let the data extractor know to ignore these fields.\n+\n+#### keys_to_ignore\n+The next thing that needs to be filled out is the keys_to_ignore.  This tells the data extractor to ignore these fields and not try to extract them.  The important thing to consider here is that the specific key should be ignored for all extractions.  This can be used for fields such as height, weight, hair colour, etc... because they do not map to anything in our internal database and we don't care about them.  The keys_to_ignore for the yaml file would be:","path":"recidiviz/ingest/extractor/README.md","position":35,"original_position":35,"commit_id":"821bfa76e4356a582b49a2ea92a3ac71c998f643","original_commit_id":"821bfa76e4356a582b49a2ea92a3ac71c998f643","user":{"login":"jessex","id":431895,"node_id":"MDQ6VXNlcjQzMTg5NQ==","avatar_url":"https://avatars1.githubusercontent.com/u/431895?v=4","gravatar_id":"","url":"https://api.github.com/users/jessex","html_url":"https://github.com/jessex","followers_url":"https://api.github.com/users/jessex/followers","following_url":"https://api.github.com/users/jessex/following{/other_user}","gists_url":"https://api.github.com/users/jessex/gists{/gist_id}","starred_url":"https://api.github.com/users/jessex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jessex/subscriptions","organizations_url":"https://api.github.com/users/jessex/orgs","repos_url":"https://api.github.com/users/jessex/repos","events_url":"https://api.github.com/users/jessex/events{/privacy}","received_events_url":"https://api.github.com/users/jessex/received_events","type":"User","site_admin":false},"body":"Backticks around `keys_to_ignore` in both places here. I promise not to make anymore backtick comments :)","created_at":"2018-12-30T05:43:51Z","updated_at":"2018-12-30T05:43:51Z","html_url":"https://github.com/Recidiviz/recidiviz-data/pull/426#discussion_r244524572","pull_request_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments/244524572"},"html":{"href":"https://github.com/Recidiviz/recidiviz-data/pull/426#discussion_r244524572"},"pull_request":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426"}}},"pull_request":{"url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426","id":241386865,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQxMzg2ODY1","html_url":"https://github.com/Recidiviz/recidiviz-data/pull/426","diff_url":"https://github.com/Recidiviz/recidiviz-data/pull/426.diff","patch_url":"https://github.com/Recidiviz/recidiviz-data/pull/426.patch","issue_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/426","number":426,"state":"open","locked":false,"title":"Documentation for the data extractor.","user":{"login":"arian487","id":1262163,"node_id":"MDQ6VXNlcjEyNjIxNjM=","avatar_url":"https://avatars0.githubusercontent.com/u/1262163?v=4","gravatar_id":"","url":"https://api.github.com/users/arian487","html_url":"https://github.com/arian487","followers_url":"https://api.github.com/users/arian487/followers","following_url":"https://api.github.com/users/arian487/following{/other_user}","gists_url":"https://api.github.com/users/arian487/gists{/gist_id}","starred_url":"https://api.github.com/users/arian487/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arian487/subscriptions","organizations_url":"https://api.github.com/users/arian487/orgs","repos_url":"https://api.github.com/users/arian487/repos","events_url":"https://api.github.com/users/arian487/events{/privacy}","received_events_url":"https://api.github.com/users/arian487/received_events","type":"User","site_admin":false},"body":"First pass at documentation for the data extractor.","created_at":"2018-12-28T19:51:31Z","updated_at":"2018-12-30T05:43:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"91bc6a684a7026672dba093f1d95522ca102edfd","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426/commits","review_comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426/comments","review_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/426/comments","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/821bfa76e4356a582b49a2ea92a3ac71c998f643","head":{"label":"Recidiviz:readme","ref":"readme","sha":"821bfa76e4356a582b49a2ea92a3ac71c998f643","user":{"login":"Recidiviz","id":35868122,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1ODY4MTIy","avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"repo":{"id":108306686,"node_id":"MDEwOlJlcG9zaXRvcnkxMDgzMDY2ODY=","name":"recidiviz-data","full_name":"Recidiviz/recidiviz-data","private":false,"owner":{"login":"Recidiviz","id":35868122,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1ODY4MTIy","avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/Recidiviz/recidiviz-data","description":"Ingesting prison and jail records from various data sources and calculating criminal justice metrics","fork":false,"url":"https://api.github.com/repos/Recidiviz/recidiviz-data","forks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/forks","keys_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/teams","hooks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/hooks","issue_events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/events{/number}","events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/events","assignees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/assignees{/user}","branches_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/branches{/branch}","tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/tags","blobs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/{sha}","languages_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/languages","stargazers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/stargazers","contributors_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contributors","subscribers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscribers","subscription_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscription","commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contents/{+path}","compare_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/merges","archive_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/downloads","issues_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues{/number}","pulls_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls{/number}","milestones_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/milestones{/number}","notifications_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/labels{/name}","releases_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/releases{/id}","deployments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/deployments","created_at":"2017-10-25T18:04:59Z","updated_at":"2018-12-28T19:14:05Z","pushed_at":"2018-12-30T05:40:17Z","git_url":"git://github.com/Recidiviz/recidiviz-data.git","ssh_url":"git@github.com:Recidiviz/recidiviz-data.git","clone_url":"https://github.com/Recidiviz/recidiviz-data.git","svn_url":"https://github.com/Recidiviz/recidiviz-data","homepage":"","size":8904,"stargazers_count":8,"watchers_count":8,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"archived":false,"open_issues_count":102,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":9,"open_issues":102,"watchers":8,"default_branch":"master"}},"base":{"label":"Recidiviz:master","ref":"master","sha":"a95745a50af4a29d049201e1568a761685756949","user":{"login":"Recidiviz","id":35868122,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1ODY4MTIy","avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"repo":{"id":108306686,"node_id":"MDEwOlJlcG9zaXRvcnkxMDgzMDY2ODY=","name":"recidiviz-data","full_name":"Recidiviz/recidiviz-data","private":false,"owner":{"login":"Recidiviz","id":35868122,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1ODY4MTIy","avatar_url":"https://avatars2.githubusercontent.com/u/35868122?v=4","gravatar_id":"","url":"https://api.github.com/users/Recidiviz","html_url":"https://github.com/Recidiviz","followers_url":"https://api.github.com/users/Recidiviz/followers","following_url":"https://api.github.com/users/Recidiviz/following{/other_user}","gists_url":"https://api.github.com/users/Recidiviz/gists{/gist_id}","starred_url":"https://api.github.com/users/Recidiviz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Recidiviz/subscriptions","organizations_url":"https://api.github.com/users/Recidiviz/orgs","repos_url":"https://api.github.com/users/Recidiviz/repos","events_url":"https://api.github.com/users/Recidiviz/events{/privacy}","received_events_url":"https://api.github.com/users/Recidiviz/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/Recidiviz/recidiviz-data","description":"Ingesting prison and jail records from various data sources and calculating criminal justice metrics","fork":false,"url":"https://api.github.com/repos/Recidiviz/recidiviz-data","forks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/forks","keys_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/teams","hooks_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/hooks","issue_events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/events{/number}","events_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/events","assignees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/assignees{/user}","branches_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/branches{/branch}","tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/tags","blobs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/refs{/sha}","trees_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/{sha}","languages_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/languages","stargazers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/stargazers","contributors_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contributors","subscribers_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscribers","subscription_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/subscription","commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/commits{/sha}","git_commits_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/git/commits{/sha}","comments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/comments{/number}","issue_comment_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/comments{/number}","contents_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/contents/{+path}","compare_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/merges","archive_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/downloads","issues_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues{/number}","pulls_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls{/number}","milestones_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/milestones{/number}","notifications_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/labels{/name}","releases_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/releases{/id}","deployments_url":"https://api.github.com/repos/Recidiviz/recidiviz-data/deployments","created_at":"2017-10-25T18:04:59Z","updated_at":"2018-12-28T19:14:05Z","pushed_at":"2018-12-30T05:40:17Z","git_url":"git://github.com/Recidiviz/recidiviz-data.git","ssh_url":"git@github.com:Recidiviz/recidiviz-data.git","clone_url":"https://github.com/Recidiviz/recidiviz-data.git","svn_url":"https://github.com/Recidiviz/recidiviz-data","homepage":"","size":8904,"stargazers_count":8,"watchers_count":8,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":9,"mirror_url":null,"archived":false,"open_issues_count":102,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":9,"open_issues":102,"watchers":8,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426"},"html":{"href":"https://github.com/Recidiviz/recidiviz-data/pull/426"},"issue":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/426"},"comments":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/issues/426/comments"},"review_comments":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426/comments"},"review_comment":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/pulls/426/commits"},"statuses":{"href":"https://api.github.com/repos/Recidiviz/recidiviz-data/statuses/821bfa76e4356a582b49a2ea92a3ac71c998f643"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"jessex"}}