{"action":"created","comment":{"url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/comments/242205869","pull_request_review_id":185663169,"id":242205869,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MjIwNTg2OQ==","diff_hunk":"@@ -0,0 +1,20 @@\n+submissions-scraper\n+-------------------\n+\n+This script is a work-around to getting data about submissions into a submissions csv (pulled from the `redistricting_plansubmissions` table). It scrapes the submissions report HTML for each submission line in the CSV and adds `competitiveness`, `split_counties`, `majority_minority`, `compactness`, and `population_equivalence` to each line.\n+\n+# Pre-requisites\n+\n+To use the script, you need to:\n+ - Install Golang on your machine. (Ensure that you setup the `$HOME/go` directly correctly or setup `$GOPATH` as outlined in documentation [here](https://github.com/golang/go/wiki/SettingGOPATH).)","path":"submissions-scraper/README.md","position":9,"original_position":9,"commit_id":"696e7791b958925fe560890de17a3873f64fcb65","original_commit_id":"696e7791b958925fe560890de17a3873f64fcb65","user":{"login":"pcaisse","id":2926237,"node_id":"MDQ6VXNlcjI5MjYyMzc=","avatar_url":"https://avatars2.githubusercontent.com/u/2926237?v=4","gravatar_id":"","url":"https://api.github.com/users/pcaisse","html_url":"https://github.com/pcaisse","followers_url":"https://api.github.com/users/pcaisse/followers","following_url":"https://api.github.com/users/pcaisse/following{/other_user}","gists_url":"https://api.github.com/users/pcaisse/gists{/gist_id}","starred_url":"https://api.github.com/users/pcaisse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcaisse/subscriptions","organizations_url":"https://api.github.com/users/pcaisse/orgs","repos_url":"https://api.github.com/users/pcaisse/repos","events_url":"https://api.github.com/users/pcaisse/events{/privacy}","received_events_url":"https://api.github.com/users/pcaisse/received_events","type":"User","site_admin":false},"body":"One small typo here: s/directly/directory","created_at":"2018-12-17T16:00:03Z","updated_at":"2018-12-17T16:00:04Z","html_url":"https://github.com/azavea/district-builder-dtl-pa/pull/133#discussion_r242205869","pull_request_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/comments/242205869"},"html":{"href":"https://github.com/azavea/district-builder-dtl-pa/pull/133#discussion_r242205869"},"pull_request":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133"}}},"pull_request":{"url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133","id":238515504,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM4NTE1NTA0","html_url":"https://github.com/azavea/district-builder-dtl-pa/pull/133","diff_url":"https://github.com/azavea/district-builder-dtl-pa/pull/133.diff","patch_url":"https://github.com/azavea/district-builder-dtl-pa/pull/133.patch","issue_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/133","number":133,"state":"open","locked":false,"title":"Add script to scrape HTML files and add data to submissions CSV","user":{"login":"shreshthkhilani","id":3959096,"node_id":"MDQ6VXNlcjM5NTkwOTY=","avatar_url":"https://avatars0.githubusercontent.com/u/3959096?v=4","gravatar_id":"","url":"https://api.github.com/users/shreshthkhilani","html_url":"https://github.com/shreshthkhilani","followers_url":"https://api.github.com/users/shreshthkhilani/followers","following_url":"https://api.github.com/users/shreshthkhilani/following{/other_user}","gists_url":"https://api.github.com/users/shreshthkhilani/gists{/gist_id}","starred_url":"https://api.github.com/users/shreshthkhilani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shreshthkhilani/subscriptions","organizations_url":"https://api.github.com/users/shreshthkhilani/orgs","repos_url":"https://api.github.com/users/shreshthkhilani/repos","events_url":"https://api.github.com/users/shreshthkhilani/events{/privacy}","received_events_url":"https://api.github.com/users/shreshthkhilani/received_events","type":"User","site_admin":false},"body":"## Overview\r\n\r\nC70 wants to include some metrics in the CSV we send them along with the submission reports. To do this, this solution takes a submissions CSV, scrapes the HTML submission report files, and builds a new CSV with additional columns for `competitiveness`, `split_counties`, `majority_minority`, `compactness`, and `population_equivalence`.\r\n\r\n### Checklist\r\n\r\n- [x] PR has a name that won't get you publicly shamed for vagueness\r\n~- [ ] Files changed in the PR have been `yapf`-ed for style violations~\r\n\r\n### Notes\r\n\r\nThis will likely run only once, on 2018-12-15 by @shreshthkhilani.\r\n\r\n\r\n## Testing Instructions\r\n\r\n * Make sure you have a few submissions locally. If not, you can do these steps on staging. \r\n * On the VM, enter the Django container `sudo docker exec -it <container_id> bash`. \r\n * Generate submission reports `for i in {1..240}; do ./manage.py submission_report $i; done`\r\n * Enter the dbshell with `./manage.py dbshell` and run `\\copy (SELECT * FROM redistricting_plansubmission) TO '/tmp/submissions.csv' WITH CSV HEADER`.\r\n * You should now have a `submissions.csv` and some `*.html`s\r\n * Outside the container, copy the csv and html files out with `sudo docker cp <container_id> <file_path> .`\r\n * Outside the VM, scp the files down with `scp varant@127.0.0:/home/vagrant/$i.html .`.\r\n * Now, build the go file `go build main.go` and run it `./main <path to csv> <path to dir with htmls>`\r\n * Check a few html files against the `output.csv` file to ensure that the values are correct.\r\n\r\nCloses [#162639036](https://www.pivotaltracker.com/story/show/162639036)\r\n","created_at":"2018-12-13T20:06:30Z","updated_at":"2018-12-17T16:00:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7ce66669a1d40f65c67181111a9301f84d125350","assignee":{"login":"pcaisse","id":2926237,"node_id":"MDQ6VXNlcjI5MjYyMzc=","avatar_url":"https://avatars2.githubusercontent.com/u/2926237?v=4","gravatar_id":"","url":"https://api.github.com/users/pcaisse","html_url":"https://github.com/pcaisse","followers_url":"https://api.github.com/users/pcaisse/followers","following_url":"https://api.github.com/users/pcaisse/following{/other_user}","gists_url":"https://api.github.com/users/pcaisse/gists{/gist_id}","starred_url":"https://api.github.com/users/pcaisse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcaisse/subscriptions","organizations_url":"https://api.github.com/users/pcaisse/orgs","repos_url":"https://api.github.com/users/pcaisse/repos","events_url":"https://api.github.com/users/pcaisse/events{/privacy}","received_events_url":"https://api.github.com/users/pcaisse/received_events","type":"User","site_admin":false},"assignees":[{"login":"pcaisse","id":2926237,"node_id":"MDQ6VXNlcjI5MjYyMzc=","avatar_url":"https://avatars2.githubusercontent.com/u/2926237?v=4","gravatar_id":"","url":"https://api.github.com/users/pcaisse","html_url":"https://github.com/pcaisse","followers_url":"https://api.github.com/users/pcaisse/followers","following_url":"https://api.github.com/users/pcaisse/following{/other_user}","gists_url":"https://api.github.com/users/pcaisse/gists{/gist_id}","starred_url":"https://api.github.com/users/pcaisse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcaisse/subscriptions","organizations_url":"https://api.github.com/users/pcaisse/orgs","repos_url":"https://api.github.com/users/pcaisse/repos","events_url":"https://api.github.com/users/pcaisse/events{/privacy}","received_events_url":"https://api.github.com/users/pcaisse/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133/commits","review_comments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133/comments","review_comment_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/comments{/number}","comments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/133/comments","statuses_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/statuses/696e7791b958925fe560890de17a3873f64fcb65","head":{"label":"azavea:feature/sk/scrape-submissions","ref":"feature/sk/scrape-submissions","sha":"696e7791b958925fe560890de17a3873f64fcb65","user":{"login":"azavea","id":595231,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5NTIzMQ==","avatar_url":"https://avatars3.githubusercontent.com/u/595231?v=4","gravatar_id":"","url":"https://api.github.com/users/azavea","html_url":"https://github.com/azavea","followers_url":"https://api.github.com/users/azavea/followers","following_url":"https://api.github.com/users/azavea/following{/other_user}","gists_url":"https://api.github.com/users/azavea/gists{/gist_id}","starred_url":"https://api.github.com/users/azavea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azavea/subscriptions","organizations_url":"https://api.github.com/users/azavea/orgs","repos_url":"https://api.github.com/users/azavea/repos","events_url":"https://api.github.com/users/azavea/events{/privacy}","received_events_url":"https://api.github.com/users/azavea/received_events","type":"Organization","site_admin":false},"repo":{"id":124141335,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQxNDEzMzU=","name":"district-builder-dtl-pa","full_name":"azavea/district-builder-dtl-pa","private":false,"owner":{"login":"azavea","id":595231,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5NTIzMQ==","avatar_url":"https://avatars3.githubusercontent.com/u/595231?v=4","gravatar_id":"","url":"https://api.github.com/users/azavea","html_url":"https://github.com/azavea","followers_url":"https://api.github.com/users/azavea/followers","following_url":"https://api.github.com/users/azavea/following{/other_user}","gists_url":"https://api.github.com/users/azavea/gists{/gist_id}","starred_url":"https://api.github.com/users/azavea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azavea/subscriptions","organizations_url":"https://api.github.com/users/azavea/orgs","repos_url":"https://api.github.com/users/azavea/repos","events_url":"https://api.github.com/users/azavea/events{/privacy}","received_events_url":"https://api.github.com/users/azavea/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/azavea/district-builder-dtl-pa","description":"DistrictBuilder is web-based, open source software for collaborative redistricting.","fork":true,"url":"https://api.github.com/repos/azavea/district-builder-dtl-pa","forks_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/forks","keys_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/keys{/key_id}","collaborators_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/teams","hooks_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/hooks","issue_events_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/events{/number}","events_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/events","assignees_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/assignees{/user}","branches_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/branches{/branch}","tags_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/tags","blobs_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/refs{/sha}","trees_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/trees{/sha}","statuses_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/statuses/{sha}","languages_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/languages","stargazers_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/stargazers","contributors_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/contributors","subscribers_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/subscribers","subscription_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/subscription","commits_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/commits{/sha}","git_commits_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/commits{/sha}","comments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/comments{/number}","issue_comment_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/comments{/number}","contents_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/contents/{+path}","compare_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/compare/{base}...{head}","merges_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/merges","archive_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/downloads","issues_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues{/number}","pulls_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls{/number}","milestones_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/milestones{/number}","notifications_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/labels{/name}","releases_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/releases{/id}","deployments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/deployments","created_at":"2018-03-06T21:33:19Z","updated_at":"2018-12-07T17:27:10Z","pushed_at":"2018-12-17T15:42:32Z","git_url":"git://github.com/azavea/district-builder-dtl-pa.git","ssh_url":"git@github.com:azavea/district-builder-dtl-pa.git","clone_url":"https://github.com/azavea/district-builder-dtl-pa.git","svn_url":"https://github.com/azavea/district-builder-dtl-pa","homepage":"http://www.districtbuilder.org/","size":423596,"stargazers_count":3,"watchers_count":3,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1,"open_issues":5,"watchers":3,"default_branch":"develop"}},"base":{"label":"azavea:develop","ref":"develop","sha":"d919e5d92133b8d80ba95d74e6a977fda8620848","user":{"login":"azavea","id":595231,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5NTIzMQ==","avatar_url":"https://avatars3.githubusercontent.com/u/595231?v=4","gravatar_id":"","url":"https://api.github.com/users/azavea","html_url":"https://github.com/azavea","followers_url":"https://api.github.com/users/azavea/followers","following_url":"https://api.github.com/users/azavea/following{/other_user}","gists_url":"https://api.github.com/users/azavea/gists{/gist_id}","starred_url":"https://api.github.com/users/azavea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azavea/subscriptions","organizations_url":"https://api.github.com/users/azavea/orgs","repos_url":"https://api.github.com/users/azavea/repos","events_url":"https://api.github.com/users/azavea/events{/privacy}","received_events_url":"https://api.github.com/users/azavea/received_events","type":"Organization","site_admin":false},"repo":{"id":124141335,"node_id":"MDEwOlJlcG9zaXRvcnkxMjQxNDEzMzU=","name":"district-builder-dtl-pa","full_name":"azavea/district-builder-dtl-pa","private":false,"owner":{"login":"azavea","id":595231,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5NTIzMQ==","avatar_url":"https://avatars3.githubusercontent.com/u/595231?v=4","gravatar_id":"","url":"https://api.github.com/users/azavea","html_url":"https://github.com/azavea","followers_url":"https://api.github.com/users/azavea/followers","following_url":"https://api.github.com/users/azavea/following{/other_user}","gists_url":"https://api.github.com/users/azavea/gists{/gist_id}","starred_url":"https://api.github.com/users/azavea/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/azavea/subscriptions","organizations_url":"https://api.github.com/users/azavea/orgs","repos_url":"https://api.github.com/users/azavea/repos","events_url":"https://api.github.com/users/azavea/events{/privacy}","received_events_url":"https://api.github.com/users/azavea/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/azavea/district-builder-dtl-pa","description":"DistrictBuilder is web-based, open source software for collaborative redistricting.","fork":true,"url":"https://api.github.com/repos/azavea/district-builder-dtl-pa","forks_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/forks","keys_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/keys{/key_id}","collaborators_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/teams","hooks_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/hooks","issue_events_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/events{/number}","events_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/events","assignees_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/assignees{/user}","branches_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/branches{/branch}","tags_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/tags","blobs_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/refs{/sha}","trees_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/trees{/sha}","statuses_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/statuses/{sha}","languages_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/languages","stargazers_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/stargazers","contributors_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/contributors","subscribers_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/subscribers","subscription_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/subscription","commits_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/commits{/sha}","git_commits_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/git/commits{/sha}","comments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/comments{/number}","issue_comment_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/comments{/number}","contents_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/contents/{+path}","compare_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/compare/{base}...{head}","merges_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/merges","archive_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/downloads","issues_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues{/number}","pulls_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls{/number}","milestones_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/milestones{/number}","notifications_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/labels{/name}","releases_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/releases{/id}","deployments_url":"https://api.github.com/repos/azavea/district-builder-dtl-pa/deployments","created_at":"2018-03-06T21:33:19Z","updated_at":"2018-12-07T17:27:10Z","pushed_at":"2018-12-17T15:42:32Z","git_url":"git://github.com/azavea/district-builder-dtl-pa.git","ssh_url":"git@github.com:azavea/district-builder-dtl-pa.git","clone_url":"https://github.com/azavea/district-builder-dtl-pa.git","svn_url":"https://github.com/azavea/district-builder-dtl-pa","homepage":"http://www.districtbuilder.org/","size":423596,"stargazers_count":3,"watchers_count":3,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":5,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1,"open_issues":5,"watchers":3,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133"},"html":{"href":"https://github.com/azavea/district-builder-dtl-pa/pull/133"},"issue":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/133"},"comments":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/issues/133/comments"},"review_comments":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133/comments"},"review_comment":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/pulls/133/commits"},"statuses":{"href":"https://api.github.com/repos/azavea/district-builder-dtl-pa/statuses/696e7791b958925fe560890de17a3873f64fcb65"}},"author_association":"NONE"}}
{"actor":{"display_login":"pcaisse"}}