{"action":"created","comment":{"url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/comments/74071941","id":74071941,"diff_hunk":"@@ -43,49 +53,78 @@ Run the import command:\n     python manage.py import_nc --dest $PWD/ncdata\n \n This took ~25 minutes on my laptop. Run ``tail -f traffic_stops.log`` to follow\n-along.\n+along.  Reusing an existing ``--dest`` directory will speed up import.  However,\n+if import code has changed since the last time the directory was used, don't\n+reuse an existing directory.\n \n Now you should be able to view data with ``runserver``:\n \n .. code-block:: bash\n \n     python manage.py runserver\n \n+Admin\n++++++\n+\n+If the state database doesn't exist or migrations haven't been applied to it,\n+fix that first.\n+\n+Access /admin/tsdata/dataset/ and create a \"dataset\" describing the data to be\n+imported.  The normal MD URL is stored in ``md.data.__init__.py``; the normal\n+NC URL is stored in ``nc.management.commands.import_nc.py``.  Newer datasets\n+may be available at https://s3-us-west-2.amazonaws.com/openpolicingdata/.  Be\n+sure to specify a destination directory.\n+\n+Once the \"dataset\" has been created, select the new dataset in list view and\n+apply the \"Import selected dataset\" action.\n \n Server\n ------\n \n-Raw NC Data\n-___________\n+The PostgreSQL user must have SUPERUSER privileges to perform the import.\n+Depending on current admin policies, that may have to be granted and\n+revoked around the import.\n \n-To start with fresh NC data, first drop the server's database:\n+Temporarily grant our PostgreSQL user SUPERUSER privileges:\n \n .. code-block:: bash\n \n-    sudo -u postgres dropdb traffic_stops_nc_staging\n+    sudo -u postgres psql -c 'ALTER USER traffic_stops_staging WITH SUPERUSER;'\n \n-Then run a **deploy** to recreate the database.\n-\n-Temporarily grant our PostgreSQL user SUPERUSER privileges:\n+When finished, revoke SUPERUSER privileges:\n \n .. code-block:: bash\n \n-    sudo -u postgres psql -c 'ALTER USER traffic_stops_production WITH SUPERUSER;'\n+    sudo -u postgres psql -c 'ALTER USER traffic_stops_staging WITH NOSUPERUSER;'\n+\n+After importing new state data into the database used by a running server,\n+cached queries will continue to be used until they expire.  To flush the\n+cache, connect to ``memcached`` using ``telnet`` or some other suitable\n+client and send the ``flush_all`` command.\n+\n+Raw NC Data\n+___________\n+\n+Command-line\n+++++++++++++\n \n Run the import command:\n \n .. code-block:: bash\n \n     sudo su - traffic_stops\n-    cd /var/www/traffic-stops\n+    cd /var/www/traffic_stops\n     source ./env/bin/activate\n     ./manage.sh import_nc --dest=/var/www/traffic_stops/data\n \n-When finished, revoke SUPERUSER privileges:\n+Reusing an existing ``--dest`` directory will speed up import.  However,\n+if import code has changed since the last time the directory was used, don't\n+reuse an existing directory.\n \n-.. code-block:: bash\n+Admin\n++++++\n \n-    sudo -u postgres psql -c 'ALTER USER traffic_stops_staging WITH NOSUPERUSER;'\n+Follow the \"Admin\" instructions above under \"Local/Development Environment\".\n ","path":"docs/data-import.rst","position":137,"original_position":137,"commit_id":"a5796324ece387342424919982d3625a2dfc1977","original_commit_id":"a5796324ece387342424919982d3625a2dfc1977","user":{"login":"rlconley","id":6653029,"avatar_url":"https://avatars.githubusercontent.com/u/6653029?v=3","gravatar_id":"","url":"https://api.github.com/users/rlconley","html_url":"https://github.com/rlconley","followers_url":"https://api.github.com/users/rlconley/followers","following_url":"https://api.github.com/users/rlconley/following{/other_user}","gists_url":"https://api.github.com/users/rlconley/gists{/gist_id}","starred_url":"https://api.github.com/users/rlconley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rlconley/subscriptions","organizations_url":"https://api.github.com/users/rlconley/orgs","repos_url":"https://api.github.com/users/rlconley/repos","events_url":"https://api.github.com/users/rlconley/events{/privacy}","received_events_url":"https://api.github.com/users/rlconley/received_events","type":"User","site_admin":false},"body":"Maybe worth adding a hint about what to enter as source and destination in the admin.","created_at":"2016-08-09T14:42:43Z","updated_at":"2016-08-09T14:42:43Z","html_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134#discussion_r74071941","pull_request_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134","_links":{"self":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/comments/74071941"},"html":{"href":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134#discussion_r74071941"},"pull_request":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134"}}},"pull_request":{"url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134","id":80608455,"html_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134","diff_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134.diff","patch_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134.patch","issue_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/134","number":134,"state":"open","locked":false,"title":"ODPM-72 - Improve dev docs around migrations and data import","user":{"login":"trawick","id":1207350,"avatar_url":"https://avatars.githubusercontent.com/u/1207350?v=3","gravatar_id":"","url":"https://api.github.com/users/trawick","html_url":"https://github.com/trawick","followers_url":"https://api.github.com/users/trawick/followers","following_url":"https://api.github.com/users/trawick/following{/other_user}","gists_url":"https://api.github.com/users/trawick/gists{/gist_id}","starred_url":"https://api.github.com/users/trawick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trawick/subscriptions","organizations_url":"https://api.github.com/users/trawick/orgs","repos_url":"https://api.github.com/users/trawick/repos","events_url":"https://api.github.com/users/trawick/events{/privacy}","received_events_url":"https://api.github.com/users/trawick/received_events","type":"User","site_admin":false},"body":"","created_at":"2016-08-09T12:05:42Z","updated_at":"2016-08-09T14:42:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f00422d060ed67d8fdc679b35d097f72664d2263","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134/commits","review_comments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134/comments","review_comment_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/comments{/number}","comments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/134/comments","statuses_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/statuses/a5796324ece387342424919982d3625a2dfc1977","head":{"label":"OpenDataPolicingNC:ODPM-72-import-docs","ref":"ODPM-72-import-docs","sha":"a5796324ece387342424919982d3625a2dfc1977","user":{"login":"OpenDataPolicingNC","id":13080532,"avatar_url":"https://avatars.githubusercontent.com/u/13080532?v=3","gravatar_id":"","url":"https://api.github.com/users/OpenDataPolicingNC","html_url":"https://github.com/OpenDataPolicingNC","followers_url":"https://api.github.com/users/OpenDataPolicingNC/followers","following_url":"https://api.github.com/users/OpenDataPolicingNC/following{/other_user}","gists_url":"https://api.github.com/users/OpenDataPolicingNC/gists{/gist_id}","starred_url":"https://api.github.com/users/OpenDataPolicingNC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OpenDataPolicingNC/subscriptions","organizations_url":"https://api.github.com/users/OpenDataPolicingNC/orgs","repos_url":"https://api.github.com/users/OpenDataPolicingNC/repos","events_url":"https://api.github.com/users/OpenDataPolicingNC/events{/privacy}","received_events_url":"https://api.github.com/users/OpenDataPolicingNC/received_events","type":"Organization","site_admin":false},"repo":{"id":15360782,"name":"Traffic-Stops","full_name":"OpenDataPolicingNC/Traffic-Stops","owner":{"login":"OpenDataPolicingNC","id":13080532,"avatar_url":"https://avatars.githubusercontent.com/u/13080532?v=3","gravatar_id":"","url":"https://api.github.com/users/OpenDataPolicingNC","html_url":"https://github.com/OpenDataPolicingNC","followers_url":"https://api.github.com/users/OpenDataPolicingNC/followers","following_url":"https://api.github.com/users/OpenDataPolicingNC/following{/other_user}","gists_url":"https://api.github.com/users/OpenDataPolicingNC/gists{/gist_id}","starred_url":"https://api.github.com/users/OpenDataPolicingNC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OpenDataPolicingNC/subscriptions","organizations_url":"https://api.github.com/users/OpenDataPolicingNC/orgs","repos_url":"https://api.github.com/users/OpenDataPolicingNC/repos","events_url":"https://api.github.com/users/OpenDataPolicingNC/events{/privacy}","received_events_url":"https://api.github.com/users/OpenDataPolicingNC/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops","description":"Open Data Policing NC aggregates, visualizes, and publishes public records related to all known traffic stops to have occurred in North Carolina since January 1, 2002.","fork":false,"url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops","forks_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/forks","keys_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/keys{/key_id}","collaborators_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/teams","hooks_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/hooks","issue_events_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/events{/number}","events_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/events","assignees_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/assignees{/user}","branches_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/branches{/branch}","tags_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/tags","blobs_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/refs{/sha}","trees_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/trees{/sha}","statuses_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/statuses/{sha}","languages_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/languages","stargazers_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/stargazers","contributors_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/contributors","subscribers_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/subscribers","subscription_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/subscription","commits_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/commits{/sha}","git_commits_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/commits{/sha}","comments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/comments{/number}","issue_comment_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/comments{/number}","contents_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/contents/{+path}","compare_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/compare/{base}...{head}","merges_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/merges","archive_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/downloads","issues_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues{/number}","pulls_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls{/number}","milestones_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/milestones{/number}","notifications_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/labels{/name}","releases_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/releases{/id}","deployments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/deployments","created_at":"2013-12-21T15:18:35Z","updated_at":"2016-08-03T16:51:59Z","pushed_at":"2016-08-09T14:36:17Z","git_url":"git://github.com/OpenDataPolicingNC/Traffic-Stops.git","ssh_url":"git@github.com:OpenDataPolicingNC/Traffic-Stops.git","clone_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops.git","svn_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops","homepage":"https://opendatapolicingnc.com/","size":3109,"stargazers_count":12,"watchers_count":12,"language":"Shell","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":15,"forks":6,"open_issues":15,"watchers":12,"default_branch":"dev"}},"base":{"label":"OpenDataPolicingNC:dev","ref":"dev","sha":"55d25e7efaa7b50b069ce6d1f3c52c3aa383d350","user":{"login":"OpenDataPolicingNC","id":13080532,"avatar_url":"https://avatars.githubusercontent.com/u/13080532?v=3","gravatar_id":"","url":"https://api.github.com/users/OpenDataPolicingNC","html_url":"https://github.com/OpenDataPolicingNC","followers_url":"https://api.github.com/users/OpenDataPolicingNC/followers","following_url":"https://api.github.com/users/OpenDataPolicingNC/following{/other_user}","gists_url":"https://api.github.com/users/OpenDataPolicingNC/gists{/gist_id}","starred_url":"https://api.github.com/users/OpenDataPolicingNC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OpenDataPolicingNC/subscriptions","organizations_url":"https://api.github.com/users/OpenDataPolicingNC/orgs","repos_url":"https://api.github.com/users/OpenDataPolicingNC/repos","events_url":"https://api.github.com/users/OpenDataPolicingNC/events{/privacy}","received_events_url":"https://api.github.com/users/OpenDataPolicingNC/received_events","type":"Organization","site_admin":false},"repo":{"id":15360782,"name":"Traffic-Stops","full_name":"OpenDataPolicingNC/Traffic-Stops","owner":{"login":"OpenDataPolicingNC","id":13080532,"avatar_url":"https://avatars.githubusercontent.com/u/13080532?v=3","gravatar_id":"","url":"https://api.github.com/users/OpenDataPolicingNC","html_url":"https://github.com/OpenDataPolicingNC","followers_url":"https://api.github.com/users/OpenDataPolicingNC/followers","following_url":"https://api.github.com/users/OpenDataPolicingNC/following{/other_user}","gists_url":"https://api.github.com/users/OpenDataPolicingNC/gists{/gist_id}","starred_url":"https://api.github.com/users/OpenDataPolicingNC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OpenDataPolicingNC/subscriptions","organizations_url":"https://api.github.com/users/OpenDataPolicingNC/orgs","repos_url":"https://api.github.com/users/OpenDataPolicingNC/repos","events_url":"https://api.github.com/users/OpenDataPolicingNC/events{/privacy}","received_events_url":"https://api.github.com/users/OpenDataPolicingNC/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops","description":"Open Data Policing NC aggregates, visualizes, and publishes public records related to all known traffic stops to have occurred in North Carolina since January 1, 2002.","fork":false,"url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops","forks_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/forks","keys_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/keys{/key_id}","collaborators_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/teams","hooks_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/hooks","issue_events_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/events{/number}","events_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/events","assignees_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/assignees{/user}","branches_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/branches{/branch}","tags_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/tags","blobs_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/refs{/sha}","trees_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/trees{/sha}","statuses_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/statuses/{sha}","languages_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/languages","stargazers_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/stargazers","contributors_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/contributors","subscribers_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/subscribers","subscription_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/subscription","commits_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/commits{/sha}","git_commits_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/git/commits{/sha}","comments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/comments{/number}","issue_comment_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/comments{/number}","contents_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/contents/{+path}","compare_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/compare/{base}...{head}","merges_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/merges","archive_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/downloads","issues_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues{/number}","pulls_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls{/number}","milestones_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/milestones{/number}","notifications_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/labels{/name}","releases_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/releases{/id}","deployments_url":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/deployments","created_at":"2013-12-21T15:18:35Z","updated_at":"2016-08-03T16:51:59Z","pushed_at":"2016-08-09T14:36:17Z","git_url":"git://github.com/OpenDataPolicingNC/Traffic-Stops.git","ssh_url":"git@github.com:OpenDataPolicingNC/Traffic-Stops.git","clone_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops.git","svn_url":"https://github.com/OpenDataPolicingNC/Traffic-Stops","homepage":"https://opendatapolicingnc.com/","size":3109,"stargazers_count":12,"watchers_count":12,"language":"Shell","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":6,"mirror_url":null,"open_issues_count":15,"forks":6,"open_issues":15,"watchers":12,"default_branch":"dev"}},"_links":{"self":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134"},"html":{"href":"https://github.com/OpenDataPolicingNC/Traffic-Stops/pull/134"},"issue":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/134"},"comments":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/issues/134/comments"},"review_comments":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134/comments"},"review_comment":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/pulls/134/commits"},"statuses":{"href":"https://api.github.com/repos/OpenDataPolicingNC/Traffic-Stops/statuses/a5796324ece387342424919982d3625a2dfc1977"}}}}
{"actor":{"display_login":"rlconley"}}