{"action":"opened","number":543,"pull_request":{"url":"https://api.github.com/repos/18F/pulse/pulls/543","id":84502640,"html_url":"https://github.com/18F/pulse/pull/543","diff_url":"https://github.com/18F/pulse/pull/543.diff","patch_url":"https://github.com/18F/pulse/pull/543.patch","issue_url":"https://api.github.com/repos/18F/pulse/issues/543","number":543,"state":"open","locked":false,"title":"WIP: Integrating subdomains","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"This PR is tracking the long-awaited work to integrate .gov subdomains into Pulse. \r\n\r\nIt will update the weekly scanning process to re-gather and re-scan subdomains, and will update the data ingest process to incorporate _aggregate_ subdomain-level statistics in the database. We're not going to store a row/entry for each individual subdomain. That would increase the complexity and size of our database immensely, as well as greatly increase churn, since subdomains are added/removed with great frequency -- much greater frequency than parent domains.\r\n\r\nIt is a **work in progress**, and I'll update when it's ready for review/merge. I wanted to get this all written out now, and will update the PR in-place.\r\n\r\n## Subdomain plan\r\n\r\nThe plan here is:\r\n\r\n* All subdomains will be pulled from **already public sources**, and will be limited to child subdomains of those .gov domains already tracked by Pulse. Subdomains ending in .mil, .us, .org, etc. will not be tracked, nor will state/local/tribal subdomains.\r\n* This will not expand the table. The table will remain with one row per-parent-domain.\r\n* Each subdomain will have **only** the `pshtt` scanner run on it. This will determine whether the subdomain is live and eligible to be counted, and what its HTTP/HTTPS/HSTS status is.\r\n* As part of the new message display for the HTTPS domains table, we'll show aggregate numbers and % of subdomains from each public source, per-parent-domain.\r\n* Each agency will have its subdomains factored into its overall % progress for HTTP, HTTPS, and HSTS. This will probably mean a separate % that is specific to subdomains, so as not to confuse people and not to diminish the work agencies have already put into their parent domains.\r\n* Subdomains will not only be scanned every week, but what subdomains are scanned will be re-gathered every week. So we'll re-scan Censys and DAP for eligible subdomains each week, meaning that new hostnames appear and disappear from eligibility from week to week.\r\n\r\nRight now, my thinking is that we're not going to offer downloadable CSVs of all of an agency's subdomains right away, but rather will link to Censys and DAP. This takes some amount of responsibility off of us for hosting data that might make agencies uncomfortable, while emphasizing their already public nature. \r\n\r\nBut at the same time, I also plan to expand our already-existing permalinks to detailed scan data per-subdomain, so that we have easy reference links for owners to figure out issues. We already have this:\r\n\r\nhttps://s3.amazonaws.com/pulse.cio.gov/live/cache/pshtt/18f.gov.json\r\n\r\nAnd so we'll likely generate and publish those per-subdomain as well. Malicious actors who wish to walk through that data can do that, but can already do that on Censys.io right now with basically the same level of friction. \r\n\r\nI feel pretty comfortable with this posture, but welcome feedback on how to proceed.\r\n\r\n## Data sources\r\n\r\nRight now, there are two public sources: \r\n\r\n#### Censys.io\r\n\r\n[Censys.io](https://censys.io) runs a zmap scan on the IPv4 space every night, and publishes the metadata of their scans. They scan several ports, including 443, and when they find TLS certifcates they record their metadata. Censys also syncs up with Certificate Transparency logs, so we're also getting hostnames from CT logs\r\nthat wouldn't otherwise have been spotted by a zmap scan. \r\n\r\nThis is helpful, because Censys' zmap scans have some completeness issues:\r\n\r\n* It is possible to opt out of their scans, they respect requests for this. This is generally uncommon, but it is notable for our purposes that the Department of Defense does currently opt out of these scans.\r\n* Their scans are unlikely to observe certificates that require a SNI extension with a particular hostname to see them. This can be a large gap.\r\n\r\nIn February 2016, @jcjones [measured the overlap between Censys' zmap scans and Certificate Transparency logs](https://tacticalsecret.com/early-impacts-of-letsencrypt/) (this is before Censys started syncing with CT) and found a surprisingly large non-overlap, with CT having overall many more unique certificates:\r\n\r\n[![certsinctversuscensys](https://cloud.githubusercontent.com/assets/4592/18334276/eb83b330-753b-11e6-9c0e-59681e03b16a.png)](https://tacticalsecret.com/early-impacts-of-letsencrypt/)\r\n\r\nSince then, Censys has begun syncing directly with a very comprehensive set of CT logs. I had originally planned to sync up with CT logs directly, but we may not need to do that now. However, if we ever do want to, this PR -- and the underlying work on domain-scan -- should make adding that data source relatively straightforward.\r\n\r\n#### Digital Analytics Program\r\n\r\nThe [Digital Analytics Program](https://analytics.usa.gov) (DAP) began publishing, [as of June 2016](https://github.com/18F/analytics-reporter/pull/165), a list of all hostnames which get reported into their underlying Google Analytics account that have at least 1 reported visit over the prior 14 days. \r\n\r\nThe CSV is here, and is updated automatically every night as part of the DAP's reporting: \r\nhttps://analytics.usa.gov/data/live/sites.csv\r\n\r\nThis CSV has a bunch of non-.gov hostnames (which are generally legitimate), but our filtering process will limit it to subdomains that are children of the parent domains we track in Pulse.\r\n\r\ncc @h-m-f-t @djharrity @alex @zakir ","created_at":"2016-09-08T15:41:54Z","updated_at":"2016-09-08T15:41:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/18F/pulse/pulls/543/commits","review_comments_url":"https://api.github.com/repos/18F/pulse/pulls/543/comments","review_comment_url":"https://api.github.com/repos/18F/pulse/pulls/comments{/number}","comments_url":"https://api.github.com/repos/18F/pulse/issues/543/comments","statuses_url":"https://api.github.com/repos/18F/pulse/statuses/b932c930b2ad6f7e25c7883c26ded6fb89857e0d","head":{"label":"18F:subdomains","ref":"subdomains","sha":"b932c930b2ad6f7e25c7883c26ded6fb89857e0d","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":31670909,"name":"pulse","full_name":"18F/pulse","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/pulse","description":"How the .gov domain space is doing at best practices and federal requirements.","fork":false,"url":"https://api.github.com/repos/18F/pulse","forks_url":"https://api.github.com/repos/18F/pulse/forks","keys_url":"https://api.github.com/repos/18F/pulse/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/pulse/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/pulse/teams","hooks_url":"https://api.github.com/repos/18F/pulse/hooks","issue_events_url":"https://api.github.com/repos/18F/pulse/issues/events{/number}","events_url":"https://api.github.com/repos/18F/pulse/events","assignees_url":"https://api.github.com/repos/18F/pulse/assignees{/user}","branches_url":"https://api.github.com/repos/18F/pulse/branches{/branch}","tags_url":"https://api.github.com/repos/18F/pulse/tags","blobs_url":"https://api.github.com/repos/18F/pulse/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/pulse/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/pulse/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/pulse/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/pulse/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/pulse/languages","stargazers_url":"https://api.github.com/repos/18F/pulse/stargazers","contributors_url":"https://api.github.com/repos/18F/pulse/contributors","subscribers_url":"https://api.github.com/repos/18F/pulse/subscribers","subscription_url":"https://api.github.com/repos/18F/pulse/subscription","commits_url":"https://api.github.com/repos/18F/pulse/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/pulse/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/pulse/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/pulse/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/pulse/contents/{+path}","compare_url":"https://api.github.com/repos/18F/pulse/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/pulse/merges","archive_url":"https://api.github.com/repos/18F/pulse/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/pulse/downloads","issues_url":"https://api.github.com/repos/18F/pulse/issues{/number}","pulls_url":"https://api.github.com/repos/18F/pulse/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/pulse/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/pulse/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/pulse/labels{/name}","releases_url":"https://api.github.com/repos/18F/pulse/releases{/id}","deployments_url":"https://api.github.com/repos/18F/pulse/deployments","created_at":"2015-03-04T17:47:35Z","updated_at":"2016-08-28T05:58:42Z","pushed_at":"2016-09-08T15:41:55Z","git_url":"git://github.com/18F/pulse.git","ssh_url":"git@github.com:18F/pulse.git","clone_url":"https://github.com/18F/pulse.git","svn_url":"https://github.com/18F/pulse","homepage":"https://pulse.cio.gov","size":8943,"stargazers_count":53,"watchers_count":53,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":44,"forks":24,"open_issues":44,"watchers":53,"default_branch":"master"}},"base":{"label":"18F:master","ref":"master","sha":"8f6899815e3821b24098f0b4a3502d07125d5b99","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":31670909,"name":"pulse","full_name":"18F/pulse","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/pulse","description":"How the .gov domain space is doing at best practices and federal requirements.","fork":false,"url":"https://api.github.com/repos/18F/pulse","forks_url":"https://api.github.com/repos/18F/pulse/forks","keys_url":"https://api.github.com/repos/18F/pulse/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/pulse/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/pulse/teams","hooks_url":"https://api.github.com/repos/18F/pulse/hooks","issue_events_url":"https://api.github.com/repos/18F/pulse/issues/events{/number}","events_url":"https://api.github.com/repos/18F/pulse/events","assignees_url":"https://api.github.com/repos/18F/pulse/assignees{/user}","branches_url":"https://api.github.com/repos/18F/pulse/branches{/branch}","tags_url":"https://api.github.com/repos/18F/pulse/tags","blobs_url":"https://api.github.com/repos/18F/pulse/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/pulse/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/pulse/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/pulse/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/pulse/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/pulse/languages","stargazers_url":"https://api.github.com/repos/18F/pulse/stargazers","contributors_url":"https://api.github.com/repos/18F/pulse/contributors","subscribers_url":"https://api.github.com/repos/18F/pulse/subscribers","subscription_url":"https://api.github.com/repos/18F/pulse/subscription","commits_url":"https://api.github.com/repos/18F/pulse/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/pulse/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/pulse/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/pulse/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/pulse/contents/{+path}","compare_url":"https://api.github.com/repos/18F/pulse/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/pulse/merges","archive_url":"https://api.github.com/repos/18F/pulse/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/pulse/downloads","issues_url":"https://api.github.com/repos/18F/pulse/issues{/number}","pulls_url":"https://api.github.com/repos/18F/pulse/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/pulse/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/pulse/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/pulse/labels{/name}","releases_url":"https://api.github.com/repos/18F/pulse/releases{/id}","deployments_url":"https://api.github.com/repos/18F/pulse/deployments","created_at":"2015-03-04T17:47:35Z","updated_at":"2016-08-28T05:58:42Z","pushed_at":"2016-09-08T15:41:55Z","git_url":"git://github.com/18F/pulse.git","ssh_url":"git@github.com:18F/pulse.git","clone_url":"https://github.com/18F/pulse.git","svn_url":"https://github.com/18F/pulse","homepage":"https://pulse.cio.gov","size":8943,"stargazers_count":53,"watchers_count":53,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":44,"forks":24,"open_issues":44,"watchers":53,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/18F/pulse/pulls/543"},"html":{"href":"https://github.com/18F/pulse/pull/543"},"issue":{"href":"https://api.github.com/repos/18F/pulse/issues/543"},"comments":{"href":"https://api.github.com/repos/18F/pulse/issues/543/comments"},"review_comments":{"href":"https://api.github.com/repos/18F/pulse/pulls/543/comments"},"review_comment":{"href":"https://api.github.com/repos/18F/pulse/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/18F/pulse/pulls/543/commits"},"statuses":{"href":"https://api.github.com/repos/18F/pulse/statuses/b932c930b2ad6f7e25c7883c26ded6fb89857e0d"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":6,"additions":135,"deletions":23,"changed_files":6}}
{"actor":{"display_login":"konklone"}}