{"action":"created","issue":{"url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/9","repository_url":"https://api.github.com/repos/dhs-ncats/pshtt","labels_url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/9/comments","events_url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/9/events","html_url":"https://github.com/dhs-ncats/pshtt/pull/9","id":171670083,"number":9,"title":"Add redirect fields, canonical endpoint field, and rework HTTP probing logic","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-17T14:20:03Z","updated_at":"2016-08-17T19:37:04Z","closed_at":"2016-08-17T19:35:14Z","pull_request":{"url":"https://api.github.com/repos/dhs-ncats/pshtt/pulls/9","html_url":"https://github.com/dhs-ncats/pshtt/pull/9","diff_url":"https://github.com/dhs-ncats/pshtt/pull/9.diff","patch_url":"https://github.com/dhs-ncats/pshtt/pull/9.patch"},"body":"This does a few things:\r\n\r\n* Adds some fields describing the nature of the immediate and eventual redirects: whether the redirects go to an `https://` URI, whether it goes to a `www` subdomain, whether the redirect leaves the hostname, and whether the redirect leaves the hostname's parent (registerable) domain.\r\n* Adds a `Canonical URL` field to the CSV and JSON output, that is determined using a port of [the logic from site-inspector](https://github.com/benbalter/site-inspector/blob/erics-mode/lib/site-inspector.rb#L201-L305). This makes use of the extra redirect fields mentioned above.\r\n* Changes the way that HTTP/HTTPS endpoints are inspected, by first hitting each endpoint with following redirects disabled. If a redirect is detected, the endpoint is pinged again with following redirects enabled, and then information about the eventual redirect is parsed. This creates some additional requests compared to the previous method, but greatly simplifies the logic, and allows the exception handling to be limited more tightly.\r\n* If an HTTPS endpoint throws an `SSLError` upon hitting it, it's retried again with certificate verification turned off. If it then works, the TLS error is overlooked and the endpoint's behavior is processed and analyzed. If it still doesn't work, then it's assumed to be a protocol error or some other novel issue that is preventing the connection from being established, and the endpoint is labeled not live.\r\n* When redirects are followed, they are followed with certificate validation turned off. This greatly reduces the risk of having a certificate error get thrown when following redirects starting from an `http:` endpoint. Detecting a certificate error there during that process isn't of any real value to us, as all we care about is the eventual redirect destination URI.\r\n* This refactoring to HTTP probing logic also removes the generic `except:` that was swallowing all errors (even `Ctrl-C` interrupts).\r\n* Unfortunately, I had to **disable requests caching** (the `--cache` option now does nothing) because of a bug in how `requests-cache` handles redirects and the `follow_redirects` and `verify` parameters. I've filed this bug here: https://github.com/reclosedev/requests-cache/issues/70 To re-enable caching, we'll either need to fix the bug in requests-cache, or route around it somehow by using another method of writing our own caching layer."},"comment":{"url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/comments/240522826","html_url":"https://github.com/dhs-ncats/pshtt/pull/9#issuecomment-240522826","issue_url":"https://api.github.com/repos/dhs-ncats/pshtt/issues/9","id":240522826,"user":{"login":"h-m-f-t","id":603901,"avatar_url":"https://avatars.githubusercontent.com/u/603901?v=3","gravatar_id":"","url":"https://api.github.com/users/h-m-f-t","html_url":"https://github.com/h-m-f-t","followers_url":"https://api.github.com/users/h-m-f-t/followers","following_url":"https://api.github.com/users/h-m-f-t/following{/other_user}","gists_url":"https://api.github.com/users/h-m-f-t/gists{/gist_id}","starred_url":"https://api.github.com/users/h-m-f-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/h-m-f-t/subscriptions","organizations_url":"https://api.github.com/users/h-m-f-t/orgs","repos_url":"https://api.github.com/users/h-m-f-t/repos","events_url":"https://api.github.com/users/h-m-f-t/events{/privacy}","received_events_url":"https://api.github.com/users/h-m-f-t/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T19:37:04Z","updated_at":"2016-08-17T19:37:04Z","body":":guitar: Most excellent! "}}
{"actor":{"display_login":"h-m-f-t"}}