{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/212","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/212/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/212/events","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/212","id":131592209,"number":212,"title":"Automatic robots.txt parsing","user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-02-05T09:13:18Z","updated_at":"2016-02-12T08:20:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/pulls/212","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/212","diff_url":"https://github.com/cgiffard/node-simplecrawler/pull/212.diff","patch_url":"https://github.com/cgiffard/node-simplecrawler/pull/212.patch"},"body":"As discussed in #153, this PR brings on-by-default support for robots.txt parsing."},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/183225136","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/212#issuecomment-183225136","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/212","id":183225136,"user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"created_at":"2016-02-12T08:20:53Z","updated_at":"2016-02-12T08:20:53Z","body":"I've added logic that should deal with multiple hostnames (protocol and port changes count as well). Now we just need to add tests for it. I'll make it so it starts a second instance of testserver on a different port. Thoughts on how to deal with the tests in the best way, @cgiffard?"}}