{"action":"closed","issue":{"url":"https://api.github.com/repos/sjdirect/abotx/issues/17","repository_url":"https://api.github.com/repos/sjdirect/abotx","labels_url":"https://api.github.com/repos/sjdirect/abotx/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/sjdirect/abotx/issues/17/comments","events_url":"https://api.github.com/repos/sjdirect/abotx/issues/17/events","html_url":"https://github.com/sjdirect/abotx/issues/17","id":307180998,"number":17,"title":"Robots.txt is not reloaded when uri scheme is changed (http/https)","user":{"login":"askoutaris","id":13051434,"avatar_url":"https://avatars2.githubusercontent.com/u/13051434?v=4","gravatar_id":"","url":"https://api.github.com/users/askoutaris","html_url":"https://github.com/askoutaris","followers_url":"https://api.github.com/users/askoutaris/followers","following_url":"https://api.github.com/users/askoutaris/following{/other_user}","gists_url":"https://api.github.com/users/askoutaris/gists{/gist_id}","starred_url":"https://api.github.com/users/askoutaris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/askoutaris/subscriptions","organizations_url":"https://api.github.com/users/askoutaris/orgs","repos_url":"https://api.github.com/users/askoutaris/repos","events_url":"https://api.github.com/users/askoutaris/events{/privacy}","received_events_url":"https://api.github.com/users/askoutaris/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-21T10:04:06Z","updated_at":"2018-03-21T16:51:19Z","closed_at":"2018-03-21T16:51:18Z","author_association":"NONE","body":"Hello, \r\n\r\nWe found Abot a few days ago, and we try its free version to see if it can meet our needs. \r\n\r\nEverything worked fine until we noticed that it crawls urls which were 'Disallow' in robots.txt.\r\n\r\nAfter some debugging, we ended up that it binds robots.txt with the initial uri scheme which is provided with the site to crawl, eg for the site https://mysite.com disallowed urls works only for https. If there is a link to http://mysite.com/somepage, Abot will ignore robots.txt and will crawl it.\r\n\r\n_*Assuming we have the following Robots.txt\r\nUser-agent: *  \r\nDisallow: /somepage_\r\n\r\nCould you help us how to deal with this issue?\r\nThank you \r\n"}}
{"actor":{"display_login":"sjdirect"}}