{"action":"closed","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/225","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/225/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/225/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/225/events","html_url":"https://github.com/Norconex/collector-http/issues/225","id":127669699,"number":225,"title":"Start URLs with and without trailing slash treated differently","user":{"login":"niels","id":16709,"avatar_url":"https://avatars.githubusercontent.com/u/16709?v=3","gravatar_id":"","url":"https://api.github.com/users/niels","html_url":"https://github.com/niels","followers_url":"https://api.github.com/users/niels/followers","following_url":"https://api.github.com/users/niels/following{/other_user}","gists_url":"https://api.github.com/users/niels/gists{/gist_id}","starred_url":"https://api.github.com/users/niels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/niels/subscriptions","organizations_url":"https://api.github.com/users/niels/orgs","repos_url":"https://api.github.com/users/niels/repos","events_url":"https://api.github.com/users/niels/events{/privacy}","received_events_url":"https://api.github.com/users/niels/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Norconex/collector-http/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Norconex/collector-http/labels/fixed/done","name":"fixed/done","color":"bfe5bf"}],"state":"closed","locked":false,"assignee":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars.githubusercontent.com/u/2326274?v=3","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Norconex/collector-http/milestones/10","html_url":"https://github.com/Norconex/collector-http/milestones/2.4.0","labels_url":"https://api.github.com/repos/Norconex/collector-http/milestones/10/labels","id":1384602,"number":10,"title":"2.4.0","description":"Feature release. ","creator":{"login":"essiembre","id":2326274,"avatar_url":"https://avatars.githubusercontent.com/u/2326274?v=3","gravatar_id":"","url":"https://api.github.com/users/essiembre","html_url":"https://github.com/essiembre","followers_url":"https://api.github.com/users/essiembre/followers","following_url":"https://api.github.com/users/essiembre/following{/other_user}","gists_url":"https://api.github.com/users/essiembre/gists{/gist_id}","starred_url":"https://api.github.com/users/essiembre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/essiembre/subscriptions","organizations_url":"https://api.github.com/users/essiembre/orgs","repos_url":"https://api.github.com/users/essiembre/repos","events_url":"https://api.github.com/users/essiembre/events{/privacy}","received_events_url":"https://api.github.com/users/essiembre/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":20,"state":"open","created_at":"2015-10-30T03:30:44Z","updated_at":"2016-02-29T03:05:02Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2016-01-20T12:14:22Z","updated_at":"2016-02-29T03:05:02Z","closed_at":"2016-02-29T03:05:02Z","body":"This might be a WONTFIX as there is an easy workaround, but sites that respond with a relative redirect for their root domain won't be handled correctly when the startURL didn't contain a trailing slash:\r\n\r\nWhen queueing a `Location` response, the queued URL will differ based on whether the `startURL` contained a trailing slash or not. Given https://norconex-redir-test.herimedia.com, which redirects to `phone.html`, the crawler will correctly recognize\r\n\r\n> DEBUG [GenericRedirectURLProvider] URL redirect: https://norconex-redir-test.herimedia.com/ -> https://norconex-redir-test.herimedia.com/phone.html\r\n\r\nbut will incorrectly recognize\r\n\r\n> DEBUG [GenericRedirectURLProvider] URL redirect: https://norconex-redir-test.herimedia.com -> https://phone.html\r\n\r\nThis sample config shows the differing behavior depending solely on whether or not the startURL contains a trailing slash:\r\n\r\n```xml\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<httpcollector id=\"test-collector\">\r\n  <crawlers>\r\n    <crawler id=\"test-crawler-1\">\r\n      <startURLs stayOnDomain=\"true\">\r\n        <url>https://norconex-redir-test.herimedia.com</url>\r\n      </startURLs>\r\n    </crawler>\r\n    <crawler id=\"test-crawler-2\">\r\n      <startURLs stayOnDomain=\"true\">\r\n        <url>https://norconex-redir-test.herimedia.com/</url>\r\n      </startURLs>\r\n    </crawler>\r\n  </crawlers>\r\n</httpcollector>\r\n```\r\n\r\ntest-crawler-2 will work as expected (following the redirect to `https://norconex-redir-test.herimedia.com/phone.html`), whereas test-crawler-1 will interpret the redirect as leading to `https://phone.html`."}}