{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1744","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1744/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1744/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1744/events","html_url":"https://github.com/scrapy/scrapy/issues/1744","id":129772654,"number":1744,"title":"Redirects do not obey deny or allow patterns","user":{"login":"trifle","id":450916,"avatar_url":"https://avatars.githubusercontent.com/u/450916?v=3","gravatar_id":"","url":"https://api.github.com/users/trifle","html_url":"https://github.com/trifle","followers_url":"https://api.github.com/users/trifle/followers","following_url":"https://api.github.com/users/trifle/following{/other_user}","gists_url":"https://api.github.com/users/trifle/gists{/gist_id}","starred_url":"https://api.github.com/users/trifle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trifle/subscriptions","organizations_url":"https://api.github.com/users/trifle/orgs","repos_url":"https://api.github.com/users/trifle/repos","events_url":"https://api.github.com/users/trifle/events{/privacy}","received_events_url":"https://api.github.com/users/trifle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-29T13:57:42Z","updated_at":"2016-01-29T14:20:50Z","closed_at":null,"body":"Currently, redirects go wherever they want, ignoring the deny_res and deny_domain patterns set on a LinkExtractor. This means that no matter how one configures the LinkExtractor, denied URLs may end up being downloaded, parsed and stored.\r\n\r\nThe best way to stop this from happening would be to write a new downloader middleware. It should subclass BaseRedirectMiddleware, load the allow/deny patterns the same way OffsiteMiddleware does and raise IgnoreRequest when redirect_urls match these patterns.\r\n\r\nThere is already one related, but less generic issue #1042 .\r\nI think I remember some discussion about a desirable refactor of the URL pattern matching logic (which is all over the place), but I can't find it currently.\r\n\r\nI'd love to provide a pull request but it's unlikely I find the time to do so and I also don't have any experience writing tests (sorry). "},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/176779083","html_url":"https://github.com/scrapy/scrapy/issues/1744#issuecomment-176779083","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1744","id":176779083,"user":{"login":"kmike","id":107893,"avatar_url":"https://avatars.githubusercontent.com/u/107893?v=3","gravatar_id":"","url":"https://api.github.com/users/kmike","html_url":"https://github.com/kmike","followers_url":"https://api.github.com/users/kmike/followers","following_url":"https://api.github.com/users/kmike/following{/other_user}","gists_url":"https://api.github.com/users/kmike/gists{/gist_id}","starred_url":"https://api.github.com/users/kmike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmike/subscriptions","organizations_url":"https://api.github.com/users/kmike/orgs","repos_url":"https://api.github.com/users/kmike/repos","events_url":"https://api.github.com/users/kmike/events{/privacy}","received_events_url":"https://api.github.com/users/kmike/received_events","type":"User","site_admin":false},"created_at":"2016-01-29T14:20:50Z","updated_at":"2016-01-29T14:20:50Z","body":"Hey @trifle,\r\n\r\nThere is no way for LinkExtractor to use a final redirected URL as an input to allow/deny patters: LinkExtractor only extracts lnks from a page, it doesn't handle downloads; that's by design. \r\n\r\nWhen is it a problem? CrawlSpider?"}}