{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/2073","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/2073/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/2073/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/2073/events","html_url":"https://github.com/scrapy/scrapy/issues/2073","id":161724972,"number":2073,"title":"Redirect Middleware doesn't respect chain order","user":{"login":"rgtk","id":765841,"avatar_url":"https://avatars.githubusercontent.com/u/765841?v=3","gravatar_id":"","url":"https://api.github.com/users/rgtk","html_url":"https://github.com/rgtk","followers_url":"https://api.github.com/users/rgtk/followers","following_url":"https://api.github.com/users/rgtk/following{/other_user}","gists_url":"https://api.github.com/users/rgtk/gists{/gist_id}","starred_url":"https://api.github.com/users/rgtk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgtk/subscriptions","organizations_url":"https://api.github.com/users/rgtk/orgs","repos_url":"https://api.github.com/users/rgtk/repos","events_url":"https://api.github.com/users/rgtk/events{/privacy}","received_events_url":"https://api.github.com/users/rgtk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-06-22T16:14:00Z","updated_at":"2016-06-24T07:22:51Z","closed_at":null,"body":"When custom middleware in `DOWNLOADER_MIDDLEWARES` is added below all redirect-related middlewares in chain, custom's `process_response` method is not executed before redirect-related ones.\r\n\r\nI tried to override chain directly in project's settings and extensions without any results. Setting redirect middlewares to None works as expected.\r\n\r\n---\r\n\r\nscrapy_redirect_first.py:\r\n```\r\nimport scrapy\r\n\r\nclass DummyDownloaderExtension(object):\r\n    def process_response(self, request, response, spider):\r\n        if 'something' in response.url:\r\n            request.meta['dont_redirect'] = True\r\n\r\n        print(\"==============> DUMMY YOU!\")\r\n        return response\r\n\r\nclass Spider(scrapy.Spider):\r\n    name = 'github-scrapy'\r\n    start_urls = ['http://blog.scrapinghub.com'] # use http, to be redirected to https\r\n    custom_settings = {\r\n        'DOWNLOADER_MIDDLEWARES': {\r\n            'scrapy_redirect_first.DummyDownloaderExtension': 50,\r\n            'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,\r\n            'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600\r\n        }\r\n    }\r\n\r\n    def parse(self, response):\r\n        return {}\r\n```\r\n\r\n---\r\nOutput:\r\n```\r\n~ $ scrapy runspider scrapy_redirect_first.py\r\n2016-06-22 18:01:41 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot)\r\n2016-06-22 18:01:41 [scrapy] INFO: Overridden settings: {}\r\n2016-06-22 18:01:41 [scrapy] INFO: Enabled extensions:\r\n['scrapy.extensions.logstats.LogStats',\r\n 'scrapy.extensions.telnet.TelnetConsole',\r\n 'scrapy.extensions.corestats.CoreStats']\r\n2016-06-22 18:01:41 [scrapy] INFO: Enabled downloader middlewares:\r\n['scrapy_redirect_first.DummyDownloaderExtension',\r\n 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\r\n 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\r\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\r\n 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\r\n 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\r\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\r\n 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\r\n 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',\r\n 'scrapy.downloadermiddlewares.stats.DownloaderStats']\r\n2016-06-22 18:01:41 [scrapy] INFO: Enabled spider middlewares:\r\n['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\r\n 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\r\n 'scrapy.spidermiddlewares.referer.RefererMiddleware',\r\n 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\r\n 'scrapy.spidermiddlewares.depth.DepthMiddleware']\r\n2016-06-22 18:01:41 [scrapy] INFO: Enabled item pipelines:\r\n[]\r\n2016-06-22 18:03:02 [scrapy] INFO: Spider opened\r\n2016-06-22 18:03:02 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2016-06-22 18:03:02 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023\r\n2016-06-22 18:03:02 [scrapy] DEBUG: Redirecting (301) to <GET https://blog.scrapinghub.com/> from <GET http://blog.scrapinghub.c>\r\n==============> DUMMY YOU!\r\n2016-06-22 18:03:03 [scrapy] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com/> (referer: None)\r\n2016-06-22 18:03:03 [scrapy] DEBUG: Scraped from <200 https://blog.scrapinghub.com/>\r\n{}\r\n2016-06-22 18:03:03 [scrapy] INFO: Closing spider (finished)\r\n2016-06-22 18:03:03 [scrapy] INFO: Dumping Scrapy stats:\r\n{'downloader/request_bytes': 436,\r\n 'downloader/request_count': 2,\r\n 'downloader/request_method_count/GET': 2,\r\n 'downloader/response_bytes': 33772,\r\n 'downloader/response_count': 2,\r\n 'downloader/response_status_count/200': 1,\r\n 'downloader/response_status_count/301': 1,\r\n 'finish_reason': 'finished',\r\n 'finish_time': datetime.datetime(2016, 6, 22, 16, 3, 3, 868303),\r\n 'item_scraped_count': 1,\r\n 'log_count/DEBUG': 4,\r\n 'log_count/INFO': 7,\r\n 'response_received_count': 1,\r\n 'scheduler/dequeued': 2,\r\n 'scheduler/dequeued/memory': 2,\r\n 'scheduler/enqueued': 2,\r\n 'scheduler/enqueued/memory': 2,\r\n 'start_time': datetime.datetime(2016, 6, 22, 16, 3, 2, 684250)}\r\n2016-06-22 18:03:03 [scrapy] INFO: Spider closed (finished)\r\n```"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/228275558","html_url":"https://github.com/scrapy/scrapy/issues/2073#issuecomment-228275558","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/2073","id":228275558,"user":{"login":"rgtk","id":765841,"avatar_url":"https://avatars.githubusercontent.com/u/765841?v=3","gravatar_id":"","url":"https://api.github.com/users/rgtk","html_url":"https://github.com/rgtk","followers_url":"https://api.github.com/users/rgtk/followers","following_url":"https://api.github.com/users/rgtk/following{/other_user}","gists_url":"https://api.github.com/users/rgtk/gists{/gist_id}","starred_url":"https://api.github.com/users/rgtk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgtk/subscriptions","organizations_url":"https://api.github.com/users/rgtk/orgs","repos_url":"https://api.github.com/users/rgtk/repos","events_url":"https://api.github.com/users/rgtk/events{/privacy}","received_events_url":"https://api.github.com/users/rgtk/received_events","type":"User","site_admin":false},"created_at":"2016-06-24T07:22:51Z","updated_at":"2016-06-24T07:22:51Z","body":"I have overlooked this. Thank you for pointing it out."}}
{"actor":{"display_login":"rgtk"}}