{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/1905","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/1905/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1905/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/1905/events","html_url":"https://github.com/scrapy/scrapy/pull/1905","id":146178967,"number":1905,"title":"[MRG+1] Modified read failure recovery in utils/gz.py to read only the last f.extrasize bytes of f.extrabuf[ ]","user":{"login":"rootAvish","id":5445560,"avatar_url":"https://avatars.githubusercontent.com/u/5445560?v=3","gravatar_id":"","url":"https://api.github.com/users/rootAvish","html_url":"https://github.com/rootAvish","followers_url":"https://api.github.com/users/rootAvish/followers","following_url":"https://api.github.com/users/rootAvish/following{/other_user}","gists_url":"https://api.github.com/users/rootAvish/gists{/gist_id}","starred_url":"https://api.github.com/users/rootAvish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rootAvish/subscriptions","organizations_url":"https://api.github.com/users/rootAvish/orgs","repos_url":"https://api.github.com/users/rootAvish/repos","events_url":"https://api.github.com/users/rootAvish/events{/privacy}","received_events_url":"https://api.github.com/users/rootAvish/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2016-04-06T03:45:12Z","updated_at":"2016-08-17T12:51:44Z","closed_at":"2016-08-17T12:51:30Z","pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/1905","html_url":"https://github.com/scrapy/scrapy/pull/1905","diff_url":"https://github.com/scrapy/scrapy/pull/1905.diff","patch_url":"https://github.com/scrapy/scrapy/pull/1905.patch"},"body":"This fixes #1606. Building off the remarks of @jdmaeyer in that bug report, the major difference between downloading the file from this server and his pastebin link, is that this server is a Microsoft IIS instance. A look  and the http request that was being sent\r\n\r\n```python\r\nIn [4]: print(request.headers)\r\n{'Accept-Language': ['en'], 'Accept-Encoding': ['gzip,deflate'], 'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'User-Agent': ['Scrapy/1.2.0dev2 (+http://scrapy.org)']}\r\n```\r\nled me to `httpcompression` since we are accepted `gzip` encoded data.\r\n\r\nI manually saved the response to a file and tried decompressing it using `gunzip`, which is also used by `scrapy.utils.gz` for the same purpose.\r\n\r\n```bash\r\nrootavish@rootavish:~/scrapy$ gunzip --decompress dbg.gz \r\n\r\ngzip: dbg.gz: unexpected end of file\r\n```\r\n\r\nHowever, the same file was handled by KDE's Ark without any issues, so I looked into the code in `scrapy.utils.gz` for problems. Changing the chunk size of the read to something smaller than the file size seemed to fix it.\r\n\r\n```python\r\n chunk = read1(f, 8196) # changing chunk size to 4096 fixed the problem\r\n ```\r\n\r\nThis behaviour is unexpected since the read should ideally only take place till the end of the file, but due to the archive not being in pristine condition(possibly due to the compression taking place on a Windows based IIS instance and decoding on *nix), there was a wrap around read happening. The ideal solution would be to patch the `gzip` code itself, but since that's not an option we can make do with monkeypatching the same.  The idea was to replace `_read_eof` with a `no-op` for partial archives, something that won't work if GzipFile does it's checksum comparison. The problem should be fixed now.\r\n\r\n\r\n```bash\r\nrootavish@rootavish:~/scrapy$ scrapy shell \"http://files.qidian.com/Author4/3615059/88542882.txt\"\r\n2016-04-06 08:34:51 [scrapy] INFO: Scrapy 1.2.0dev2 started (bot: scrapybot)\r\n2016-04-06 08:34:51 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}\r\n2016-04-06 08:34:51 [scrapy] INFO: Enabled extensions:\r\n['scrapy.extensions.telnet.TelnetConsole',\r\n 'scrapy.extensions.corestats.CoreStats']\r\n2016-04-06 08:34:51 [scrapy] INFO: Enabled downloader middlewares:\r\n['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\r\n 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\r\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\r\n 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\r\n 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\r\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\r\n 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\r\n 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',\r\n 'scrapy.downloadermiddlewares.stats.DownloaderStats']\r\n2016-04-06 08:34:51 [scrapy] INFO: Enabled spider middlewares:\r\n['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\r\n 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\r\n 'scrapy.spidermiddlewares.referer.RefererMiddleware',\r\n 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\r\n 'scrapy.spidermiddlewares.depth.DepthMiddleware']\r\n2016-04-06 08:34:51 [scrapy] INFO: Enabled item pipelines:\r\n[]\r\n2016-04-06 08:34:51 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023\r\n2016-04-06 08:34:51 [scrapy] INFO: Spider opened\r\n2016-04-06 08:34:54 [scrapy] DEBUG: Crawled (200) <GET http://files.qidian.com/Author4/3615059/88542882.txt> (referer: None)\r\n[s] Available Scrapy objects:\r\n[s]   crawler    <scrapy.crawler.Crawler object at 0x7f7b70ce6b50>\r\n[s]   item       {}\r\n[s]   request    <GET http://files.qidian.com/Author4/3615059/88542882.txt>\r\n[s]   response   <200 http://files.qidian.com/Author4/3615059/88542882.txt>\r\n[s]   settings   <scrapy.settings.Settings object at 0x7f7b70ce69d0>\r\n[s]   spider     <DefaultSpider 'default' at 0x7f7b6febb090>\r\n[s] Useful shortcuts:\r\n[s]   shelp()           Shell help (print this help)\r\n[s]   fetch(req_or_url) Fetch request (or URL) and update local objects\r\n[s]   view(response)    View response in a browser\r\nIn [1]: import requests\r\n\r\nIn [2]: resp = requests.get(\"http://files.qidian.com/Author4/3615059/88542882.txt\")\r\n2016-04-06 08:35:20 [urllib3.connectionpool] INFO: Starting new HTTP connection (1): files.qidian.com\r\n2016-04-06 08:35:20 [urllib3.connectionpool] DEBUG: Setting read timeout to None\r\n2016-04-06 08:35:20 [urllib3.connectionpool] DEBUG: \"GET /Author4/3615059/88542882.txt HTTP/1.1\" 200 None\r\n\r\nIn [3]: response.body_as_unicode() == resp.text\r\nOut[3]: True\r\n\r\nIn [4]: response.body.count('document')\r\nOut[4]: 1\r\n```"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/240401305","html_url":"https://github.com/scrapy/scrapy/pull/1905#issuecomment-240401305","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1905","id":240401305,"user":{"login":"redapple","id":886296,"avatar_url":"https://avatars.githubusercontent.com/u/886296?v=3","gravatar_id":"","url":"https://api.github.com/users/redapple","html_url":"https://github.com/redapple","followers_url":"https://api.github.com/users/redapple/followers","following_url":"https://api.github.com/users/redapple/following{/other_user}","gists_url":"https://api.github.com/users/redapple/gists{/gist_id}","starred_url":"https://api.github.com/users/redapple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redapple/subscriptions","organizations_url":"https://api.github.com/users/redapple/orgs","repos_url":"https://api.github.com/users/redapple/repos","events_url":"https://api.github.com/users/redapple/events{/privacy}","received_events_url":"https://api.github.com/users/redapple/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T12:51:44Z","updated_at":"2016-08-17T12:51:44Z","body":"Thanks @rootAvish "}}
{"actor":{"display_login":"redapple"}}