{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/1724","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/1724/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/1724/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/1724/events","html_url":"https://github.com/GoogleCloudPlatform/gcloud-python/issues/1724","id":148729676,"number":1724,"title":"Download Fails Gzip Decompression","user":{"login":"jjangsangy","id":3851682,"avatar_url":"https://avatars.githubusercontent.com/u/3851682?v=3","gravatar_id":"","url":"https://api.github.com/users/jjangsangy","html_url":"https://github.com/jjangsangy","followers_url":"https://api.github.com/users/jjangsangy/followers","following_url":"https://api.github.com/users/jjangsangy/following{/other_user}","gists_url":"https://api.github.com/users/jjangsangy/gists{/gist_id}","starred_url":"https://api.github.com/users/jjangsangy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jjangsangy/subscriptions","organizations_url":"https://api.github.com/users/jjangsangy/orgs","repos_url":"https://api.github.com/users/jjangsangy/repos","events_url":"https://api.github.com/users/jjangsangy/events{/privacy}","received_events_url":"https://api.github.com/users/jjangsangy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/labels/storage","name":"storage","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":41,"created_at":"2016-04-15T18:15:16Z","updated_at":"2016-04-20T01:47:37Z","closed_at":null,"body":"I have files that have been uploaded using the gcloud command line interface with the  `-z` flag which applies gzip content-encoding on files during transfer.\r\n\r\n `gcloud -m cp -z text gs://<bucket>`\r\n\r\nIt appears that running the `blob.download_to_file()` and `blob.download_to_string()` raise a `httplib2.FailedToDecompressContent` exception on about 10% of the files. \r\n\r\n```python\r\nTraceback (most recent call last):\r\n  File \"manage.py\", line 10, in <module>\r\n    execute_from_command_line(sys.argv)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/__init__.py\", line 354, in execute_from_command_line\r\n    utility.execute()\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/__init__.py\", line 346, in execute\r\n    self.fetch_command(subcommand).run_from_argv(self.argv)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/base.py\", line 394, in run_from_argv\r\n    self.execute(*args, **cmd_options)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/django/core/management/base.py\", line 445, in execute\r\n    output = self.handle(*args, **options)\r\n  File \"/Users/sanghan/Projects/memexcadastre/download/management/commands/download.py\", line 44, in handle\r\n    file.download_to_filename(filename)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/storage/blob.py\", line 329, in download_to_filename\r\n    self.download_to_file(file_obj, client=client)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/storage/blob.py\", line 314, in download_to_file\r\n    download.initialize_download(request, client._connection.http)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/transfer.py\", line 347, in initialize_download\r\n    self.bytes_http or http, http_request)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/http_wrapper.py\", line 405, in make_api_request\r\n    check_response_func=check_response_func)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/gcloud/streaming/http_wrapper.py\", line 353, in _make_api_request_no_retry\r\n    redirections=redirections, connection_type=connection_type)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/oauth2client/client.py\", line 622, in new_request\r\n    redirections, connection_type)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py\", line 1609, in request\r\n    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py\", line 1351, in _request\r\n    (response, content) = self._conn_request(conn, request_uri, method, body, headers)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py\", line 1337, in _conn_request\r\n    content = _decompressContent(response, content)\r\n  File \"/Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages/httplib2/__init__.py\", line 410, in _decompressContent\r\n    raise FailedToDecompressContent(_(\"Content purported to be compressed with %s but failed to decompress.\") % response.get('content-encoding'), response, content)\r\nhttplib2.FailedToDecompressContent: Content purported to be compressed with gzip but failed to decompress.\r\n```\r\n\r\nHere is some context on the platform I'm running this on.\r\n\r\n```python\r\n>>> import pkg_resources\r\n>>> pkg_resources.get_build_platform()\r\n'macosx-10.11-x86_64'\r\n```\r\n\r\n```\r\n$ pip show gcloud\r\n\r\nMetadata-Version: 2.0\r\nName: gcloud\r\nVersion: 0.12.0\r\nSummary: API Client library for Google Cloud\r\nHome-page: https://github.com/GoogleCloudPlatform/gcloud-python\r\nAuthor: Google Cloud Platform\r\nAuthor-email: jjg+gcloud-python@google.com\r\nInstaller: pip\r\nLicense: Apache 2.0\r\nLocation: /Users/sanghan/Projects/memexcadastre/venv/lib/python2.7/site-packages\r\nRequires: pyOpenSSL, six, httplib2, protobuf, oauth2client, googleapis-common-protos\r\n```\r\n\r\n```\r\n$ python --version\r\nPython 2.7.11\r\n```\r\n\r\nThe files that do decompress correctly seem to be successful in decompressing, but the ones that fail also fail every time. This also seems to be filesize independent as I have both large and files successfully decompress on download.\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/comments/212206825","html_url":"https://github.com/GoogleCloudPlatform/gcloud-python/issues/1724#issuecomment-212206825","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/issues/1724","id":212206825,"user":{"login":"thobrla","id":3239120,"avatar_url":"https://avatars.githubusercontent.com/u/3239120?v=3","gravatar_id":"","url":"https://api.github.com/users/thobrla","html_url":"https://github.com/thobrla","followers_url":"https://api.github.com/users/thobrla/followers","following_url":"https://api.github.com/users/thobrla/following{/other_user}","gists_url":"https://api.github.com/users/thobrla/gists{/gist_id}","starred_url":"https://api.github.com/users/thobrla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thobrla/subscriptions","organizations_url":"https://api.github.com/users/thobrla/orgs","repos_url":"https://api.github.com/users/thobrla/repos","events_url":"https://api.github.com/users/thobrla/events{/privacy}","received_events_url":"https://api.github.com/users/thobrla/received_events","type":"User","site_admin":false},"created_at":"2016-04-20T01:47:37Z","updated_at":"2016-04-20T01:47:37Z","body":"You can modify the a copy of the gsutil code to set a breakpoint, but the key for gsutil is [overriding httplib2's decompression for downloaded bytes](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/gcs_json_media.py#L554).\r\n\r\nBefore we had this code, we had another approach where we [hashed on the fly before the decompression occurred](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/gcs_json_media.py#L205).  We still need that code for on-the-fly-hashing progress callbacks, but I think the comments around gzip there may be out of date now."}}