{"action":"created","issue":{"url":"https://api.github.com/repos/wal-e/wal-e/issues/257","repository_url":"https://api.github.com/repos/wal-e/wal-e","labels_url":"https://api.github.com/repos/wal-e/wal-e/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/wal-e/wal-e/issues/257/comments","events_url":"https://api.github.com/repos/wal-e/wal-e/issues/257/events","html_url":"https://github.com/wal-e/wal-e/issues/257","id":167511625,"number":257,"title":"WABS Backup failing with \"unprocessed exception has avoided all error handling\"","user":{"login":"bmleshopventory","id":5365353,"avatar_url":"https://avatars.githubusercontent.com/u/5365353?v=3","gravatar_id":"","url":"https://api.github.com/users/bmleshopventory","html_url":"https://github.com/bmleshopventory","followers_url":"https://api.github.com/users/bmleshopventory/followers","following_url":"https://api.github.com/users/bmleshopventory/following{/other_user}","gists_url":"https://api.github.com/users/bmleshopventory/gists{/gist_id}","starred_url":"https://api.github.com/users/bmleshopventory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmleshopventory/subscriptions","organizations_url":"https://api.github.com/users/bmleshopventory/orgs","repos_url":"https://api.github.com/users/bmleshopventory/repos","events_url":"https://api.github.com/users/bmleshopventory/events{/privacy}","received_events_url":"https://api.github.com/users/bmleshopventory/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-26T03:07:04Z","updated_at":"2016-07-28T20:34:49Z","closed_at":null,"body":"The logs look like this: \r\n\r\n```\r\nJul 25 21:36:26 sv-db1 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (1): svproddbbackup.blob.core.windows.net\r\nJul 25 21:36:27 sv-db1 wal_e.worker.upload INFO     MSG: completed archiving to a file#012        DETAIL: Archiving to \"wabs://db-backup/wal_005/0000000500003313000000D5.00000028.backup.lzo\" complete at 00KiB/s.#012        STRUCTURED: time=2016-07-26T01:36:27.035835-00 pid=28070 action=push-wal key=wabs://db-backup/wal_005/0000000500003313000000D5.00000028.backup.lzo prefix= rate=00 seg=0000000500003313000000D5.00000028.backup state=complete\r\nJul 25 21:36:27 sv-db1 wal_e.blobstore.wabs.wabs_util WARNING  MSG: retrying file upload from unexpected exception#012        DETAIL: The exception type is <class 'select.error'> and its value is (4, 'Interrupted system call') and its traceback is   File \"/usr/local/lib/python2.7/dist-packages/wal_e/retries.py\", line 62, in shim#012            return f(*args, **kwargs)#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/blobstore/wabs/wabs_util.py\", line 90, in upload_chunk#012            block_id, content_md5=check_sum)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/blob/blobservice.py\", line 2366, in put_block#012            self._perform_request(request)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py\", line 171, in _perform_request#012            resp = self._filter(request)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py\", line 160, in _perform_request_worker#012            return self._httpclient.perform_request(request)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/_http/httpclient.py\", line 181, in perform_request#012            self.send_request_body(connection, request.body)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/_http/httpclient.py\", line 143, in send_request_body#012            connection.send(request_body)#012          File \"/usr/local/lib/python2.7/dist-packages/azure/storage/_http/requestsclient.py\", line 81, in send#012            self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)#012          File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 475, in request#012            resp = self.send(prep, **send_kwargs)#012          File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 585, in send#012            r = adapter.send(request, **kwargs)#012          File \"/usr/local/lib/python2.7/dist-packages/requests/adapters.py\", line 403, in send#012            timeout=timeout#012          File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py\", line 578, in urlopen#012            chunked=chunked)#012          File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py\", line 385, in _make_request#012            httplib_response = conn.getresponse(buffering=True)#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/cmd.py\", line 32, in monkey#012            return original(*args, **kwargs)#012          File \"/usr/lib/python2.7/httplib.py\", line 1051, in getresponse#012            response.begin()#012          File \"/usr/lib/python2.7/httplib.py\", line 415, in begin#012            version, status, reason = self._read_status()#012          File \"/usr/lib/python2.7/httplib.py\", line 371, in _read_status#012            line = self.fp.readline(_MAXLINE + 1)#012          File \"/usr/lib/python2.7/socket.py\", line 476, in readline#012            data = self._sock.recv(self._rbufsize)#012          File \"/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py\", line 207, in recv#012            [self.socket], [], [], self.socket.gettimeout())#012          There have been 1 attempts to upload  file wabs://db-backup/basebackups_005/base_0000000500003313000000D5_00000040/tar_partitions/part_00000228.tar.lzo so far.#012        STRUCTURED: time=2016-07-26T01:36:27.306487-00 pid=20812\r\nJul 25 21:36:27 sv-db1 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (2): svproddbbackup.blob.core.windows.net\r\nJul 25 21:36:27 sv-db1 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (3): svproddbbackup.blob.core.windows.net\r\nJul 25 21:36:27 sv-db1 requests.packages.urllib3.connectionpool INFO     Starting new HTTPS connection (4): svproddbbackup.blob.core.windows.net\r\nJul 25 21:36:28 sv-db1 wal_e.main   CRITICAL MSG: An unprocessed exception has avoided all error handling#012        DETAIL: Traceback (most recent call last):#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/cmd.py\", line 578, in main#012            pool_size=args.pool_size)#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/operator/backup.py\", line 197, in database_backup#012            **kwargs)#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/operator/backup.py\", line 502, in _upload_pg_cluster_dir#012            pool.put(tpart)#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload_pool.py\", line 108, in put#012            self._wait()#012          File \"/usr/local/lib/python2.7/dist-packages/wal_e/worker/upload_pool.py\", line 65, in _wait#012            raise val#012        error: (4, 'Interrupted system call')#012        #012        STRUCTURED: time=2016-07-26T01:36:28.635565-00 pid=20812\r\n```\r\n\r\nThe interrupted system call is 4 which seems to indicate a timeout has occurred?"},"comment":{"url":"https://api.github.com/repos/wal-e/wal-e/issues/comments/236016893","html_url":"https://github.com/wal-e/wal-e/issues/257#issuecomment-236016893","issue_url":"https://api.github.com/repos/wal-e/wal-e/issues/257","id":236016893,"user":{"login":"bmleshopventory","id":5365353,"avatar_url":"https://avatars.githubusercontent.com/u/5365353?v=3","gravatar_id":"","url":"https://api.github.com/users/bmleshopventory","html_url":"https://github.com/bmleshopventory","followers_url":"https://api.github.com/users/bmleshopventory/followers","following_url":"https://api.github.com/users/bmleshopventory/following{/other_user}","gists_url":"https://api.github.com/users/bmleshopventory/gists{/gist_id}","starred_url":"https://api.github.com/users/bmleshopventory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmleshopventory/subscriptions","organizations_url":"https://api.github.com/users/bmleshopventory/orgs","repos_url":"https://api.github.com/users/bmleshopventory/repos","events_url":"https://api.github.com/users/bmleshopventory/events{/privacy}","received_events_url":"https://api.github.com/users/bmleshopventory/received_events","type":"User","site_admin":false},"created_at":"2016-07-28T20:34:49Z","updated_at":"2016-07-28T20:34:49Z","body":"It actually does not succeed. We actually hadn't had a backup complete in about 10 days without realizing it and we triggered a backup every night. I switched to the S3 integration and that seems to be going smoothly."}}
{"actor":{"display_login":"bmleshopventory"}}