{"action":"created","issue":{"url":"https://api.github.com/repos/fabric/fabric/issues/1533","repository_url":"https://api.github.com/repos/fabric/fabric","labels_url":"https://api.github.com/repos/fabric/fabric/issues/1533/labels{/name}","comments_url":"https://api.github.com/repos/fabric/fabric/issues/1533/comments","events_url":"https://api.github.com/repos/fabric/fabric/issues/1533/events","html_url":"https://github.com/fabric/fabric/issues/1533","id":192745667,"number":1533,"title":"put() does not reconnect in the connection cache","user":{"login":"satosi-k","id":1468149,"avatar_url":"https://avatars.githubusercontent.com/u/1468149?v=3","gravatar_id":"","url":"https://api.github.com/users/satosi-k","html_url":"https://github.com/satosi-k","followers_url":"https://api.github.com/users/satosi-k/followers","following_url":"https://api.github.com/users/satosi-k/following{/other_user}","gists_url":"https://api.github.com/users/satosi-k/gists{/gist_id}","starred_url":"https://api.github.com/users/satosi-k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satosi-k/subscriptions","organizations_url":"https://api.github.com/users/satosi-k/orgs","repos_url":"https://api.github.com/users/satosi-k/repos","events_url":"https://api.github.com/users/satosi-k/events{/privacy}","received_events_url":"https://api.github.com/users/satosi-k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-01T03:27:07Z","updated_at":"2016-12-01T16:38:52Z","closed_at":null,"body":"### Abstract\r\n\r\nI noticed that there is a difference in the behavior of run(), sudo() and put().\r\nIt is whether to reconnect in the connection cache or not.\r\n\r\nrun()/sudo(): reconnect\r\nput(): does NOT reconnect\r\n\r\nImportant point: This issue occurs only when using Fabric as a library.\r\n\r\n### Steps to reproduce\r\n\r\n#### run()/sudo(): reconnect\r\n\r\nFirst of all, run()/sudo() behavior is the following.\r\nThis is expected behavior.\r\n\r\n1: Execute any command using run() or sudo().\r\n(Vagrant is used but this is not essentially)\r\n```\r\n>>> from fabric.api import run\r\n>>> from fabric.api import settings\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     run('exit 0')\r\n... \r\n[127.0.0.1:2222] run: exit 0\r\n''\r\n```\r\n\r\n2: Shutdown the remote host\r\n```\r\n$ vagrant halt\r\n```\r\n\r\n3: Retry execution (failure is expected, this is no problem)\r\n```\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     run('exit 0')\r\n... \r\n[127.0.0.1:2222] run: exit 0\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 2, in <module>\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 677, in host_prompting_wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/operations.py\", line 1088, in run\r\n    shell_escape=shell_escape, capture_buffer_size=capture_buffer_size,\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/operations.py\", line 928, in _run_command\r\n    channel=default_channel(), command=wrapped_command, pty=pty,\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/state.py\", line 429, in default_channel\r\n    chan = _open_session()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/state.py\", line 416, in _open_session\r\n    return connections[env.host_string].get_transport().open_session()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 159, in __getitem__\r\n    self.connect(key)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 151, in connect\r\n    user, host, port, cache=self, seek_gateway=seek_gateway)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 603, in connect\r\n    raise NetworkError(msg, e)\r\nfabric.exceptions.NetworkError: Low level socket error connecting to host 127.0.0.1 on port 2222: Unable to connect to port 2222 on 127.0.0.1 (tried 1 time)\r\n```\r\n\r\nWhen this timing, the connection cache turn into not active.\r\n```\r\n>>> from fabric.state import connections\r\n>>> connections\r\n{'vagrant@127.0.0.1:2222': <paramiko.client.SSHClient object at 0x10ed5d4d0>}\r\n>>> connections['vagrant@127.0.0.1:2222']._transport.active\r\nFalse\r\n```\r\n\r\n4: Boot the remote host\r\n```\r\n$ vagrant up\r\n```\r\n\r\n5: Retry execution\r\n```\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     run('exit 0')\r\n... \r\n[127.0.0.1:2222] run: exit 0\r\n''\r\n```\r\n\r\nRetry succeed moreover the connection cache also activated.\r\n```\r\n>>> connections['vagrant@127.0.0.1:2222']._transport.active\r\nTrue\r\n```\r\n\r\nThe above is run()/sudo() behavior. This is expected.\r\n\r\n#### put(): does NOT reconnect\r\n\r\nNext, put() behavior.\r\n\r\n1: Prepare sample file\r\n```\r\n$ touch sample.txt\r\n```\r\n\r\n2: Copy the file to the remote host\r\n```\r\n>>> from fabric.api import put\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     put(local_path='sample.txt', remote_path='/tmp')\r\n... \r\n[127.0.0.1:2222] put: sample.txt -> /tmp/sample.txt\r\n['/tmp/sample.txt']\r\n```\r\n\r\nWhen this timing, Of course the connection cache is active.\r\n```\r\n>>> connections['vagrant@127.0.0.1:2222']._transport.active\r\nTrue\r\n```\r\n\r\n3: Shutdown the remote host\r\n```\r\n$ vagrant halt\r\n```\r\n\r\n4: Retry execution (failure is expected, this is no problem)\r\n```\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     put(local_path='sample.txt', remote_path='/tmp')\r\n... \r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 2, in <module>\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 677, in host_prompting_wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/operations.py\", line 345, in put\r\n    ftp = SFTP(env.host_string)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/sftp.py\", line 33, in __init__\r\n    self.ftp = connections[host_string].open_sftp()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/client.py\", line 458, in open_sftp\r\n    return self._transport.open_sftp_client()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 907, in open_sftp_client\r\n    return SFTPClient.from_transport(self)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/sftp_client.py\", line 128, in from_transport\r\n    max_packet_size=max_packet_size)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 704, in open_session\r\n    timeout=timeout)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 788, in open_channel\r\n    raise SSHException('SSH session not active')\r\nparamiko.ssh_exception.SSHException: SSH session not active\r\n```\r\n\r\nWhen this timing, the connection cache turn into not active.\r\n```\r\n>>> connections['vagrant@127.0.0.1:2222']._transport.active\r\nFalse\r\n```\r\n\r\n5: Boot the remote host\r\n```\r\n$ vagrant up\r\n```\r\n\r\n6: Retry execution (Oops! failure has continued...)\r\n```\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     put(local_path='sample.txt', remote_path='/tmp')\r\n... \r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 2, in <module>\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/network.py\", line 677, in host_prompting_wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/operations.py\", line 345, in put\r\n    ftp = SFTP(env.host_string)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/fabric/sftp.py\", line 33, in __init__\r\n    self.ftp = connections[host_string].open_sftp()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/client.py\", line 458, in open_sftp\r\n    return self._transport.open_sftp_client()\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 907, in open_sftp_client\r\n    return SFTPClient.from_transport(self)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/sftp_client.py\", line 128, in from_transport\r\n    max_packet_size=max_packet_size)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 704, in open_session\r\n    timeout=timeout)\r\n  File \"/Users/satoshi-k/.virtualenvs/fabric/lib/python2.7/site-packages/paramiko/transport.py\", line 788, in open_channel\r\n    raise SSHException('SSH session not active')\r\nparamiko.ssh_exception.SSHException: SSH session not active\r\n```\r\n\r\nthe connection cache also remains inactive.\r\n```\r\n>>> connections['vagrant@127.0.0.1:2222']._transport.active\r\nFalse\r\n```\r\n\r\n### My humble opinion\r\n\r\nThe above behavior is not intuitive and to be the cause of bugs.\r\n\r\n### Workaround\r\n\r\nWorkaround is to clear the connection cache.\r\nThe cause of this issue is that put() does not reconnect.\r\nTherefore, If the connection cache cleared, the issue will not occur.\r\n\r\n1: Clear the connection cache\r\n```\r\n>>> from fabric.network import disconnect_all\r\n>>> disconnect_all()\r\nDisconnecting from vagrant@127.0.0.1:2222... done.\r\n```\r\n\r\n2: Retry execution (It succeed)\r\n```\r\n>>> with settings(user='vagrant', password='vagrant', host_string='127.0.0.1:2222', warn_only=True):\r\n...     put(local_path='sample.txt', remote_path='/tmp')\r\n... \r\n[127.0.0.1:2222] put: sample.txt -> /tmp/sample.txt\r\n['/tmp/sample.txt']\r\n```\r\n\r\n### My environment\r\n\r\n```\r\n$ pip list --format=columns\r\nPackage    Version\r\n---------- -------\r\necdsa      0.13   \r\nFabric     1.12.0 \r\nparamiko   1.17.2 \r\npip        9.0.1  \r\npycrypto   2.6.1  \r\nsetuptools 29.0.1 \r\nwheel      0.29.0 \r\n$ sw_vers \r\nProductName:\tMac OS X\r\nProductVersion:\t10.11.6\r\nBuildVersion:\t15G1108\r\n$ python --version\r\nPython 2.7.10\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/fabric/fabric/issues/comments/264223539","html_url":"https://github.com/fabric/fabric/issues/1533#issuecomment-264223539","issue_url":"https://api.github.com/repos/fabric/fabric/issues/1533","id":264223539,"user":{"login":"jconard3","id":6401914,"avatar_url":"https://avatars.githubusercontent.com/u/6401914?v=3","gravatar_id":"","url":"https://api.github.com/users/jconard3","html_url":"https://github.com/jconard3","followers_url":"https://api.github.com/users/jconard3/followers","following_url":"https://api.github.com/users/jconard3/following{/other_user}","gists_url":"https://api.github.com/users/jconard3/gists{/gist_id}","starred_url":"https://api.github.com/users/jconard3/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jconard3/subscriptions","organizations_url":"https://api.github.com/users/jconard3/orgs","repos_url":"https://api.github.com/users/jconard3/repos","events_url":"https://api.github.com/users/jconard3/events{/privacy}","received_events_url":"https://api.github.com/users/jconard3/received_events","type":"User","site_admin":false},"created_at":"2016-12-01T16:38:52Z","updated_at":"2016-12-01T16:38:52Z","body":"I've been successful in recreating this issue with put() as well as get() by manually setting `connection[...]._transport.active` to `False`. It definitely seems like `run()` will ignore the cache during these conditions. Taking a quick look at `put()`'s method and the traceback makes me suspect that the issue here lies in the SFTP method. Next step is to investigate `open_sftp()` for further clues."}}
{"actor":{"display_login":"jconard3"}}