{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/34509","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/34509/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/34509/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/34509/events","html_url":"https://github.com/saltstack/salt/issues/34509","id":164301481,"number":34509,"title":"No atomic  thin.tgz deploy","user":{"login":"srkunze","id":1389648,"avatar_url":"https://avatars.githubusercontent.com/u/1389648?v=3","gravatar_id":"","url":"https://api.github.com/users/srkunze","html_url":"https://github.com/srkunze","followers_url":"https://api.github.com/users/srkunze/followers","following_url":"https://api.github.com/users/srkunze/following{/other_user}","gists_url":"https://api.github.com/users/srkunze/gists{/gist_id}","starred_url":"https://api.github.com/users/srkunze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srkunze/subscriptions","organizations_url":"https://api.github.com/users/srkunze/orgs","repos_url":"https://api.github.com/users/srkunze/repos","events_url":"https://api.github.com/users/srkunze/events{/privacy}","received_events_url":"https://api.github.com/users/srkunze/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Core","name":"Core","color":"fef2c0"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Fixed%20Pending%20Verification","name":"Fixed Pending Verification","color":"d4c5f9"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P2","name":"P2","color":"155fb5"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Salt-SSH","name":"Salt-SSH","color":"006b75"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":3167,"closed_issues":3944,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2016-08-25T22:01:19Z","due_on":null,"closed_at":null},"comments":23,"created_at":"2016-07-07T12:51:35Z","updated_at":"2016-08-26T13:00:47Z","closed_at":null,"body":"It appears to me that the deploy of the salt-ssh thin.tgz is not atomic and very sensitive to environmental influences. The following minion has a bad network connectivity.\r\n\r\n\r\n### First run:\r\n```\r\nminion:\r\n    ----------\r\n    retcode:\r\n        255\r\n    stderr:\r\n        Connection to <IP> timed out while waiting to read\r\n    stdout:\r\n```\r\n\r\n### Second run (which can also occur randomly after deleting the .root_dcdf8c_salt dir)\r\n```\r\nminion:\r\n    ----------\r\n    retcode:\r\n        12\r\n    stderr:\r\n        1f8c882cd3d9c0813c2b44aec0873eebaf20bf11\r\n        6e862ca205fbf41b72be7729b685f23fa7b104a2\r\n        WARNING: checksum mismatch for \"/var/tmp/.root_dcdf8c_salt/salt-thin.tgz\"\r\n    stdout:\r\n        ERROR: Failure deploying thin: \r\n        1f8c882cd3d9c0813c2b44aec0873eebaf20bf11\r\n        6e862ca205fbf41b72be7729b685f23fa7b104a2\r\n        WARNING: checksum mismatch for \"/var/tmp/.root_dcdf8c_salt/salt-thin.tgz\"\r\n```\r\n\r\n### Alternative second run\r\n```\r\nminion:\r\n    ----------\r\n    retcode:\r\n        11\r\n    stderr:\r\n        WARNING: Unable to locate current thin  version: /var/tmp/.root_dcdf8c_salt/version.\r\n    stdout:\r\n        ERROR: Failure deploying thin: /usr/bin/scp\r\n        _edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878\r\n        deploy\r\n        \r\n        WARNING: Unable to locate current thin  version: /var/tmp/.root_dcdf8c_salt/version.\r\n```\r\n\r\n```\r\nsalt:/srv # salt-ssh --versions-report\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: 0.8.6\r\n       cherrypy: Not Installed\r\n       dateutil: 2.1\r\n          gitdb: 0.5.4\r\n      gitpython: 0.3.2 RC1\r\n          ioflo: Not Installed\r\n         Jinja2: 2.8\r\n        libgit2: 0.21.0\r\n        libnacl: Not Installed\r\n       M2Crypto: 0.22\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.7\r\n   mysql-python: Not Installed\r\n      pycparser: 2.10\r\n       pycrypto: 2.6.1\r\n         pygit2: 0.21.2\r\n         Python: 2.7.3 (default, Apr 14 2012, 08:58:41) [GCC]\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 15.2.0\r\n           RAET: Not Installed\r\n          smmap: 0.8.2\r\n        timelib: Not Installed\r\n        Tornado: 4.3\r\n            ZMQ: 4.1.2\r\n \r\nSystem Versions:\r\n           dist: SuSE 12.3 x86_64\r\n        machine: x86_64\r\n        release: 3.7.10-1.40-desktop\r\n         system: Linux\r\n        version: openSUSE  12.3 x86_64\r\n```\r\n\r\n\r\nNOTE: sometimes, it's a single tgz, sometimes there are two tgz in this directory and sometimes, there a lot of files.\r\n\r\n\r\nWhy does salt not recover here automatically?"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/242728833","html_url":"https://github.com/saltstack/salt/issues/34509#issuecomment-242728833","issue_url":"https://api.github.com/repos/saltstack/salt/issues/34509","id":242728833,"user":{"login":"srkunze","id":1389648,"avatar_url":"https://avatars.githubusercontent.com/u/1389648?v=3","gravatar_id":"","url":"https://api.github.com/users/srkunze","html_url":"https://github.com/srkunze","followers_url":"https://api.github.com/users/srkunze/followers","following_url":"https://api.github.com/users/srkunze/following{/other_user}","gists_url":"https://api.github.com/users/srkunze/gists{/gist_id}","starred_url":"https://api.github.com/users/srkunze/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srkunze/subscriptions","organizations_url":"https://api.github.com/users/srkunze/orgs","repos_url":"https://api.github.com/users/srkunze/repos","events_url":"https://api.github.com/users/srkunze/events{/privacy}","received_events_url":"https://api.github.com/users/srkunze/received_events","type":"User","site_admin":false},"created_at":"2016-08-26T13:00:47Z","updated_at":"2016-08-26T13:00:47Z","body":"I tried more things out. Hope that this will help you to find the right spot.\r\n\r\nWe have several groups of minions. Depending on the number of deployment failures, the largest group does not recover at all after repeated execution. When \"healing\" some minions minions by running them alone, after a critical mass is reached, the largest group can recover completely. (example see below)\r\n\r\nThe smallest group needs only 2 or 3 rounds of re-running to recover completely.\r\n\r\nI also noticed that only a fixed number of minions can recover per run. 4 seems to be the max for all groups (but it can be less, even 0).\r\n\r\n\r\n----\r\n\r\n```\r\nsalt-ssh 'large*' test.ping # 20x without recovery\r\n\r\nsalt-ssh large1 test.ping\r\nsalt-ssh large2 test.ping\r\n.... # etc.\r\n\r\nsalt-ssh 'large*' test.ping # 5x then completely recovered\r\n```"}}
{"actor":{"display_login":"srkunze"}}