{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/35097","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/35097/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/35097/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/35097/events","html_url":"https://github.com/saltstack/salt/issues/35097","id":168679434,"number":35097,"title":"Minions die with \"un-handled exception from the multiprocessing process\"","user":{"login":"jwhite530","id":1134216,"avatar_url":"https://avatars.githubusercontent.com/u/1134216?v=3","gravatar_id":"","url":"https://api.github.com/users/jwhite530","html_url":"https://github.com/jwhite530","followers_url":"https://api.github.com/users/jwhite530/followers","following_url":"https://api.github.com/users/jwhite530/following{/other_user}","gists_url":"https://api.github.com/users/jwhite530/gists{/gist_id}","starred_url":"https://api.github.com/users/jwhite530/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwhite530/subscriptions","organizations_url":"https://api.github.com/users/jwhite530/orgs","repos_url":"https://api.github.com/users/jwhite530/repos","events_url":"https://api.github.com/users/jwhite530/events{/privacy}","received_events_url":"https://api.github.com/users/jwhite530/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Pending%20Discussion","name":"Pending Discussion","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","html_url":"https://github.com/saltstack/salt/milestones/Blocked","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":590,"closed_issues":1049,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2016-10-03T19:23:11Z","due_on":null,"closed_at":null},"comments":15,"created_at":"2016-08-01T15:52:33Z","updated_at":"2016-10-04T12:28:23Z","closed_at":null,"body":"### Description of Issue/Question\r\nRecently I have discovered that some of my minions are becoming unresponsive when trying to do anything:\r\n\r\n```\r\ncn42:\r\n    Minion did not return. [No response]\r\ncn11:\r\n    Minion did not return. [No response]\r\ncn58:\r\n    Minion did not return. [No response]\r\ncn59:\r\n    Minion did not return. [No response]\r\n```\r\n\r\nI discovered they log this error then log nothing further until the salt-minion daemon is restarted:\r\n\r\n```\r\n2016-07-30 15:23:24,881 [salt.utils.process][ERROR   ][73716] An un-handled exception from the multiprocessing process 'SignalHandlingMultiprocessingProcess-1:151' was caught:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/process.py\", line 613, in _run\r\n    return self._original_run()\r\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 114, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/schedule.py\", line 652, in handle_func\r\n    with salt.utils.fopen(fn_, 'rb') as fp_:\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/__init__.py\", line 1209, in fopen\r\n    fhandle = open(*args, **kwargs)\r\nIOError: [Errno 2] No such file or directory: '/var/cache/salt/minion/proc/20160730152324736016'\r\n2016-07-30 15:23:25,060 [salt.minion      ][ERROR   ][73716] Exception [Errno 32] Broken pipe occurred in scheduled job\r\n```\r\n\r\nHere's another example from another minion:\r\n\r\n```\r\n2016-07-31 22:01:26,626 [salt.utils.process][ERROR   ][73692] An un-handled exception from the multiprocessing process 'SignalHandlingMultiprocessingProcess-1:182' was caught:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/process.py\", line 613, in _run\r\n    return self._original_run()\r\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 114, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/schedule.py\", line 652, in handle_func\r\n    with salt.utils.fopen(fn_, 'rb') as fp_:\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/__init__.py\", line 1209, in fopen\r\n    fhandle = open(*args, **kwargs)\r\nIOError: [Errno 2] No such file or directory: '/var/cache/salt/minion/proc/20160731220126606879'\r\n2016-07-31 22:01:26,650 [salt.minion      ][ERROR   ][73692] Exception [Errno 32] Broken pipe occurred in scheduled job\r\n```\r\n\r\n### Setup\r\nI have dozens of minions.  This issue seems to effect them randomly.  I cannot attribute any change to any state, schedule, version, or anything else.  They just started randomly failing.\r\n\r\n### Versions Report\r\nI have not been able to attribute this to a specific version.  All my minions run 2016.3.1 as does my master.\r\n\r\nMaster:\r\n\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n \r\nSystem Versions:\r\n           dist: centos 7.2.1511 Core\r\n        machine: x86_64\r\n        release: 3.10.0-327.4.5.el7.x86_64\r\n         system: Linux\r\n        version: CentOS Linux 7.2.1511 Core\r\n```\r\n\r\nMinion:\r\n\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.7\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n \r\nSystem Versions:\r\n           dist: centos 7.2.1511 Core\r\n        machine: x86_64\r\n        release: 3.10.0-327.13.1.el7.x86_64\r\n         system: Linux\r\n        version: CentOS Linux 7.2.1511 Core\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/251373883","html_url":"https://github.com/saltstack/salt/issues/35097#issuecomment-251373883","issue_url":"https://api.github.com/repos/saltstack/salt/issues/35097","id":251373883,"user":{"login":"diego-treitos","id":644696,"avatar_url":"https://avatars.githubusercontent.com/u/644696?v=3","gravatar_id":"","url":"https://api.github.com/users/diego-treitos","html_url":"https://github.com/diego-treitos","followers_url":"https://api.github.com/users/diego-treitos/followers","following_url":"https://api.github.com/users/diego-treitos/following{/other_user}","gists_url":"https://api.github.com/users/diego-treitos/gists{/gist_id}","starred_url":"https://api.github.com/users/diego-treitos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diego-treitos/subscriptions","organizations_url":"https://api.github.com/users/diego-treitos/orgs","repos_url":"https://api.github.com/users/diego-treitos/repos","events_url":"https://api.github.com/users/diego-treitos/events{/privacy}","received_events_url":"https://api.github.com/users/diego-treitos/received_events","type":"User","site_admin":false},"created_at":"2016-10-04T12:28:23Z","updated_at":"2016-10-04T12:28:23Z","body":"I dont have `multiprocessing: False` in any minion.\r\n\r\nThe last one I got using 100% of the CPU was dumping this in the logs:\r\n\r\n```\r\n2016-03-28 18:38:20,198 [salt.scripts     ][ERROR   ][17987] Attempt to authenticate with the salt master failed\r\n2016-03-28 18:38:20,200 [salt.scripts     ][WARNING ][17987] ** Restarting minion **\r\n```\r\nThe \"funny\" thing is that this minion is a version `2015.5.3 (Lithium)` with a master `2016.3.2 (Boron)`.\r\n\r\nOn the other hand I could execute commands there. Actually I am finding the minions using 100% of cpu by using `salt '*' cmd.run uptime` so I can see the load of the server.\r\n\r\nNot sure anymore if this is the same problem or an additional one."}}
{"actor":{"display_login":"diego-treitos"}}