{"action":"closed","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/35097","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/35097/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/35097/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/35097/events","html_url":"https://github.com/saltstack/salt/issues/35097","id":168679434,"number":35097,"title":"Minions die with \"un-handled exception from the multiprocessing process\"","user":{"login":"jwhite530","id":1134216,"avatar_url":"https://avatars.githubusercontent.com/u/1134216?v=3","gravatar_id":"","url":"https://api.github.com/users/jwhite530","html_url":"https://github.com/jwhite530","followers_url":"https://api.github.com/users/jwhite530/followers","following_url":"https://api.github.com/users/jwhite530/following{/other_user}","gists_url":"https://api.github.com/users/jwhite530/gists{/gist_id}","starred_url":"https://api.github.com/users/jwhite530/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwhite530/subscriptions","organizations_url":"https://api.github.com/users/jwhite530/orgs","repos_url":"https://api.github.com/users/jwhite530/repos","events_url":"https://api.github.com/users/jwhite530/events{/privacy}","received_events_url":"https://api.github.com/users/jwhite530/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/2016.3.4","name":"2016.3.4","color":"fbca04"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Blocker","name":"Blocker","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Core","name":"Core","color":"fef2c0"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Critical","name":"Critical","color":"e95800"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Fixed%20Pending%20Verification","name":"Fixed Pending Verification","color":"d4c5f9"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P3","name":"P3","color":"0a3d77"},{"url":"https://api.github.com/repos/saltstack/salt/labels/TEAM%20Core","name":"TEAM Core","color":"A9A9A9"}],"state":"closed","locked":false,"assignee":{"login":"rallytime","id":6547906,"avatar_url":"https://avatars.githubusercontent.com/u/6547906?v=3","gravatar_id":"","url":"https://api.github.com/users/rallytime","html_url":"https://github.com/rallytime","followers_url":"https://api.github.com/users/rallytime/followers","following_url":"https://api.github.com/users/rallytime/following{/other_user}","gists_url":"https://api.github.com/users/rallytime/gists{/gist_id}","starred_url":"https://api.github.com/users/rallytime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rallytime/subscriptions","organizations_url":"https://api.github.com/users/rallytime/orgs","repos_url":"https://api.github.com/users/rallytime/repos","events_url":"https://api.github.com/users/rallytime/events{/privacy}","received_events_url":"https://api.github.com/users/rallytime/received_events","type":"User","site_admin":false},"assignees":[{"login":"rallytime","id":6547906,"avatar_url":"https://avatars.githubusercontent.com/u/6547906?v=3","gravatar_id":"","url":"https://api.github.com/users/rallytime","html_url":"https://github.com/rallytime","followers_url":"https://api.github.com/users/rallytime/followers","following_url":"https://api.github.com/users/rallytime/following{/other_user}","gists_url":"https://api.github.com/users/rallytime/gists{/gist_id}","starred_url":"https://api.github.com/users/rallytime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rallytime/subscriptions","organizations_url":"https://api.github.com/users/rallytime/orgs","repos_url":"https://api.github.com/users/rallytime/repos","events_url":"https://api.github.com/users/rallytime/events{/privacy}","received_events_url":"https://api.github.com/users/rallytime/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/92","html_url":"https://github.com/saltstack/salt/milestones/C%202","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/92/labels","id":2014595,"number":92,"title":"C 2","description":"","creator":{"login":"meggiebot","id":12242451,"avatar_url":"https://avatars.githubusercontent.com/u/12242451?v=3","gravatar_id":"","url":"https://api.github.com/users/meggiebot","html_url":"https://github.com/meggiebot","followers_url":"https://api.github.com/users/meggiebot/followers","following_url":"https://api.github.com/users/meggiebot/following{/other_user}","gists_url":"https://api.github.com/users/meggiebot/gists{/gist_id}","starred_url":"https://api.github.com/users/meggiebot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meggiebot/subscriptions","organizations_url":"https://api.github.com/users/meggiebot/orgs","repos_url":"https://api.github.com/users/meggiebot/repos","events_url":"https://api.github.com/users/meggiebot/events{/privacy}","received_events_url":"https://api.github.com/users/meggiebot/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":21,"state":"open","created_at":"2016-09-19T18:14:50Z","updated_at":"2016-10-24T17:50:56Z","due_on":"2016-10-28T00:00:00Z","closed_at":null},"comments":19,"created_at":"2016-08-01T15:52:33Z","updated_at":"2016-10-24T17:50:56Z","closed_at":"2016-10-24T17:50:56Z","body":"### Description of Issue/Question\r\nRecently I have discovered that some of my minions are becoming unresponsive when trying to do anything:\r\n\r\n```\r\ncn42:\r\n    Minion did not return. [No response]\r\ncn11:\r\n    Minion did not return. [No response]\r\ncn58:\r\n    Minion did not return. [No response]\r\ncn59:\r\n    Minion did not return. [No response]\r\n```\r\n\r\nI discovered they log this error then log nothing further until the salt-minion daemon is restarted:\r\n\r\n```\r\n2016-07-30 15:23:24,881 [salt.utils.process][ERROR   ][73716] An un-handled exception from the multiprocessing process 'SignalHandlingMultiprocessingProcess-1:151' was caught:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/process.py\", line 613, in _run\r\n    return self._original_run()\r\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 114, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/schedule.py\", line 652, in handle_func\r\n    with salt.utils.fopen(fn_, 'rb') as fp_:\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/__init__.py\", line 1209, in fopen\r\n    fhandle = open(*args, **kwargs)\r\nIOError: [Errno 2] No such file or directory: '/var/cache/salt/minion/proc/20160730152324736016'\r\n2016-07-30 15:23:25,060 [salt.minion      ][ERROR   ][73716] Exception [Errno 32] Broken pipe occurred in scheduled job\r\n```\r\n\r\nHere's another example from another minion:\r\n\r\n```\r\n2016-07-31 22:01:26,626 [salt.utils.process][ERROR   ][73692] An un-handled exception from the multiprocessing process 'SignalHandlingMultiprocessingProcess-1:182' was caught:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/process.py\", line 613, in _run\r\n    return self._original_run()\r\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 114, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/schedule.py\", line 652, in handle_func\r\n    with salt.utils.fopen(fn_, 'rb') as fp_:\r\n  File \"/usr/lib/python2.7/site-packages/salt/utils/__init__.py\", line 1209, in fopen\r\n    fhandle = open(*args, **kwargs)\r\nIOError: [Errno 2] No such file or directory: '/var/cache/salt/minion/proc/20160731220126606879'\r\n2016-07-31 22:01:26,650 [salt.minion      ][ERROR   ][73692] Exception [Errno 32] Broken pipe occurred in scheduled job\r\n```\r\n\r\n### Setup\r\nI have dozens of minions.  This issue seems to effect them randomly.  I cannot attribute any change to any state, schedule, version, or anything else.  They just started randomly failing.\r\n\r\n### Versions Report\r\nI have not been able to attribute this to a specific version.  All my minions run 2016.3.1 as does my master.\r\n\r\nMaster:\r\n\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n \r\nSystem Versions:\r\n           dist: centos 7.2.1511 Core\r\n        machine: x86_64\r\n        release: 3.10.0-327.4.5.el7.x86_64\r\n         system: Linux\r\n        version: CentOS Linux 7.2.1511 Core\r\n```\r\n\r\nMinion:\r\n\r\n```\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n \r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: Not Installed\r\n       dateutil: 1.5\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.2\r\n        libgit2: Not Installed\r\n        libnacl: Not Installed\r\n       M2Crypto: Not Installed\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.7\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: Not Installed\r\n         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.7.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: Not Installed\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n \r\nSystem Versions:\r\n           dist: centos 7.2.1511 Core\r\n        machine: x86_64\r\n        release: 3.10.0-327.13.1.el7.x86_64\r\n         system: Linux\r\n        version: CentOS Linux 7.2.1511 Core\r\n```"}}
{"actor":{"display_login":"meggiebot"}}