{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/34215","repository_url":"https://api.github.com/repos/saltstack/salt","labels_url":"https://api.github.com/repos/saltstack/salt/issues/34215/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/34215/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/34215/events","html_url":"https://github.com/saltstack/salt/issues/34215","id":161747211,"number":34215,"title":"salt-master crashes every few days","user":{"login":"rvora","id":2090822,"avatar_url":"https://avatars.githubusercontent.com/u/2090822?v=3","gravatar_id":"","url":"https://api.github.com/users/rvora","html_url":"https://github.com/rvora","followers_url":"https://api.github.com/users/rvora/followers","following_url":"https://api.github.com/users/rvora/following{/other_user}","gists_url":"https://api.github.com/users/rvora/gists{/gist_id}","starred_url":"https://api.github.com/users/rvora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvora/subscriptions","organizations_url":"https://api.github.com/users/rvora/orgs","repos_url":"https://api.github.com/users/rvora/repos","events_url":"https://api.github.com/users/rvora/events{/privacy}","received_events_url":"https://api.github.com/users/rvora/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Info%20Needed","name":"Info Needed","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/37","html_url":"https://github.com/saltstack/salt/milestones/Blocked","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/37/labels","id":343741,"number":37,"title":"Blocked","description":"Issues which are blocked in some way, whether they need further discussion, clarification, justification, or are blocked for some other reason.","creator":{"login":"basepi","id":702318,"avatar_url":"https://avatars.githubusercontent.com/u/702318?v=3","gravatar_id":"","url":"https://api.github.com/users/basepi","html_url":"https://github.com/basepi","followers_url":"https://api.github.com/users/basepi/followers","following_url":"https://api.github.com/users/basepi/following{/other_user}","gists_url":"https://api.github.com/users/basepi/gists{/gist_id}","starred_url":"https://api.github.com/users/basepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/basepi/subscriptions","organizations_url":"https://api.github.com/users/basepi/orgs","repos_url":"https://api.github.com/users/basepi/repos","events_url":"https://api.github.com/users/basepi/events{/privacy}","received_events_url":"https://api.github.com/users/basepi/received_events","type":"User","site_admin":false},"open_issues":455,"closed_issues":908,"state":"open","created_at":"2013-05-28T22:24:12Z","updated_at":"2016-06-28T19:38:06Z","due_on":null,"closed_at":null},"comments":9,"created_at":"2016-06-22T17:59:53Z","updated_at":"2016-06-28T21:17:38Z","closed_at":null,"body":"### Description of Issue/Question\r\nEvery few days (4 or 5 days) salt-master stops responding and we need to restart salt-master.\r\n\r\n### Setup\r\nWe have a number of salt-syndics that connect to this salt-master. Syndics are very old version (2014.7.5). We get a fair number of \"Salt minion claiming to be <redacted> has attempted tocommunicate with the master and could not be verified\" messages in the logs but functionally everything works until salt-master crashes.\r\n\r\n### Steps to Reproduce Issue\r\n\r\n```\r\n2016-06-22 08:31:39,001 [salt.master                                          ][ERROR   ][7815] Unable to decrypt token: Unable to decrypt message\r\n2016-06-22 08:31:39,006 [salt.master                                          ][ERROR   ][7815] Salt minion claiming to be <redacted> has attempted tocommunicate with the master and could not be verified\r\n2016-06-22 08:31:39,006 [salt.master                                          ][WARNING ][7815] Minion id <redacted> is not who it says it is!\r\n2016-06-22 08:31:39,455 [salt.transport.ipc                                   ][ERROR   ][7786] Exception occurred in Subscriber while handling stream: Already reading\r\n2016-06-22 08:31:39,456 [salt.utils.process                                   ][ERROR   ][7786] An un-handled exception from the multiprocessing process 'Reactor-6' was caught:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/process.py\", line 613, in _run\r\n    return self._original_run()\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/reactor.py\", line 215, in run\r\n    for data in self.event.iter_events(full=True):\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/event.py\", line 620, in iter_events\r\n    data = self.get_event(tag=tag, full=full, match_type=match_type)\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/event.py\", line 579, in get_event\r\n    ret = self._get_event(wait, tag, match_func, no_block)\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/event.py\", line 484, in _get_event\r\n    raw = self.subscriber.read_sync(timeout=wait)\r\n  File \"/usr/lib/python2.6/site-packages/salt/transport/ipc.py\", line 654, in read_sync\r\n    return ret_future.result()\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/concurrent.py\", line 214, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/gen.py\", line 230, in wrapper\r\n    yielded = next(result)\r\n  File \"/usr/lib/python2.6/site-packages/salt/transport/ipc.py\", line 631, in _read_sync\r\n    raise exc_to_raise  # pylint: disable=E0702\r\nAssertionError: Already reading\r\n2016-06-22 08:31:39,475 [salt.master                                          ][ERROR   ][7814] Error in function _minion_event:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.6/site-packages/salt/master.py\", line 1540, in run_func\r\n    ret = getattr(self, func)(load)\r\n  File \"/usr/lib/python2.6/site-packages/salt/master.py\", line 1306, in _minion_event\r\n    self.masterapi._minion_event(load)\r\n  File \"/usr/lib/python2.6/site-packages/salt/daemons/masterapi.py\", line 752, in _minion_event\r\n    self.event.fire_event(event, event['tag'])  # old dup event\r\n  File \"/usr/lib/python2.6/site-packages/salt/utils/event.py\", line 662, in fire_event\r\n    self.io_loop.run_sync(lambda: self.pusher.send(msg))\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/ioloop.py\", line 444, in run_sync\r\n    return future_cell[0].result()\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/concurrent.py\", line 214, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/gen.py\", line 876, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/lib/python2.6/site-packages/salt/transport/ipc.py\", line 389, in send\r\n    yield self.stream.write(pack)\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/gen.py\", line 870, in run\r\n    value = future.result()\r\n  File \"/usr/lib64/python2.6/site-packages/tornado/concurrent.py\", line 214, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 3, in raise_exc_info\r\nStreamClosedError\r\n```\r\n\r\nand the StreamClosedError exception keeps repeating about 15 times (within 2 or 3 seconds) and then the log stops completely.\r\n\r\n\r\n### Versions Report\r\n(Provided by running `salt --versions-report`. Please also mention any differences in master/minion versions.)\r\n\r\nSalt-Master (master of masters that crashes every few days):\r\n\r\n```\r\n[root@salty salt]# salt --versions\r\nSalt Version:\r\n           Salt: 2016.3.1\r\n\r\nDependency Versions:\r\n           cffi: Not Installed\r\n       cherrypy: 3.2.2\r\n       dateutil: Not Installed\r\n          gitdb: Not Installed\r\n      gitpython: Not Installed\r\n          ioflo: Not Installed\r\n         Jinja2: 2.7.3\r\n        libgit2: 0.20.0\r\n        libnacl: Not Installed\r\n       M2Crypto: 0.20.2\r\n           Mako: Not Installed\r\n   msgpack-pure: Not Installed\r\n msgpack-python: 0.4.6\r\n   mysql-python: Not Installed\r\n      pycparser: Not Installed\r\n       pycrypto: 2.6.1\r\n         pygit2: 0.20.3\r\n         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)\r\n   python-gnupg: Not Installed\r\n         PyYAML: 3.11\r\n          PyZMQ: 14.5.0\r\n           RAET: Not Installed\r\n          smmap: Not Installed\r\n        timelib: 0.2.4\r\n        Tornado: 4.2.1\r\n            ZMQ: 4.0.5\r\n\r\nSystem Versions:\r\n           dist: centos 6.8 Final\r\n        machine: x86_64\r\n        release: 2.6.32-642.1.1.el6.x86_64\r\n         system: Linux\r\n        version: CentOS 6.8 Final\r\n```\r\n\r\nOn the Salt-syndics with older versions:\r\n\r\n```\r\n               Salt: 2014.7.5\r\n             Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)\r\n             Jinja2: 2.2.1\r\n           M2Crypto: 0.20.2\r\n     msgpack-python: 0.4.6\r\n       msgpack-pure: Not Installed\r\n           pycrypto: 2.0.1\r\n            libnacl: Not Installed\r\n             PyYAML: 3.10\r\n              ioflo: Not Installed\r\n              PyZMQ: 14.3.1\r\n               RAET: Not Installed\r\n                ZMQ: 3.2.5\r\n               Mako: Not Installed\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/229186638","html_url":"https://github.com/saltstack/salt/issues/34215#issuecomment-229186638","issue_url":"https://api.github.com/repos/saltstack/salt/issues/34215","id":229186638,"user":{"login":"rvora","id":2090822,"avatar_url":"https://avatars.githubusercontent.com/u/2090822?v=3","gravatar_id":"","url":"https://api.github.com/users/rvora","html_url":"https://github.com/rvora","followers_url":"https://api.github.com/users/rvora/followers","following_url":"https://api.github.com/users/rvora/following{/other_user}","gists_url":"https://api.github.com/users/rvora/gists{/gist_id}","starred_url":"https://api.github.com/users/rvora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rvora/subscriptions","organizations_url":"https://api.github.com/users/rvora/orgs","repos_url":"https://api.github.com/users/rvora/repos","events_url":"https://api.github.com/users/rvora/events{/privacy}","received_events_url":"https://api.github.com/users/rvora/received_events","type":"User","site_admin":false},"created_at":"2016-06-28T21:17:37Z","updated_at":"2016-06-28T21:17:37Z","body":"I'm using salt-api.\n\nRajul\n\nRajul Vora - typos by Android\nOn Jun 28, 2016 1:58 PM, \"Megan Wilhite\" <notifications@github.com> wrote:\n\n> Also @DrMerlin <https://github.com/DrMerlin> @deuscapturus\n> <https://github.com/deuscapturus> or @rvora <https://github.com/rvora>\n> are all of you using salt-api when this occurs? I'm trying to narrow this\n> down because I have not seen this in any of my environments.\n>\n> Also seems to maybe be correlated to pillars @deuscapturus\n> <https://github.com/deuscapturus> is there anything unique about your\n> pillars? maybe git pillars, external pillar of some sorts? Anything that\n> will help me to replicate this issue.\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/saltstack/salt/issues/34215#issuecomment-229181171>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe/AB_nRnoWPEK7s27m53nHozjI59QnENGRks5qQYrogaJpZM4I8DdA>\n> .\n>\n"}}
{"actor":{"display_login":"rvora"}}