{"action":"created","issue":{"url":"https://api.github.com/repos/saltstack/salt/issues/27421","labels_url":"https://api.github.com/repos/saltstack/salt/issues/27421/labels{/name}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/27421/comments","events_url":"https://api.github.com/repos/saltstack/salt/issues/27421/events","html_url":"https://github.com/saltstack/salt/issues/27421","id":108397950,"number":27421,"title":"exhausting all the worker threads on the master with publish","user":{"login":"mimianddaniel","id":11415415,"avatar_url":"https://avatars.githubusercontent.com/u/11415415?v=3","gravatar_id":"","url":"https://api.github.com/users/mimianddaniel","html_url":"https://github.com/mimianddaniel","followers_url":"https://api.github.com/users/mimianddaniel/followers","following_url":"https://api.github.com/users/mimianddaniel/following{/other_user}","gists_url":"https://api.github.com/users/mimianddaniel/gists{/gist_id}","starred_url":"https://api.github.com/users/mimianddaniel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mimianddaniel/subscriptions","organizations_url":"https://api.github.com/users/mimianddaniel/orgs","repos_url":"https://api.github.com/users/mimianddaniel/repos","events_url":"https://api.github.com/users/mimianddaniel/events{/privacy}","received_events_url":"https://api.github.com/users/mimianddaniel/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/saltstack/salt/labels/Bug","name":"Bug","color":"e11d21"},{"url":"https://api.github.com/repos/saltstack/salt/labels/Core","name":"Core","color":"fef2c0"},{"url":"https://api.github.com/repos/saltstack/salt/labels/High%20Severity","name":"High Severity","color":"ff9143"},{"url":"https://api.github.com/repos/saltstack/salt/labels/P2","name":"P2","color":"155fb5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/saltstack/salt/milestones/13","html_url":"https://github.com/saltstack/salt/milestones/Approved","labels_url":"https://api.github.com/repos/saltstack/salt/milestones/13/labels","id":9265,"number":13,"title":"Approved","description":"All issues that are ready to be worked on, both bugs and features.","creator":{"login":"thatch45","id":507599,"avatar_url":"https://avatars.githubusercontent.com/u/507599?v=3","gravatar_id":"","url":"https://api.github.com/users/thatch45","html_url":"https://github.com/thatch45","followers_url":"https://api.github.com/users/thatch45/followers","following_url":"https://api.github.com/users/thatch45/following{/other_user}","gists_url":"https://api.github.com/users/thatch45/gists{/gist_id}","starred_url":"https://api.github.com/users/thatch45/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatch45/subscriptions","organizations_url":"https://api.github.com/users/thatch45/orgs","repos_url":"https://api.github.com/users/thatch45/repos","events_url":"https://api.github.com/users/thatch45/events{/privacy}","received_events_url":"https://api.github.com/users/thatch45/received_events","type":"User","site_admin":false},"open_issues":2818,"closed_issues":3151,"state":"open","created_at":"2011-05-14T04:00:56Z","updated_at":"2016-01-06T21:31:56Z","due_on":null,"closed_at":null},"comments":29,"created_at":"2015-09-25T20:00:44Z","updated_at":"2016-01-06T21:32:58Z","closed_at":null,"body":"running 2015.5.5, with master and syndics in the topology \r\nBased on my testing, if i have submit more events than the number of mworkers, I get the following error. \r\nI have already upped my mworkers numbers and tested at 128 threads. It looks like salt is able to handle ~128 events published at once but beyond that it gets the error below. \r\n\r\nSubmitted over ~300 publish events at once caused the following,\r\n```python-traceback\r\n015-09-25 21:51:34,553 [salt.payload                             ][INFO    ][16240] SaltReqTimeoutError: after 15 seconds. (Try 1 of 3)\r\n2015-09-25 21:51:35,136 [salt.payload                             ][INFO    ][15983] SaltReqTimeoutError: after 15 seconds. (Try 1 of 3)\r\n<snip>\r\nTraceback (most recent call last):\r\n  File \"/opt/blue-python/2.7/lib/python2.7/site-packages/salt/master.py\", line 1460, in run_func\r\n    ret = getattr(self, func)(load)\r\n  File \"/opt/blue-python/2.7/lib/python2.7/site-packages/salt/master.py\", line 1391, in minion_pub\r\n    return self.masterapi.minion_pub(clear_load)\r\n  File \"/opt/blue-python/2.7/lib/python2.7/site-packages/salt/daemons/masterapi.py\", line 874, in minion_pub\r\n    ret['jid'] = self.local.cmd_async(**pub_load)\r\n  File \"/opt/blue-python/2.7/lib/python2.7/site-packages/salt/client/__init__.py\", line 333, in cmd_async\r\n    **kwargs)\r\n  File \"/opt/blue-python/2.7/lib/python2.7/site-packages/salt/client/__init__.py\", line 299, in run_job\r\n    raise SaltClientError(general_exception)\r\nSaltClientError: Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased\r\n```"},"comment":{"url":"https://api.github.com/repos/saltstack/salt/issues/comments/169469640","html_url":"https://github.com/saltstack/salt/issues/27421#issuecomment-169469640","issue_url":"https://api.github.com/repos/saltstack/salt/issues/27421","id":169469640,"user":{"login":"cachedout","id":111616,"avatar_url":"https://avatars.githubusercontent.com/u/111616?v=3","gravatar_id":"","url":"https://api.github.com/users/cachedout","html_url":"https://github.com/cachedout","followers_url":"https://api.github.com/users/cachedout/followers","following_url":"https://api.github.com/users/cachedout/following{/other_user}","gists_url":"https://api.github.com/users/cachedout/gists{/gist_id}","starred_url":"https://api.github.com/users/cachedout/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cachedout/subscriptions","organizations_url":"https://api.github.com/users/cachedout/orgs","repos_url":"https://api.github.com/users/cachedout/repos","events_url":"https://api.github.com/users/cachedout/events{/privacy}","received_events_url":"https://api.github.com/users/cachedout/received_events","type":"User","site_admin":false},"created_at":"2016-01-06T21:32:57Z","updated_at":"2016-01-06T21:32:57Z","body":"@mimianddaniel @SEJeff and @plastikos \r\n\r\nI wanted to circle back around on this one. I have marked #27606 as a backport for the upcoming release of 2015.8 but I wanted to know if anybody has had a chance to try these changes in production?"}}