{"action":"created","comment":{"url":"https://api.github.com/repos/saltstack/salt/pulls/comments/231981386","pull_request_review_id":173070917,"id":231981386,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzMTk4MTM4Ng==","diff_hunk":"@@ -905,12 +908,14 @@ def publish(self, load):\n             'Sending payload to publish daemon. jid=%s size=%d',\n             load.get('jid', None), len(payload),\n         )\n-        pub_sock.send(payload)\n+        if msg_queue:\n+            msg_queue.put(payload)\n+        else:\n+            pub_sock.send(payload)\n+            pub_sock.close()","path":"salt/transport/zeromq.py","position":96,"original_position":96,"commit_id":"c5e3360e96de3fcb51eac3801011d58bbdfcaa2a","original_commit_id":"c5e3360e96de3fcb51eac3801011d58bbdfcaa2a","user":{"login":"dwoz","id":1527763,"node_id":"MDQ6VXNlcjE1Mjc3NjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1527763?v=4","gravatar_id":"","url":"https://api.github.com/users/dwoz","html_url":"https://github.com/dwoz","followers_url":"https://api.github.com/users/dwoz/followers","following_url":"https://api.github.com/users/dwoz/following{/other_user}","gists_url":"https://api.github.com/users/dwoz/gists{/gist_id}","starred_url":"https://api.github.com/users/dwoz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwoz/subscriptions","organizations_url":"https://api.github.com/users/dwoz/orgs","repos_url":"https://api.github.com/users/dwoz/repos","events_url":"https://api.github.com/users/dwoz/events{/privacy}","received_events_url":"https://api.github.com/users/dwoz/received_events","type":"User","site_admin":false},"body":"I did try storing the context and socket on the class so it won't get created each time but I still saw the issue. @kingsquirrel152 also talks about trying that as a fix in #36469","created_at":"2018-11-08T17:03:04Z","updated_at":"2018-11-08T17:03:04Z","html_url":"https://github.com/saltstack/salt/pull/50424#discussion_r231981386","pull_request_url":"https://api.github.com/repos/saltstack/salt/pulls/50424","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/saltstack/salt/pulls/comments/231981386"},"html":{"href":"https://github.com/saltstack/salt/pull/50424#discussion_r231981386"},"pull_request":{"href":"https://api.github.com/repos/saltstack/salt/pulls/50424"}},"in_reply_to_id":231949998},"pull_request":{"url":"https://api.github.com/repos/saltstack/salt/pulls/50424","id":229216613,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI5MjE2NjEz","html_url":"https://github.com/saltstack/salt/pull/50424","diff_url":"https://github.com/saltstack/salt/pull/50424.diff","patch_url":"https://github.com/saltstack/salt/pull/50424.patch","issue_url":"https://api.github.com/repos/saltstack/salt/issues/50424","number":50424,"state":"open","locked":false,"title":"Optionally use multiprocessing.Queue for MWorker to publisher","user":{"login":"dwoz","id":1527763,"node_id":"MDQ6VXNlcjE1Mjc3NjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1527763?v=4","gravatar_id":"","url":"https://api.github.com/users/dwoz","html_url":"https://github.com/dwoz","followers_url":"https://api.github.com/users/dwoz/followers","following_url":"https://api.github.com/users/dwoz/following{/other_user}","gists_url":"https://api.github.com/users/dwoz/gists{/gist_id}","starred_url":"https://api.github.com/users/dwoz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwoz/subscriptions","organizations_url":"https://api.github.com/users/dwoz/orgs","repos_url":"https://api.github.com/users/dwoz/repos","events_url":"https://api.github.com/users/dwoz/events{/privacy}","received_events_url":"https://api.github.com/users/dwoz/received_events","type":"User","site_admin":false},"body":"### What does this PR do?\r\n\r\nAdds an option to use a `multiprocessing.Queue` instead of ZeroMQ PUSH/PULL sockets for messages relayed from MWorkers to the publisher daemon process.\r\n\r\n### What issues does this PR fix or reference?\r\n\r\n#36469\r\n#46553\r\n#50393\r\n\r\n### Previous Behavior\r\n\r\nZeroMQ's PUSH/PULL sockets are unreliable when dealing with large messages (size) or large number of messages sent quickly.\r\n\r\n### New Behavior\r\n\r\nOptionally a multiprocessing.Queue for messages moving from MWorkers to the publisher daemon. Messages will get published reliably.\r\n\r\n### Tests written?\r\n\r\nNo - This patch has been tested on via jenkin's branch tests on a fork of 2017.7\r\n\r\n### Commits signed with GPG?\r\n\r\nYes","created_at":"2018-11-08T00:48:44Z","updated_at":"2018-11-08T17:03:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"08fb6bf32e796be90357c035ff1aef7ca8344b6d","assignee":null,"assignees":[],"requested_reviewers":[{"login":"DmitryKuzmenko","id":4807387,"node_id":"MDQ6VXNlcjQ4MDczODc=","avatar_url":"https://avatars0.githubusercontent.com/u/4807387?v=4","gravatar_id":"","url":"https://api.github.com/users/DmitryKuzmenko","html_url":"https://github.com/DmitryKuzmenko","followers_url":"https://api.github.com/users/DmitryKuzmenko/followers","following_url":"https://api.github.com/users/DmitryKuzmenko/following{/other_user}","gists_url":"https://api.github.com/users/DmitryKuzmenko/gists{/gist_id}","starred_url":"https://api.github.com/users/DmitryKuzmenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DmitryKuzmenko/subscriptions","organizations_url":"https://api.github.com/users/DmitryKuzmenko/orgs","repos_url":"https://api.github.com/users/DmitryKuzmenko/repos","events_url":"https://api.github.com/users/DmitryKuzmenko/events{/privacy}","received_events_url":"https://api.github.com/users/DmitryKuzmenko/received_events","type":"User","site_admin":false}],"requested_teams":[{"name":"team-transport","id":2458520,"node_id":"MDQ6VGVhbTI0NTg1MjA=","slug":"team-transport","description":"Transport Layer Reviewers","privacy":"closed","url":"https://api.github.com/teams/2458520","members_url":"https://api.github.com/teams/2458520/members{/member}","repositories_url":"https://api.github.com/teams/2458520/repos","permission":"pull"}],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/saltstack/salt/pulls/50424/commits","review_comments_url":"https://api.github.com/repos/saltstack/salt/pulls/50424/comments","review_comment_url":"https://api.github.com/repos/saltstack/salt/pulls/comments{/number}","comments_url":"https://api.github.com/repos/saltstack/salt/issues/50424/comments","statuses_url":"https://api.github.com/repos/saltstack/salt/statuses/c5e3360e96de3fcb51eac3801011d58bbdfcaa2a","head":{"label":"dwoz:pub_srv_msgs","ref":"pub_srv_msgs","sha":"c5e3360e96de3fcb51eac3801011d58bbdfcaa2a","user":{"login":"dwoz","id":1527763,"node_id":"MDQ6VXNlcjE1Mjc3NjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1527763?v=4","gravatar_id":"","url":"https://api.github.com/users/dwoz","html_url":"https://github.com/dwoz","followers_url":"https://api.github.com/users/dwoz/followers","following_url":"https://api.github.com/users/dwoz/following{/other_user}","gists_url":"https://api.github.com/users/dwoz/gists{/gist_id}","starred_url":"https://api.github.com/users/dwoz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwoz/subscriptions","organizations_url":"https://api.github.com/users/dwoz/orgs","repos_url":"https://api.github.com/users/dwoz/repos","events_url":"https://api.github.com/users/dwoz/events{/privacy}","received_events_url":"https://api.github.com/users/dwoz/received_events","type":"User","site_admin":false},"repo":{"id":116609151,"node_id":"MDEwOlJlcG9zaXRvcnkxMTY2MDkxNTE=","name":"salt","full_name":"dwoz/salt","private":false,"owner":{"login":"dwoz","id":1527763,"node_id":"MDQ6VXNlcjE1Mjc3NjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1527763?v=4","gravatar_id":"","url":"https://api.github.com/users/dwoz","html_url":"https://github.com/dwoz","followers_url":"https://api.github.com/users/dwoz/followers","following_url":"https://api.github.com/users/dwoz/following{/other_user}","gists_url":"https://api.github.com/users/dwoz/gists{/gist_id}","starred_url":"https://api.github.com/users/dwoz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwoz/subscriptions","organizations_url":"https://api.github.com/users/dwoz/orgs","repos_url":"https://api.github.com/users/dwoz/repos","events_url":"https://api.github.com/users/dwoz/events{/privacy}","received_events_url":"https://api.github.com/users/dwoz/received_events","type":"User","site_admin":false},"html_url":"https://github.com/dwoz/salt","description":"Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here: ","fork":true,"url":"https://api.github.com/repos/dwoz/salt","forks_url":"https://api.github.com/repos/dwoz/salt/forks","keys_url":"https://api.github.com/repos/dwoz/salt/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dwoz/salt/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dwoz/salt/teams","hooks_url":"https://api.github.com/repos/dwoz/salt/hooks","issue_events_url":"https://api.github.com/repos/dwoz/salt/issues/events{/number}","events_url":"https://api.github.com/repos/dwoz/salt/events","assignees_url":"https://api.github.com/repos/dwoz/salt/assignees{/user}","branches_url":"https://api.github.com/repos/dwoz/salt/branches{/branch}","tags_url":"https://api.github.com/repos/dwoz/salt/tags","blobs_url":"https://api.github.com/repos/dwoz/salt/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dwoz/salt/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dwoz/salt/git/refs{/sha}","trees_url":"https://api.github.com/repos/dwoz/salt/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dwoz/salt/statuses/{sha}","languages_url":"https://api.github.com/repos/dwoz/salt/languages","stargazers_url":"https://api.github.com/repos/dwoz/salt/stargazers","contributors_url":"https://api.github.com/repos/dwoz/salt/contributors","subscribers_url":"https://api.github.com/repos/dwoz/salt/subscribers","subscription_url":"https://api.github.com/repos/dwoz/salt/subscription","commits_url":"https://api.github.com/repos/dwoz/salt/commits{/sha}","git_commits_url":"https://api.github.com/repos/dwoz/salt/git/commits{/sha}","comments_url":"https://api.github.com/repos/dwoz/salt/comments{/number}","issue_comment_url":"https://api.github.com/repos/dwoz/salt/issues/comments{/number}","contents_url":"https://api.github.com/repos/dwoz/salt/contents/{+path}","compare_url":"https://api.github.com/repos/dwoz/salt/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dwoz/salt/merges","archive_url":"https://api.github.com/repos/dwoz/salt/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dwoz/salt/downloads","issues_url":"https://api.github.com/repos/dwoz/salt/issues{/number}","pulls_url":"https://api.github.com/repos/dwoz/salt/pulls{/number}","milestones_url":"https://api.github.com/repos/dwoz/salt/milestones{/number}","notifications_url":"https://api.github.com/repos/dwoz/salt/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dwoz/salt/labels{/name}","releases_url":"https://api.github.com/repos/dwoz/salt/releases{/id}","deployments_url":"https://api.github.com/repos/dwoz/salt/deployments","created_at":"2018-01-08T00:07:09Z","updated_at":"2018-10-24T19:04:45Z","pushed_at":"2018-11-08T07:09:35Z","git_url":"git://github.com/dwoz/salt.git","ssh_url":"git@github.com:dwoz/salt.git","clone_url":"https://github.com/dwoz/salt.git","svn_url":"https://github.com/dwoz/salt","homepage":"https://repo.saltstack.com/","size":418550,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"saltstack:2017.7","ref":"2017.7","sha":"ef4509e779c1ba9bd00512554e2de14ae394676f","user":{"login":"saltstack","id":1147473,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNDc0NzM=","avatar_url":"https://avatars3.githubusercontent.com/u/1147473?v=4","gravatar_id":"","url":"https://api.github.com/users/saltstack","html_url":"https://github.com/saltstack","followers_url":"https://api.github.com/users/saltstack/followers","following_url":"https://api.github.com/users/saltstack/following{/other_user}","gists_url":"https://api.github.com/users/saltstack/gists{/gist_id}","starred_url":"https://api.github.com/users/saltstack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saltstack/subscriptions","organizations_url":"https://api.github.com/users/saltstack/orgs","repos_url":"https://api.github.com/users/saltstack/repos","events_url":"https://api.github.com/users/saltstack/events{/privacy}","received_events_url":"https://api.github.com/users/saltstack/received_events","type":"Organization","site_admin":false},"repo":{"id":1390248,"node_id":"MDEwOlJlcG9zaXRvcnkxMzkwMjQ4","name":"salt","full_name":"saltstack/salt","private":false,"owner":{"login":"saltstack","id":1147473,"node_id":"MDEyOk9yZ2FuaXphdGlvbjExNDc0NzM=","avatar_url":"https://avatars3.githubusercontent.com/u/1147473?v=4","gravatar_id":"","url":"https://api.github.com/users/saltstack","html_url":"https://github.com/saltstack","followers_url":"https://api.github.com/users/saltstack/followers","following_url":"https://api.github.com/users/saltstack/following{/other_user}","gists_url":"https://api.github.com/users/saltstack/gists{/gist_id}","starred_url":"https://api.github.com/users/saltstack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saltstack/subscriptions","organizations_url":"https://api.github.com/users/saltstack/orgs","repos_url":"https://api.github.com/users/saltstack/repos","events_url":"https://api.github.com/users/saltstack/events{/privacy}","received_events_url":"https://api.github.com/users/saltstack/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/saltstack/salt","description":"Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here: ","fork":false,"url":"https://api.github.com/repos/saltstack/salt","forks_url":"https://api.github.com/repos/saltstack/salt/forks","keys_url":"https://api.github.com/repos/saltstack/salt/keys{/key_id}","collaborators_url":"https://api.github.com/repos/saltstack/salt/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/saltstack/salt/teams","hooks_url":"https://api.github.com/repos/saltstack/salt/hooks","issue_events_url":"https://api.github.com/repos/saltstack/salt/issues/events{/number}","events_url":"https://api.github.com/repos/saltstack/salt/events","assignees_url":"https://api.github.com/repos/saltstack/salt/assignees{/user}","branches_url":"https://api.github.com/repos/saltstack/salt/branches{/branch}","tags_url":"https://api.github.com/repos/saltstack/salt/tags","blobs_url":"https://api.github.com/repos/saltstack/salt/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/saltstack/salt/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/saltstack/salt/git/refs{/sha}","trees_url":"https://api.github.com/repos/saltstack/salt/git/trees{/sha}","statuses_url":"https://api.github.com/repos/saltstack/salt/statuses/{sha}","languages_url":"https://api.github.com/repos/saltstack/salt/languages","stargazers_url":"https://api.github.com/repos/saltstack/salt/stargazers","contributors_url":"https://api.github.com/repos/saltstack/salt/contributors","subscribers_url":"https://api.github.com/repos/saltstack/salt/subscribers","subscription_url":"https://api.github.com/repos/saltstack/salt/subscription","commits_url":"https://api.github.com/repos/saltstack/salt/commits{/sha}","git_commits_url":"https://api.github.com/repos/saltstack/salt/git/commits{/sha}","comments_url":"https://api.github.com/repos/saltstack/salt/comments{/number}","issue_comment_url":"https://api.github.com/repos/saltstack/salt/issues/comments{/number}","contents_url":"https://api.github.com/repos/saltstack/salt/contents/{+path}","compare_url":"https://api.github.com/repos/saltstack/salt/compare/{base}...{head}","merges_url":"https://api.github.com/repos/saltstack/salt/merges","archive_url":"https://api.github.com/repos/saltstack/salt/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/saltstack/salt/downloads","issues_url":"https://api.github.com/repos/saltstack/salt/issues{/number}","pulls_url":"https://api.github.com/repos/saltstack/salt/pulls{/number}","milestones_url":"https://api.github.com/repos/saltstack/salt/milestones{/number}","notifications_url":"https://api.github.com/repos/saltstack/salt/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/saltstack/salt/labels{/name}","releases_url":"https://api.github.com/repos/saltstack/salt/releases{/id}","deployments_url":"https://api.github.com/repos/saltstack/salt/deployments","created_at":"2011-02-20T20:16:56Z","updated_at":"2018-11-08T17:01:29Z","pushed_at":"2018-11-08T17:01:02Z","git_url":"git://github.com/saltstack/salt.git","ssh_url":"git@github.com:saltstack/salt.git","clone_url":"https://github.com/saltstack/salt.git","svn_url":"https://github.com/saltstack/salt","homepage":"https://repo.saltstack.com/","size":420043,"stargazers_count":9366,"watchers_count":9366,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4390,"mirror_url":null,"archived":false,"open_issues_count":2444,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":4390,"open_issues":2444,"watchers":9366,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/saltstack/salt/pulls/50424"},"html":{"href":"https://github.com/saltstack/salt/pull/50424"},"issue":{"href":"https://api.github.com/repos/saltstack/salt/issues/50424"},"comments":{"href":"https://api.github.com/repos/saltstack/salt/issues/50424/comments"},"review_comments":{"href":"https://api.github.com/repos/saltstack/salt/pulls/50424/comments"},"review_comment":{"href":"https://api.github.com/repos/saltstack/salt/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/saltstack/salt/pulls/50424/commits"},"statuses":{"href":"https://api.github.com/repos/saltstack/salt/statuses/c5e3360e96de3fcb51eac3801011d58bbdfcaa2a"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"dwoz"}}