{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/1468","repository_url":"https://api.github.com/repos/flux-framework/flux-core","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1468/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1468/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1468/events","html_url":"https://github.com/flux-framework/flux-core/issues/1468","id":314356531,"number":1468,"title":"wreck permission denied feeds into dangling ROUTER DEALER and PAIR sockets and death","user":{"login":"trws","id":660149,"avatar_url":"https://avatars1.githubusercontent.com/u/660149?v=4","gravatar_id":"","url":"https://api.github.com/users/trws","html_url":"https://github.com/trws","followers_url":"https://api.github.com/users/trws/followers","following_url":"https://api.github.com/users/trws/following{/other_user}","gists_url":"https://api.github.com/users/trws/gists{/gist_id}","starred_url":"https://api.github.com/users/trws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trws/subscriptions","organizations_url":"https://api.github.com/users/trws/orgs","repos_url":"https://api.github.com/users/trws/repos","events_url":"https://api.github.com/users/trws/events{/privacy}","received_events_url":"https://api.github.com/users/trws/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-04-14T19:47:05Z","updated_at":"2018-04-15T16:26:02Z","closed_at":null,"author_association":"OWNER","body":"I have *no idea* why this happened, but this is the death output from a 1500 node run that died this morning.  External evidence makes me think that one or more nodes hung, causing literally thousands of jobs to sit in either submitted or runrequest for hours, then unhung and scheduled all of those in about 5 seconds, printed this, and died an unceremonious death.  I have the full (but rather voluminous) logfile if anyone wants it.\r\n\r\n```\r\nbroker: entering event loop\r\nbroker: entering event loop\r\ninitial load complete, reloading sched\r\nsched reload complete, preparing environment\r\n37086:owner\r\n59021:owner\r\n25332:owner\r\n37084:owner\r\n59751:owner\r\nENV READY\r\nlaunching shell without PTY\r\necho $FLUX_URI\r\nlocal:///var/tmp/flux-lOf9SM\r\nwrexecd exec: Permission denied\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'ROUTER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:478\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'DEALER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:549\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:21 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:21 dangling sockets: cannot terminate ZMQ safely\r\nwrexecd exec: Permission denied\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'ROUTER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:478\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'DEALER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:549\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:23 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:23 dangling sockets: cannot terminate ZMQ safely\r\nwrexecd exec: Permission denied\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'ROUTER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:478\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'DEALER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:549\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:24 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:24 dangling sockets: cannot terminate ZMQ safely\r\nwrexecd exec: Permission denied\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'ROUTER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:478\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'DEALER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:549\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:25 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:25 dangling sockets: cannot terminate ZMQ safely\r\nwrexecd exec: Permission denied\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'ROUTER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:478\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'DEALER' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/overlay.c:549\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/broker/module.c:571\r\nE: (flux-broker) 18-04-14 10:36:30 dangling 'PAIR' socket created at /g/g12/scogland/projects/flux/flux-core/src/connectors/shmem/shmem.c:259\r\nE: (flux-broker) 18-04-14 10:36:30 dangling sockets: cannot terminate ZMQ safely\r\n[mpiexec@sierra503] control_cb (pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed\r\n[mpiexec@sierra503] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status\r\n[mpiexec@sierra503] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event\r\n[mpiexec@sierra503] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion\r\n```"},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/381419053","html_url":"https://github.com/flux-framework/flux-core/issues/1468#issuecomment-381419053","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1468","id":381419053,"user":{"login":"garlick","id":169947,"avatar_url":"https://avatars1.githubusercontent.com/u/169947?v=4","gravatar_id":"","url":"https://api.github.com/users/garlick","html_url":"https://github.com/garlick","followers_url":"https://api.github.com/users/garlick/followers","following_url":"https://api.github.com/users/garlick/following{/other_user}","gists_url":"https://api.github.com/users/garlick/gists{/gist_id}","starred_url":"https://api.github.com/users/garlick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garlick/subscriptions","organizations_url":"https://api.github.com/users/garlick/orgs","repos_url":"https://api.github.com/users/garlick/repos","events_url":"https://api.github.com/users/garlick/events{/privacy}","received_events_url":"https://api.github.com/users/garlick/received_events","type":"User","site_admin":false},"created_at":"2018-04-15T16:26:02Z","updated_at":"2018-04-15T16:26:02Z","author_association":"OWNER","body":"Oh good point!   Sorry I may have been hasty in my assumptions about the\nfatality.\n\nOn Sun, Apr 15, 2018, 9:19 AM Mark Grondona <notifications@github.com>\nwrote:\n\n> If zeromq sockets can't survive a fork/destroy, I'm actually surprised zmq\n> doesn't install its own pthread_atfork handlers. The job module closes\n> all fds before exec, so I'm struggling to figure out how erroneous\n> destruction of the stale zmq sockets in the forked child could affect the\n> parent broker process.\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/flux-framework/flux-core/issues/1468#issuecomment-381418553>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AAKX25t4Pf3Iiz38kD-JKLM3GNfwIpXPks5to3MHgaJpZM4TVMz2>\n> .\n>\n"}}
{"actor":{"display_login":"garlick"}}