{"action":"created","issue":{"url":"https://api.github.com/repos/zeromq/jeromq/issues/328","repository_url":"https://api.github.com/repos/zeromq/jeromq","labels_url":"https://api.github.com/repos/zeromq/jeromq/issues/328/labels{/name}","comments_url":"https://api.github.com/repos/zeromq/jeromq/issues/328/comments","events_url":"https://api.github.com/repos/zeromq/jeromq/issues/328/events","html_url":"https://github.com/zeromq/jeromq/issues/328","id":134410960,"number":328,"title":"Thread death in iothread-# threads","user":{"login":"cystek","id":4002272,"avatar_url":"https://avatars.githubusercontent.com/u/4002272?v=3","gravatar_id":"","url":"https://api.github.com/users/cystek","html_url":"https://github.com/cystek","followers_url":"https://api.github.com/users/cystek/followers","following_url":"https://api.github.com/users/cystek/following{/other_user}","gists_url":"https://api.github.com/users/cystek/gists{/gist_id}","starred_url":"https://api.github.com/users/cystek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cystek/subscriptions","organizations_url":"https://api.github.com/users/cystek/orgs","repos_url":"https://api.github.com/users/cystek/repos","events_url":"https://api.github.com/users/cystek/events{/privacy}","received_events_url":"https://api.github.com/users/cystek/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-17T21:50:42Z","updated_at":"2016-03-12T13:04:19Z","closed_at":null,"body":"I've been experiencing thread death in the iothread-# threads. It is very intermittent, but every so often a number of them will throw the following exception and die (until there are no more iothreads left to process request in which case ZMQ hangs.\r\n\r\nzmq.ZError$IOException: java.io.IOException: Broken pipe\r\n        at zmq.Signaler.send(Signaler.java:125)\r\n        at zmq.Mailbox.send(Mailbox.java:85)\r\n        at zmq.Ctx.sendCommand(Ctx.java:403)\r\n        at zmq.ZObject.sendCommand(ZObject.java:362)\r\n        at zmq.ZObject.sendPipeTermAck(ZObject.java:238)\r\n        at zmq.Pipe.delimit(Pipe.java:513)\r\n        at zmq.Pipe.read(Pipe.java:209)\r\n        at zmq.SessionBase.pullMsg(SessionBase.java:191)\r\n        at zmq.Dealer$DealerSession.pullMsg(Dealer.java:24)\r\n        at zmq.V1Encoder.messageReady(V1Encoder.java:84)\r\n        at zmq.V1Encoder.next(V1Encoder.java:60)\r\n        at zmq.EncoderBase.getData(EncoderBase.java:78)\r\n        at zmq.StreamEngine.outEvent(StreamEngine.java:369)\r\n        at zmq.StreamEngine.activateOut(StreamEngine.java:444)\r\n        at zmq.SessionBase.readActivated(SessionBase.java:295)\r\n        at zmq.Dealer$DealerSession.readActivated(Dealer.java:24)\r\n        at zmq.Pipe.processActivateRead(Pipe.java:290)\r\n        at zmq.ZObject.processCommand(ZObject.java:57)\r\n        at zmq.IOThread.inEvent(IOThread.java:93)\r\n        at zmq.Poller.run(Poller.java:247)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.IOException: Broken pipe\r\n        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)\r\n        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)\r\n        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)\r\n        at sun.nio.ch.IOUtil.write(IOUtil.java:65)\r\n        at sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:167)\r\n        at zmq.Signaler.send(Signaler.java:121)\r\n        at zmq.Mailbox.send(Mailbox.java:85)\r\n        at zmq.Ctx.sendCommand(Ctx.java:403)\r\n        at zmq.ZObject.sendCommand(ZObject.java:362)\r\n        at zmq.ZObject.sendPipeTermAck(ZObject.java:238)\r\n        at zmq.Pipe.delimit(Pipe.java:513)\r\n        at zmq.Pipe.read(Pipe.java:209)\r\n        at zmq.SessionBase.pullMsg(SessionBase.java:191)\r\n        at zmq.Dealer$DealerSession.pullMsg(Dealer.java:24)\r\n        at zmq.V1Encoder.messageReady(V1Encoder.java:84)\r\n        at zmq.V1Encoder.next(V1Encoder.java:60)\r\n        at zmq.EncoderBase.getData(EncoderBase.java:78)\r\n        at zmq.StreamEngine.outEvent(StreamEngine.java:369)\r\n        at zmq.StreamEngine.activateOut(StreamEngine.java:444)\r\n        at zmq.SessionBase.readActivated(SessionBase.java:295)\r\n        at zmq.Dealer$DealerSession.readActivated(Dealer.java:24)\r\n        at zmq.Pipe.processActivateRead(Pipe.java:290)\r\n        at zmq.ZObject.processCommand(ZObject.java:57)\r\n        at zmq.IOThread.inEvent(IOThread.java:93)\r\n        at zmq.Poller.run(Poller.java:247)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n\r\nSince they handle all communications, I don't think iothreads should be allowed to die (at least not without replacing them with a fresh thread). I tried a custom build of trunk where I catch the exception in Poller.run() and log the stack traces to know when this would have occurred and it does seem that the problem resolves itself afterwards, so that could potentially be a fix unless there is a better place along the stack to handle the exception."},"comment":{"url":"https://api.github.com/repos/zeromq/jeromq/issues/comments/195736838","html_url":"https://github.com/zeromq/jeromq/issues/328#issuecomment-195736838","issue_url":"https://api.github.com/repos/zeromq/jeromq/issues/328","id":195736838,"user":{"login":"miniway","id":725129,"avatar_url":"https://avatars.githubusercontent.com/u/725129?v=3","gravatar_id":"","url":"https://api.github.com/users/miniway","html_url":"https://github.com/miniway","followers_url":"https://api.github.com/users/miniway/followers","following_url":"https://api.github.com/users/miniway/following{/other_user}","gists_url":"https://api.github.com/users/miniway/gists{/gist_id}","starred_url":"https://api.github.com/users/miniway/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miniway/subscriptions","organizations_url":"https://api.github.com/users/miniway/orgs","repos_url":"https://api.github.com/users/miniway/repos","events_url":"https://api.github.com/users/miniway/events{/privacy}","received_events_url":"https://api.github.com/users/miniway/received_events","type":"User","site_admin":false},"created_at":"2016-03-12T13:04:19Z","updated_at":"2016-03-12T13:04:19Z","body":"That's right io/reaper threads shouldn't be dead in the middle of processing. Also we need to find out what causes those thread to be dead to handle the situation better.\r\n\r\nDo you have any reproducible test cases even though it happens rarely?"}}