{"action":"created","issue":{"url":"https://api.github.com/repos/alibaba/jstorm/issues/377","repository_url":"https://api.github.com/repos/alibaba/jstorm","labels_url":"https://api.github.com/repos/alibaba/jstorm/issues/377/labels{/name}","comments_url":"https://api.github.com/repos/alibaba/jstorm/issues/377/comments","events_url":"https://api.github.com/repos/alibaba/jstorm/issues/377/events","html_url":"https://github.com/alibaba/jstorm/issues/377","id":187247445,"number":377,"title":"怀疑遇到一个死锁的bug.","user":{"login":"yunfan123","id":5563799,"avatar_url":"https://avatars.githubusercontent.com/u/5563799?v=3","gravatar_id":"","url":"https://api.github.com/users/yunfan123","html_url":"https://github.com/yunfan123","followers_url":"https://api.github.com/users/yunfan123/followers","following_url":"https://api.github.com/users/yunfan123/following{/other_user}","gists_url":"https://api.github.com/users/yunfan123/gists{/gist_id}","starred_url":"https://api.github.com/users/yunfan123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yunfan123/subscriptions","organizations_url":"https://api.github.com/users/yunfan123/orgs","repos_url":"https://api.github.com/users/yunfan123/repos","events_url":"https://api.github.com/users/yunfan123/events{/privacy}","received_events_url":"https://api.github.com/users/yunfan123/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-11-04T03:43:06Z","updated_at":"2016-11-04T10:02:38Z","closed_at":null,"body":"我的job大概是这样，两个spout 从kafka里面读数据，发给下层的bolt。\r\n我用的是ShellBolt和ShellSpout，运行一段时间某些worker hang死了。\r\n看jstack，SpoutExecutor都阻塞在DisruptorQueueImpl.publishDirect里面 MultiProducerSequencer.next(MultiProducerSequencer.java:128)  里面的while true 循环，其他线程（boltExecutor、TransferRunnable、Boltdeserializer等）都卡在consumeBatchWhenAvailable\r\n(DisruptorQueueImpl.java:157) 。感觉像是spout想将消息不经过网络IO 直接通过 DisruptorQueueImpl 发给bolt，但是发不出去。然后bolt想要从DisruptorQueueImpl中消费消息又消费不了。\r\n\r\n\r\n详细的调用栈：\r\n用Jstack看：\r\nbolt的jstack都如下：\r\nBoltExecutor：\r\n> \"calc_mapping_stat:37-BoltExecutors\" #65 prio=10 os_prio=0 tid=0x00007faa50003800 nid=0x2a6e5 runnable [0x00007faac9136000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000700442fe0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:37) at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:157) at com.alibaba.jstorm.task.execute.BoltExecutors.run(BoltExecutors.java:137) at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95) at java.lang.Thread.run(Thread.java:745)\r\n\r\nBolt TransferRunnable:\r\n\r\n> \"calc_mapping_stat:197-TransferRunnable\" #70 prio=5 os_prio=0 tid=0x00007faa3c002000 nid=0x2a72d runnable [0x00007faac8d32000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007003ff3c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:37) at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:157) at com.alibaba.jstorm.task.TaskTransfer$TransferRunnable.run(TaskTransfer.java:201) at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95) at java.lang.Thread.run(Thread.java:745)\r\n\r\nSpout的 executor jstack都如下：\r\n\r\n> \"jstorm_helloworld_log_p26:320-SingleThreadSpoutExecutors\" #56 prio=10 os_prio=0 tid=0x00007faa6800c800 nid=0x2a680 sleeping[0x00007fab2071e000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at backtype.storm.utils.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:128) at backtype.storm.utils.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:100) at backtype.storm.utils.disruptor.RingBuffer.next(RingBuffer.java:206) at backtype.storm.utils.DisruptorQueueImpl.publishDirect(DisruptorQueueImpl.java:245) at backtype.storm.utils.DisruptorQueueImpl.publish(DisruptorQueueImpl.java:238) at backtype.storm.utils.DisruptorQueueImpl.publish(DisruptorQueueImpl.java:211) at com.alibaba.jstorm.task.TaskBatchTransfer.transfer(TaskBatchTransfer.java:107) at com.alibaba.jstorm.task.execute.spout.SpoutBatchCollector.sendBatch(SpoutBatchCollector.java:123) at com.alibaba.jstorm.task.execute.spout.SpoutBatchCollector.sendBatch(SpoutBatchCollector.java:135) at com.alibaba.jstorm.task.execute.spout.SpoutBatchCollector.sendSpoutMsg(SpoutBatchCollector.java:86) at com.alibaba.jstorm.task.execute.spout.SpoutCollector.emit(SpoutCollector.java:112) at backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:81) at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:167) at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:93) at com.alibaba.jstorm.task.execute.spout.SpoutExecutors.nextTuple(SpoutExecutors.java:175) at com.alibaba.jstorm.task.execute.spout.SingleThreadSpoutExecutors.run(SingleThreadSpoutExecutors.java:76) at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95) at java.lang.Thread.run(Thread.java:745) \r\n\r\n\r\n> \"client-schedule-service-3\" #48 prio=5 os_prio=0 tid=0x00007faa94010000 nid=0x2a631 waiting on condition [0x00007fab21027000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007007e74a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)"},"comment":{"url":"https://api.github.com/repos/alibaba/jstorm/issues/comments/258388334","html_url":"https://github.com/alibaba/jstorm/issues/377#issuecomment-258388334","issue_url":"https://api.github.com/repos/alibaba/jstorm/issues/377","id":258388334,"user":{"login":"yunfan123","id":5563799,"avatar_url":"https://avatars.githubusercontent.com/u/5563799?v=3","gravatar_id":"","url":"https://api.github.com/users/yunfan123","html_url":"https://github.com/yunfan123","followers_url":"https://api.github.com/users/yunfan123/followers","following_url":"https://api.github.com/users/yunfan123/following{/other_user}","gists_url":"https://api.github.com/users/yunfan123/gists{/gist_id}","starred_url":"https://api.github.com/users/yunfan123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yunfan123/subscriptions","organizations_url":"https://api.github.com/users/yunfan123/orgs","repos_url":"https://api.github.com/users/yunfan123/repos","events_url":"https://api.github.com/users/yunfan123/events{/privacy}","received_events_url":"https://api.github.com/users/yunfan123/received_events","type":"User","site_admin":false},"created_at":"2016-11-04T10:02:38Z","updated_at":"2016-11-04T10:02:38Z","body":"不知道我的理解对不对，还有就是我用的配置是spout.single.thread 为 True，不过这个从栈中也能看到。\r\n这个时候jstorm spout 的ack和executor在同样一个线程。\r\n\r\n假设在数据量很大的情况下，我的bolt处理不过来，这时候假设所有的cache都被打满。\r\n因为cache满，spout.nexttuple 发送阻塞，而ack和它在一个线程，这时候造成无法处理ack数据。\r\n因为cache满，这时候bolt等待ack发送，但是由于ack发送给spout，spout又hang在emit数据。\r\n这样造成的死锁？"}}
{"actor":{"display_login":"yunfan123"}}