title: Storm topology processing slowing down gradually
body: <p>I have been reading about apache Storm tried few examples from storm-starter. Also learnt about how to tune the topology and how to scale it to perform fast enough to meet the required throughput.</p>
  
  <p>I have created example topology with acking enabled, i am able to achieve 3K-5K messages processing per second. It performs really fast in initial 10 to 15min or around 1mil to 2mil message and then it starts slowing down. On storm UI, I can see the overall latency starts going up gradually and does not comes back, after a while the processing drops to only few hundred a second. I am getting exact same behavior for all the typologies i tried, the simplest one is to just read from kafka using KafkaSpout and send it to transform bolt parse the msg and send it to kafka again using KafkaBolt. The parser is very fast as it takes less than a millisecond to parse the message. I tried few option of increasing/describing the parallelism, changing the buffer sizes etc. but same behavior. Please help me to find out the reason for gradual slowness in the topology. Here is the config i am using</p>
  
  <pre><code>1 Nimbus machine (4 CPU) 24GB RAM
  2 Supervisor machines (8CPU) and using 1 thread per core with 24GB RAM
  4 Node kafka cluster running on above 2 supervisor machines (each topic has 4 partitions)
  
  KafkaSpout(2 parallelism)--&gt;TransformerBolt(8)--&gt;KafkaBolt(2)
  
  topology.executor.receive.buffer.size: 65536
  topology.executor.send.buffer.size: 65536
  topology.spout.max.batch.size: 65536
  topology.transfer.buffer.size: 32
  topology.receiver.buffer.size: 8
  topology.max.spout.pending: 250
  </code></pre>
  
  <p>At the start
  <a href="https://i.stack.imgur.com/SLiKl.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SLiKl.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/QO7eb.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QO7eb.jpg" alt="enter image description here"></a></p>
  
  <p>After few minutes
  <a href="https://i.stack.imgur.com/RLWzM.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RLWzM.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/ZCGRt.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZCGRt.jpg" alt="enter image description here"></a></p>
  
  <p>After 45 min - latency started going up
  <a href="https://i.stack.imgur.com/yAROe.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yAROe.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/1IWYL.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1IWYL.jpg" alt="enter image description here"></a></p>
  
  <p>After 80 min - Latency will keep going up and will go till 100 sec by the time it reaches 8 to 10mil messages
  <a href="https://i.stack.imgur.com/fzPIw.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fzPIw.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/gWypA.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gWypA.jpg" alt="enter image description here"></a></p>
  
  <p>Visual VM screenshot
  <a href="https://i.stack.imgur.com/VjVRK.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VjVRK.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/1ud21.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1ud21.jpg" alt="enter image description here"></a></p>
  
  <p>Threads
  <a href="https://i.stack.imgur.com/sD0K7.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/sD0K7.jpg" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/2HDLZ.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2HDLZ.jpg" alt="enter image description here"></a></p>