title: Strange delays in spark streaming
body: <p>I have recently been using spark streaming to process data in kafka.</p>
  
  <p>After the application is started and a few batches are finished, there is a continuous delay.</p>
  
  <p>Most of the time, data processing is completed within 1-5 seconds.</p>
  
  <p>However, after several batches, it took 41 ~ 45 seconds continuously, and most of the delay occurred in the area that fetches data from stage0.</p>
  
  <p>I accidentally found the Kafka request.timemout.ms setting to be 40 seconds by default and changed this setting to 10 seconds.</p>
  
  <p>I then restarted the application and observed that the batch was completed in 11 to 15 seconds.</p>
  
  <p>Actual processing time is 1-5 sec. I can not understand this delay.</p>
  
  <p>What is wrong?</p>
  
  <p>My environment is as follows.</p>
  
  <p>Spark streaming 2.1.0(createDirectStream)</p>
  
  <p>Kafka : 0.10.1</p>
  
  <p>Batch interval : 20s</p>
  
  <p>Request.timeout.ms : 10s</p>
  
  <p>/////</p>
  
  <p>The following capture is the graph when request.timeout.ms is set to 8 seconds.</p>
  
  <p><a href="https://i.stack.imgur.com/9nP5H.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9nP5H.png" alt="enter image description here"></a></p>