title: No response from minion with long-running commands
body: <p>I have one environment where salt-master &lt;-> salt-minion communication has been apparently well established: 4505 and 4506 TCP ports are open, key has been accepted and some functions from test module are working fine:</p>
  
  <pre><code>root@minion01 # telnet master01 4505
  Trying 100.134.0.200...
  Connected to master01.
  Escape character is '^]'.
  
  [user@master01 ~]$ salt-key -L
  Accepted Keys:
  minion01
  
  [user@master01 ~]$ salt 'minion01' test.ping
  minion01:
      True
  
  [user@master01 ~]$ salt 'minion01' test.version
  minion01:
      2015.8.8
  </code></pre>
  
  <p>However, when I'm trying to execute some binaries, I'm getting no response:</p>
  
  <pre><code>[user@master01 ~]$ salt 'minion01' cmd.script 'salt://bin/test01' args='bla'
  minion01:
      Minion did not return. [No response]
  </code></pre>
  
  <p>And I've found and interesting function also from test module to debug this issues: <a href="https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.test.html#salt.modules.test.rand_sleep" rel="nofollow">test.rand_sleep</a></p>
  
  <p>If I run that function with debug flag:</p>
  
  <pre><code>[user@master01 ~]$ salt 'minion01' test.rand_sleep -l debug
  [DEBUG   ] Reading configuration from /salt/etc/master
  [DEBUG   ] Missing configuration file: /home/user/.saltrc
  [DEBUG   ] Configuration file path: /salt/etc/master
  [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
  [DEBUG   ] Reading configuration from /salt/etc/master
  [DEBUG   ] Missing configuration file: /home/user/.saltrc
  [DEBUG   ] MasterEvent PUB socket URI: ipc:///salt/salt/cache/.salt-unix/master_event_pub.ipc
  [DEBUG   ] MasterEvent PULL socket URI: ipc:///salt/salt/cache/.salt-unix/master_event_pull.ipc
  [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/salt/pki/master', 'master01_master', 'tcp://127.0.0.1:4506', 'clear')
  [DEBUG   ] LazyLoaded local_cache.get_load
  [DEBUG   ] Reading minion list from /salt/salt/cache/jobs/59/be0dc5d330a8a183114e4826349b02/.minions.p
  [DEBUG   ] get_iter_returns for jid 20161018122035908234 sent to set(['minion01']) will timeout at 12:20:40.920216
  [DEBUG   ] Checking whether jid 20161018122035908234 is still running
  [DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/salt/pki/master', 'master01_master', 'tcp://127.0.0.1:4506', 'clear')
  [DEBUG   ] LazyLoaded no_return.output
  minion01:
      Minion did not return. [No response]
  </code></pre>
  
  <p>And if I sniff traffic... I actually see packets going in the proper way:</p>
  
  <pre><code>[root@master01 ~]# tcpdump dst 100.134.0.239 -i any and portrange 4505-4506
  12:24:57.051480 IP master01.4505 &gt; minion01.49194: Flags [P.], seq 5698:5893, ack 1, win 115, length 195
  12:25:02.064182 IP master01.4505 &gt; minion01.49194: Flags [P.], seq 5893:6104, ack 1, win 115, length 211
  12:25:04.296926 IP master01.4506 &gt; minion01.49747: Flags [S.], seq 2770281963, ack 425526514, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
  12:25:04.297389 IP master01.4506 &gt; minion01.49747: Flags [P.], seq 1:11, ack 1, win 115, length 10
  12:25:04.297402 IP master01.4506 &gt; minion01.49747: Flags [.], ack 11, win 115, length 0
  12:25:04.297419 IP master01.4506 &gt; minion01.49747: Flags [P.], seq 11:12, ack 11, win 115, length 1
  12:25:04.297680 IP master01.4506 &gt; minion01.49747: Flags [P.], seq 12:13, ack 13, win 115, length 1
  12:25:04.297704 IP master01.4506 &gt; minion01.49747: Flags [P.], seq 13:15, ack 13, win 115, length 2
  12:25:04.302246 IP master01.4506 &gt; minion01.49748: Flags [S.], seq 2644063969, ack 425575524, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
  12:25:04.302570 IP master01.4506 &gt; minion01.49748: Flags [P.], seq 1:11, ack 1, win 115, length 10
  ...
  </code></pre>
  
  <p>And coming back...</p>
  
  <pre><code>[root@master01 ~]# tcpdump src 100.134.0.239 -i any and portrange 4505-4506
  12:27:09.254011 IP minion01.49194 &gt; master01.4505: Flags [.], ack 819763160, win 49640, length 0
  12:27:11.606309 IP minion01.49782 &gt; master01.4506: Flags [S], seq 462752770, win 49640, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
  12:27:11.606671 IP minion01.49782 &gt; master01.4506: Flags [.], ack 1832475241, win 49640, length 0
  12:27:11.606856 IP minion01.49782 &gt; master01.4506: Flags [P.], seq 0:10, ack 1, win 49640, length 10
  12:27:11.607084 IP minion01.49782 &gt; master01.4506: Flags [.], ack 12, win 49640, length 0
  12:27:11.607133 IP minion01.49782 &gt; master01.4506: Flags [P.], seq 10:12, ack 12, win 49640, length 2
  12:27:11.607532 IP minion01.49782 &gt; master01.4506: Flags [.], ack 13, win 49640, length 0
  12:27:11.607608 IP minion01.49782 &gt; master01.4506: Flags [P.], seq 12:14, ack 15, win 49640, length 2
  12:27:11.611069 IP minion01.49783 &gt; master01.4506: Flags [S], seq 462854740, win 49640, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
  </code></pre>
  
  <p>So It looks like communications are somehow suddenly interrupted but, why? How is it possible that some features from test module works and some others does not?</p>
  
  <p>Thanks In advance. Any clue/hint would be appreciated.</p>
  
  <p><strong>Update</strong>: If I execute the salt cmd.script command with timeout flag <code>-t</code> it does work. However, in many other cases I did not need that option. Main difference observed between this case and the successful ones is the absent of following debug messages:</p>
  
  <pre><code>[DEBUG   ] Checking whether jid 20161019054212008948 is still running
  </code></pre>
  
  <p>Even if the minion is configured with custom keepalive settings:</p>
  
  <pre><code>root@minion01 # cat /salt/etc/minion | grep -v ^# | grep -i keepalive
  tcp_keepalive_idle: 60
  tcp_keepalive_cnt: 3
  tcp_keepalive_intvl: 5
  </code></pre>
  
  <p>By the way, there is just one single network element between salt-master (master01) and salt-minion (minion01) which is the internal firewall. Everywhere is properly set the MTU to 1500.</p>