title: The fix client can receive incoming messages but cannot send outgoing heartbeat message
body: <p><strong>We have built a fix client. The fix client can receive incoming messages but cannot send outgoing heartbeat message or reply the TestRequest message</strong> after the last heartbeat was sent, something is triggered to stop sending heartbeat anymore from client side.</p>
  <p>fix version: <strong>fix5.0</strong></p>
  <p>The same incident happened before, we have tcpdump for one session in that time
  <a href="https://i.stack.imgur.com/bvUK9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bvUK9.png" alt="[1]: https://i.stack.imgur.com/oKx49.png" /></a></p>
  <p>we deploy every fix session to separated k8s pods.</p>
  <ol>
  <li>We doubted it's CPU resource issue because the load average is high around the issue time, but it's not solved after we add more cpu cores. we think the load average is high because of fix reconnection.</li>
  <li>We doubted it's IO issue because we use AWS efs which shared by 3 sessions for logging and message store. but it's still not solved after we use pod affinity to assign 3 sessions to different nodes.</li>
  <li>It's not a network issue either, since we can receive fix messages, other sessions worked well at that time. We have disabled <strong>SNAT</strong> in k8s cluster too.</li>
  </ol>
  <p>We are using quickfixj 2.2.0 to create a fix client, we have 3 sessions, which are deployed to k8s pods in separated nodes.</p>
  <ol>
  <li><strong>rate</strong> session to get fx price from server</li>
  <li><strong>order</strong> session to get transaction(execution report) messages from server, we only send logon/heartbeat/logout messages to server.</li>
  <li><strong>backoffice</strong> session to get marketstatus</li>
  </ol>
  <p>We use apache camel quickfixj component to make our programming easy. It works well in most time, but it keeps happening to reconnect to fix servers in 3 sessions, the frequency is like once a month, mostly only 2 sessions have issues.</p>
  <p><code>heartbeatInt = 30s</code></p>
  <p>The fix <strong>event</strong> messages at client side</p>
  <pre><code>20201004-21:10:53.203 Already disconnected: Verifying message failed: quickfix.SessionException: Logon state is not valid for message (MsgType=1)
  20201004-21:10:53.271 MINA session created: local=/172.28.65.164:44974, class org.apache.mina.transport.socket.nio.NioSocketSession, remote=/10.60.45.132:11050
  20201004-21:10:53.537 Initiated logon request
  20201004-21:10:53.643 Setting DefaultApplVerID (1137=9) from Logon
  20201004-21:10:53.643 Logon contains ResetSeqNumFlag=Y, resetting sequence numbers to 1
  20201004-21:10:53.643 Received logon
  </code></pre>
  <p>The fix <strong>incoming</strong> messages at client side</p>
  <pre><code>8=FIXT.1.1☺9=65☺35=0☺34=2513☺49=Quote1☺52=20201004-21:09:02.887☺56=TA_Quote1☺10=186☺
  8=FIXT.1.1☺9=65☺35=0☺34=2514☺49=Quote1☺52=20201004-21:09:33.089☺56=TA_Quote1☺10=185☺
  8=FIXT.1.1☺9=74☺35=1☺34=2515☺49=Quote1☺52=20201004-21:09:48.090☺56=TA_Quote1☺112=TEST☺10=203☺ 
  ----- 21:10:53.203 Already disconnected ----
  8=FIXT.1.1☺9=87☺35=A☺34=1☺49=Quote1☺52=20201004-21:10:53.639☺56=TA_Quote1☺98=0☺108=30☺141=Y☺1137=9☺10=183☺
  8=FIXT.1.1☺9=62☺35=0☺34=2☺49=Quote1☺52=20201004-21:11:23.887☺56=TA_Quote1☺10=026☺
  </code></pre>
  <p>The fix <strong>outgoing</strong> messages at client side</p>
  <pre><code>8=FIXT.1.1☺9=65☺35=0☺34=2513☺49=TA_Quote1☺52=20201004-21:09:02.884☺56=Quote1☺10=183☺
  ---- no heartbeat message around 21:09:32 ----
  ---- 21:10:53.203 Already disconnected ---
  8=FIXT.1.1☺9=134☺35=A☺34=1☺49=TA_Quote1☺52=20201004-21:10:53.433☺56=Quote1☺98=0☺108=30☺141=Y☺553=xxxx☺554=xxxxx☺1137=9☺10=098☺
  8=FIXT.1.1☺9=62☺35=0☺34=2☺49=TA_Quote1☺52=20201004-21:11:23.884☺56=Quote1☺10=023☺
  8=FIXT.1.1☺9=62☺35=0☺34=3☺49=TA_Quote1☺52=20201004-21:11:53.884☺56=Quote1☺10=027☺
  </code></pre>
  <p>Thread dump when TEST message from server was received.BTW, The gist is from our development environment which has the same deployment.
  <a href="https://gist.github.com/hitxiang/345c8f699b4ad1271749e00b7517bef6" rel="nofollow noreferrer">https://gist.github.com/hitxiang/345c8f699b4ad1271749e00b7517bef6</a></p>
  <p>We had enabled the debug log at quickfixj, but not much information, only logs for messages receieved.</p>