title: Mongodb "NetworkInterfaceExceededTimeLimit" replicaset
body: <h3>Solved:</h3>
  <p>It most have been a version specific bug. Updating to mongodb-org@4.0.5 solved the issue and the machines connect fine. I'll leave the question for eventualy someone who faces the same issue.</p>
  <h3>Original question:</h3>
  <p>I have two servers in a two member replica set.</p>
  <p>mx.aireclaim.com is the PRIMARY and vimax.aireclaim.com is the SECONDARY. After a reboot on mx.aireclaim.com I'm faced with a connection issue on the secondary side. I've checked the firewall and can access the 27017 port from each to machine to the other.</p>
  <p>I keep getting this error in my secondary:</p>
  <pre><code>Error in heartbeat (...) response status: NetworkInterfaceExceededTimeLimit:
  Couldn't get a connection within the time limit
  </code></pre>
  <p>Is there something foul I'm missing? I've tried syncing the date on both machines, disabling firewalls, multiple restarts, and config changes, to no avail.</p>
  <p>Here are the replicaset config and rs.status():</p>
  <h1>PRIMARY</h1>
  <pre><code>    aireclaimRs:PRIMARY&gt; rs.config()
      {
          &quot;_id&quot; : &quot;aireclaimRs&quot;,
          &quot;version&quot; : 3,
          &quot;protocolVersion&quot; : NumberLong(1),
          &quot;members&quot; : [
              {
                  &quot;_id&quot; : 0,
                  &quot;host&quot; : &quot;vimax.aireclaim.com:27017&quot;,
                  &quot;arbiterOnly&quot; : false,
                  &quot;buildIndexes&quot; : true,
                  &quot;hidden&quot; : false,
                  &quot;priority&quot; : 2,
                  &quot;tags&quot; : {
                      
                  },
                  &quot;slaveDelay&quot; : NumberLong(0),
                  &quot;votes&quot; : 1
              },
              {
                  &quot;_id&quot; : 1,
                  &quot;host&quot; : &quot;mx.aireclaim.com:27017&quot;,
                  &quot;arbiterOnly&quot; : false,
                  &quot;buildIndexes&quot; : true,
                  &quot;hidden&quot; : false,
                  &quot;priority&quot; : 1,
                  &quot;tags&quot; : {
                      
                  },
                  &quot;slaveDelay&quot; : NumberLong(0),
                  &quot;votes&quot; : 1
              }
          ],
          &quot;settings&quot; : {
              &quot;chainingAllowed&quot; : true,
              &quot;heartbeatIntervalMillis&quot; : 2000,
              &quot;heartbeatTimeoutSecs&quot; : 10,
              &quot;electionTimeoutMillis&quot; : 10000,
              &quot;catchUpTimeoutMillis&quot; : -1,
              &quot;catchUpTakeoverDelayMillis&quot; : 30000,
              &quot;getLastErrorModes&quot; : {
                  
              },
              &quot;getLastErrorDefaults&quot; : {
                  &quot;w&quot; : 1,
                  &quot;wtimeout&quot; : 0
              },
              &quot;replicaSetId&quot; : ObjectId(&quot;5be0f8105dfcbe069f5c8533&quot;)
          }
      }
      aireclaimRs:PRIMARY&gt; rs.status()
      {
          &quot;set&quot; : &quot;aireclaimRs&quot;,
          &quot;date&quot; : ISODate(&quot;2020-03-23T20:08:25.719Z&quot;),
          &quot;myState&quot; : 1,
          &quot;term&quot; : NumberLong(20),
          &quot;heartbeatIntervalMillis&quot; : NumberLong(2000),
          &quot;optimes&quot; : {
              &quot;lastCommittedOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(0, 0),
                  &quot;t&quot; : NumberLong(-1)
              },
              &quot;appliedOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(1584994100, 1),
                  &quot;t&quot; : NumberLong(20)
              },
              &quot;durableOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(1584994100, 1),
                  &quot;t&quot; : NumberLong(20)
              }
          },
          &quot;members&quot; : [
              {
                  &quot;_id&quot; : 0,
                  &quot;name&quot; : &quot;vimax.aireclaim.com:27017&quot;,
                  &quot;health&quot; : 1,
                  &quot;state&quot; : 2,
                  &quot;stateStr&quot; : &quot;SECONDARY&quot;,
                  &quot;uptime&quot; : 1595,
                  &quot;optime&quot; : {
                      &quot;ts&quot; : Timestamp(1584838063, 1),
                      &quot;t&quot; : NumberLong(11)
                  },
                  &quot;optimeDurable&quot; : {
                      &quot;ts&quot; : Timestamp(1584838063, 1),
                      &quot;t&quot; : NumberLong(11)
                  },
                  &quot;optimeDate&quot; : ISODate(&quot;2020-03-22T00:47:43Z&quot;),
                  &quot;optimeDurableDate&quot; : ISODate(&quot;2020-03-22T00:47:43Z&quot;),
                  &quot;lastHeartbeat&quot; : ISODate(&quot;2020-03-23T20:08:24.148Z&quot;),
                  &quot;lastHeartbeatRecv&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),
                  &quot;pingMs&quot; : NumberLong(0),
                  &quot;configVersion&quot; : 3
              },
              {
                  &quot;_id&quot; : 1,
                  &quot;name&quot; : &quot;mx.aireclaim.com:27017&quot;,
                  &quot;health&quot; : 1,
                  &quot;state&quot; : 1,
                  &quot;stateStr&quot; : &quot;PRIMARY&quot;,
                  &quot;uptime&quot; : 8376,
                  &quot;optime&quot; : {
                      &quot;ts&quot; : Timestamp(1584994100, 1),
                      &quot;t&quot; : NumberLong(20)
                  },
                  &quot;optimeDate&quot; : ISODate(&quot;2020-03-23T20:08:20Z&quot;),
                  &quot;electionTime&quot; : Timestamp(1584992519, 1),
                  &quot;electionDate&quot; : ISODate(&quot;2020-03-23T19:41:59Z&quot;),
                  &quot;configVersion&quot; : 3,
                  &quot;self&quot; : true
              }
          ],
          &quot;ok&quot; : 1,
          &quot;operationTime&quot; : Timestamp(1584994100, 1),
          &quot;$clusterTime&quot; : {
              &quot;clusterTime&quot; : Timestamp(1584994100, 1),
              &quot;signature&quot; : {
                  &quot;hash&quot; : BinData(0,&quot;oGCz38lgYjsGTrWc3maAD2vyc6M=&quot;),
                  &quot;keyId&quot; : NumberLong(&quot;6765287379189104641&quot;)
              }
          }
      }
  </code></pre>
  <h1>SECONDARY</h1>
  <pre><code>    aireclaimRs:SECONDARY&gt; rs.status()
      {
          &quot;set&quot; : &quot;aireclaimRs&quot;,
          &quot;date&quot; : ISODate(&quot;2020-03-23T19:56:37.132Z&quot;),
          &quot;myState&quot; : 2,
          &quot;term&quot; : NumberLong(20),
          &quot;syncingTo&quot; : &quot;&quot;,
          &quot;syncSourceHost&quot; : &quot;&quot;,
          &quot;syncSourceId&quot; : -1,
          &quot;heartbeatIntervalMillis&quot; : NumberLong(2000),
          &quot;optimes&quot; : {
              &quot;lastCommittedOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(0, 0),
                  &quot;t&quot; : NumberLong(-1)
              },
              &quot;appliedOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(1584838063, 1),
                  &quot;t&quot; : NumberLong(11)
              },
              &quot;durableOpTime&quot; : {
                  &quot;ts&quot; : Timestamp(1584838063, 1),
                  &quot;t&quot; : NumberLong(11)
              }
          },
          &quot;members&quot; : [
              {
                  &quot;_id&quot; : 0,
                  &quot;name&quot; : &quot;vimax.aireclaim.com:27017&quot;,
                  &quot;health&quot; : 1,
                  &quot;state&quot; : 2,
                  &quot;stateStr&quot; : &quot;SECONDARY&quot;,
                  &quot;uptime&quot; : 890,
                  &quot;optime&quot; : {
                      &quot;ts&quot; : Timestamp(1584838063, 1),
                      &quot;t&quot; : NumberLong(11)
                  },
                  &quot;optimeDate&quot; : ISODate(&quot;2020-03-22T00:47:43Z&quot;),
                  &quot;syncingTo&quot; : &quot;&quot;,
                  &quot;syncSourceHost&quot; : &quot;&quot;,
                  &quot;syncSourceId&quot; : -1,
                  &quot;infoMessage&quot; : &quot;&quot;,
                  &quot;configVersion&quot; : 3,
                  &quot;self&quot; : true,
                  &quot;lastHeartbeatMessage&quot; : &quot;&quot;
              },
              {
                  &quot;_id&quot; : 1,
                  &quot;name&quot; : &quot;mx.aireclaim.com:27017&quot;,
                  &quot;health&quot; : 0,
                  &quot;state&quot; : 8,
                  &quot;stateStr&quot; : &quot;(not reachable/healthy)&quot;,
                  &quot;uptime&quot; : 0,
                  &quot;optime&quot; : {
                      &quot;ts&quot; : Timestamp(0, 0),
                      &quot;t&quot; : NumberLong(-1)
                  },
                  &quot;optimeDurable&quot; : {
                      &quot;ts&quot; : Timestamp(0, 0),
                      &quot;t&quot; : NumberLong(-1)
                  },
                  &quot;optimeDate&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),
                  &quot;optimeDurableDate&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),
                  &quot;lastHeartbeat&quot; : ISODate(&quot;2020-03-23T19:56:31.106Z&quot;),
                  &quot;lastHeartbeatRecv&quot; : ISODate(&quot;2020-03-23T19:56:36.034Z&quot;),
                  &quot;pingMs&quot; : NumberLong(0),
                  &quot;lastHeartbeatMessage&quot; : &quot;Couldn't get a connection within the time limit&quot;,
                  &quot;syncingTo&quot; : &quot;&quot;,
                  &quot;syncSourceHost&quot; : &quot;&quot;,
                  &quot;syncSourceId&quot; : -1,
                  &quot;infoMessage&quot; : &quot;&quot;,
                  &quot;configVersion&quot; : -1
              }
          ],
          &quot;ok&quot; : 1,
          &quot;operationTime&quot; : Timestamp(1584838063, 1),
          &quot;$clusterTime&quot; : {
              &quot;clusterTime&quot; : Timestamp(1584993390, 1),
              &quot;signature&quot; : {
                  &quot;hash&quot; : BinData(0,&quot;zINnjtBKXZ14gJrxrUbT2zyurqQ=&quot;),
                  &quot;keyId&quot; : NumberLong(&quot;6765287379189104641&quot;)
              }
          }
      }
  </code></pre>
  <p>This is what is happening in the logs:</p>
  <h3>Primary</h3>
  <pre><code>tail /var/log/mongodb/mongodb.log
  2020-03-23T21:08:09.420+0100 I NETWORK  [conn2630] end connection 144.76.84.5:50114 (2 connections now open)
  2020-03-23T21:08:29.420+0100 I NETWORK  [listener] connection accepted from 144.76.84.5:50118 #2631 (3 connections now open)
  2020-03-23T21:08:29.420+0100 I NETWORK  [conn2631] end connection 144.76.84.5:50118 (2 connections now open)
  2020-03-23T21:08:49.420+0100 I NETWORK  [listener] connection accepted from 144.76.84.5:50122 #2632 (3 connections now open)
  2020-03-23T21:08:49.420+0100 I NETWORK  [conn2632] end connection 144.76.84.5:50122 (2 connections now open)
  2020-03-23T21:09:09.421+0100 I NETWORK  [listener] connection accepted from 144.76.84.5:50126 #2633 (3 connections now open)
  2020-03-23T21:09:09.421+0100 I NETWORK  [conn2633] end connection 144.76.84.5:50126 (2 connections now open)
  2020-03-23T21:09:18.717+0100 I NETWORK  [conn2629] end connection 127.0.0.1:60316 (1 connection now open)
  2020-03-23T21:09:29.421+0100 I NETWORK  [listener] connection accepted from 144.76.84.5:50130 #2634 (2 connections now open)
  2020-03-23T21:09:29.421+0100 I NETWORK  [conn2634] end connection 144.76.84.5:50130 (1 connection now open)
  </code></pre>
  <h3>Secondary</h3>
  <pre><code>tail /var/log/mongodb/mongod.log 
  2020-03-23T20:48:13.408+0100 I REPL     [replexec-1] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
  2020-03-23T20:48:17.592+0100 I REPL_HB  [replexec-1] Error in heartbeat (requestId: 74) to mx.aireclaim.com:27017, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit
  2020-03-23T20:48:19.658+0100 I REPL     [rsBackgroundSync] waiting for 2 pings from other members before syncing
  2020-03-23T20:48:24.791+0100 I REPL     [replexec-1] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
  2020-03-23T20:48:28.092+0100 I REPL_HB  [replexec-1] Error in heartbeat (requestId: 76) to mx.aireclaim.com:27017, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit
  2020-03-23T20:48:29.593+0100 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mx.aireclaim.com:27017 - NetworkInterfaceExceededTimeLimit: Operation timed out
  2020-03-23T20:48:29.593+0100 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mx.aireclaim.com:27017
  2020-03-23T20:48:34.661+0100 I REPL     [rsBackgroundSync] waiting for 2 pings from other members before syncing
  2020-03-23T20:48:35.839+0100 I REPL     [replexec-1] Not starting an election, since we are not electable due to: Not standing for election because I cannot see a majority (mask 0x1)
  2020-03-23T20:48:38.593+0100 I REPL_HB  [replexec-1] Error in heartbeat (requestId: 77) to mx.aireclaim.com:27017, response status: NetworkInterfaceExceededTimeLimit: Couldn't get a connection within the time limit
  </code></pre>
  <p>So it looks like the primary accepts the connection then ends it.</p>
  <p>You would think that there is some conectivity issue between the two machines, but I can :</p>
  <pre><code>aireclaim@aireclaim-platform-server:~&gt; mongo mx.aireclaim.com
  MongoDB shell version v3.6.10
  connecting to: mongodb://mx.aireclaim.com:27017/test?gssapiServiceName=mongodb
  Implicit session: session { &quot;id&quot; : UUID(&quot;de193403-69ca-4f74-a6c5-ee384eb420d7&quot;) }
  MongoDB server version: 3.6.3
  aireclaimRs:PRIMARY&gt; 
  </code></pre>
  <p>and:</p>
  <pre><code>root@mx  /home/aireclaim  mongo vimax.aireclaim.com
  MongoDB shell version v3.6.3
  connecting to: mongodb://vimax.aireclaim.com:27017/test
  MongoDB server version: 3.6.10
  aireclaimRs:SECONDARY&gt; 
  </code></pre>
  <p>Also I can atest that both machines have port 27017 open and that they are configured to run on those Ips.</p>
  <p>In fact the entire thing remains funcional somehow. The clients that atempt to connect to the replicaset manage to do it, but It looks as though the secondary is no longer replicating.</p>
  <p>Any insight would be a godsend.</p>