title: AWS MongoDB [ReplicationExecutor] Error in heartbeat request to xxx ExceededTimeLimit: Operation timed out
body: <p>I have replica set (hosted on amazon) which has:</p>
  
  <ul>
  <li>primary </li>
  <li>secondary </li>
  <li>arbiter</li>
  </ul>
  
  <p>All of them are version 3.2.9 and this replica is making one shard in my sharded cluster (if that is important although I think it is not).</p>
  
  <p>When I type <code>rs.status()</code> on primary it says that cannot reach secondary, secondary cannot reach arbiter and arbiter cannot reach primary:</p>
  
  <p>On Primary:
  <code>{
      "set" : "rs0",
      "date" : ISODate("2017-06-16T18:43:23.992Z"),
      "myState" : 1,
      "term" : NumberLong(-1),
      "heartbeatIntervalMillis" : NumberLong(2000),
      "members" : [
          {
              "_id" : 3,
              "name" : "secondaryIP:27017",
              "health" : 0,
              "state" : 8,
              "stateStr" : "(not reachable/healthy)",
              "uptime" : 0,
              "optime" : Timestamp(0, 0),
              "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
              "lastHeartbeat" : ISODate("2017-06-16T18:43:22.767Z"),
              "lastHeartbeatRecv" : ISODate("2017-06-16T18:43:22.931Z"),
              "pingMs" : NumberLong(0),
              "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
              "configVersion" : -1
          },
  </code></p>
  
  <p>On Secondary: 
  <code>{"set" : "rs0",
      "date" : ISODate("2017-06-16T19:08:23.364Z"),
      "myState" : 2,
      "term" : NumberLong(-1),
      "heartbeatIntervalMillis" : NumberLong(2000),
      "members" : [
          {
              "_id" : 2,
              "name" : "ip-x-x-x-x:27017",
              "health" : 0,
              "state" : 8,
              "stateStr" : "(not reachable/healthy)",
              "uptime" : 0,
              "lastHeartbeat" : ISODate("2017-06-16T19:08:21.404Z"),
              "lastHeartbeatRecv" : ISODate("2017-06-16T19:08:21.424Z"),
              "pingMs" : NumberLong(0),
              "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
              "configVersion" : -1
          }</code></p>
  
  <p>On Arbiter: 
  <code>{"set" : "rs0",
      "date" : ISODate("2017-06-16T17:54:22.552Z"),
      "myState" : 7,
      "term" : NumberLong(-1),
      "heartbeatIntervalMillis" : NumberLong(2000),
      "members" : [
          {
              "_id" : 4,
              "name" : "primaryIP:27017",
              "health" : 0,
              "state" : 8,
              "stateStr" : "(not reachable/healthy)",
              "uptime" : 0,
              "optime" : Timestamp(0, 0),
              "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
              "lastHeartbeat" : ISODate("2017-06-16T17:54:13.027Z"),
              "lastHeartbeatRecv" : ISODate("2017-06-16T17:54:22.107Z"),
              "pingMs" : NumberLong(0),
              "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
              "configVersion" : -1
          }</code></p>
  
  <p>From the error log on primary:
  <code>2017-06-16T18:40:25.030+0000 I NETWORK  [initandlisten] connection accepted from appIP:60768 #256386 (23 connections now open)
  2017-06-16T18:40:28.762+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to secondaryIP:27017 - ExceededTimeLimit: Operation timed out
  2017-06-16T18:40:28.762+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to secondaryIP:27017; ExceededTimeLimit: Operation timed out
  2017-06-16T18:40:28.762+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to secondaryIP:27017
  2017-06-16T18:40:34.761+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to secondaryIP:27017; ExceededTimeLimit: Couldn't get a connection within the time limit
  2017-06-16T18:40:46.761+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to secondaryIP:27017; ExceededTimeLimit: Couldn't get a connection within the time limit
  2017-06-16T18:40:48.762+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to secondaryIP:27017 - ExceededTimeLimit: Operation timed out
  2017-06-16T18:40:48.762+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to secondaryIP:27017; ExceededTimeLimit: Operation timed out
  2017-06-16T18:40:48.762+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to secondaryIP:27017
  </code></p>
  
  <p>I am unable to connect to any of the IPs with <code>mongo --host &lt;IP&gt;</code> and I get timed out.</p>
  
  <p>All the replica sets are in the same EC2 security group.</p>
  
  <p>This is very similar to <a href="https://stackoverflow.com/questions/38485285/mongodb-replica-heartbeat-request-time-exceeded">this issue</a> but I couldn't understand the answer given on that issue.</p>
  
  <p>I can provide more info if needed.</p>
  
  <p>The database is still reachable and responsive in the app but there are some inconsistencies with numbers and logs which are likely to be due to this connection errors.</p>
  
  <p>Any help would be appreciated.</p>