{"action":"opened","issue":{"url":"https://api.github.com/repos/akka/akka/issues/20296","repository_url":"https://api.github.com/repos/akka/akka","labels_url":"https://api.github.com/repos/akka/akka/issues/20296/labels{/name}","comments_url":"https://api.github.com/repos/akka/akka/issues/20296/comments","events_url":"https://api.github.com/repos/akka/akka/issues/20296/events","html_url":"https://github.com/akka/akka/issues/20296","id":147717126,"number":20296,"title":"quarantined node downed with auto-down-unreachable-after=off","user":{"login":"michaelom","id":4853579,"avatar_url":"https://avatars.githubusercontent.com/u/4853579?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelom","html_url":"https://github.com/michaelom","followers_url":"https://api.github.com/users/michaelom/followers","following_url":"https://api.github.com/users/michaelom/following{/other_user}","gists_url":"https://api.github.com/users/michaelom/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelom/subscriptions","organizations_url":"https://api.github.com/users/michaelom/orgs","repos_url":"https://api.github.com/users/michaelom/repos","events_url":"https://api.github.com/users/michaelom/events{/privacy}","received_events_url":"https://api.github.com/users/michaelom/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-12T11:25:20Z","updated_at":"2016-04-12T11:25:20Z","closed_at":null,"body":"akka 2.4.1\r\n\r\nDuring what seems to be a network partition a cluster member downed a quarantined member:\r\n\r\n```\r\n[akka.tcp://SomeActorSystem@172.30.14.139:51000] with UID [-1507123525] irrecoverably failed. Quarantining address.\r\n\r\njava.lang.IllegalStateException: Error encountered while processing system message acknowledgement buffer: [-1 {}] ack: ACK[0, {}]\r\n\tat akka.remote.ReliableDeliverySupervisor$$anonfun$receive$1.applyOrElse(Endpoint.scala:288)\r\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:480)\r\n\tat akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:189)\r\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)\r\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:495)\r\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)\r\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:224)\r\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:234)\r\n\tat scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\r\n\tat scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\r\n\tat scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\r\nCaused by: java.lang.IllegalArgumentException: Highest SEQ so far was -1 but cumulative ACK is 0\r\n\r\nAssociation to [akka.tcp://SomeActorSystem@172.30.14.139:51000] having UID [-1507123525] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.\r\n\r\nCluster Node [akka.tcp://SomeActorSystem@172.30.14.124:51000] - Marking node as TERMINATED [akka.tcp://AccountActorSystem@172.30.14.139:51000], due to quarantine\r\n\r\nCluster Node [akka.tcp://SomeActorSystem@172.30.14.124:51000] - Marking unreachable node [akka.tcp://AccountActorSystem@172.30.14.139:51000] as [Down]\r\n\r\nCluster Node [akka.tcp://SomeActorSystem@172.30.14.124:51000] - Leader can perform its duties again\r\n\r\nCluster Node [akka.tcp://SomeActorSystem@172.30.14.124:51000] - Leader is removing unreachable node [akka.tcp://SomeActorSystem@172.30.14.139:51000]\r\n\r\nMember removed [akka.tcp://SomeActorSystem@172.30.14.139:51000]\r\n```\r\n\r\nQuarantined nodes seemed to be downed even with auto downing disabled. That's not what I expected.\r\n\r\n\r\n```  \r\ndef quarantined(node: UniqueAddress): Unit = {\r\n    val localGossip = latestGossip\r\n    if (localGossip.hasMember(node)) {\r\n      val newReachability = latestGossip.overview.reachability.terminated(selfUniqueAddress, node)\r\n      val newOverview = localGossip.overview copy (reachability = newReachability)\r\n      val newGossip = localGossip copy (overview = newOverview)\r\n      updateLatestGossip(newGossip)\r\n      log.warning(\"Cluster Node [{}] - Marking node as TERMINATED [{}], due to quarantine\",\r\n        selfAddress, node.address)\r\n      publish(latestGossip)\r\n      downing(node.address)\r\n    }\r\n  }\r\n```"}}