{"action":"created","issue":{"url":"https://api.github.com/repos/akka/akka/issues/20942","repository_url":"https://api.github.com/repos/akka/akka","labels_url":"https://api.github.com/repos/akka/akka/issues/20942/labels{/name}","comments_url":"https://api.github.com/repos/akka/akka/issues/20942/comments","events_url":"https://api.github.com/repos/akka/akka/issues/20942/events","html_url":"https://github.com/akka/akka/issues/20942","id":165059755,"number":20942,"title":"After new incarnation of cluster node with singleton is started, handover is not properly processed","user":{"login":"412b","id":12436202,"avatar_url":"https://avatars.githubusercontent.com/u/12436202?v=3","gravatar_id":"","url":"https://api.github.com/users/412b","html_url":"https://github.com/412b","followers_url":"https://api.github.com/users/412b/followers","following_url":"https://api.github.com/users/412b/following{/other_user}","gists_url":"https://api.github.com/users/412b/gists{/gist_id}","starred_url":"https://api.github.com/users/412b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/412b/subscriptions","organizations_url":"https://api.github.com/users/412b/orgs","repos_url":"https://api.github.com/users/412b/repos","events_url":"https://api.github.com/users/412b/events{/privacy}","received_events_url":"https://api.github.com/users/412b/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akka/akka/labels/1%20-%20triaged","name":"1 - triaged","color":"5319e7"},{"url":"https://api.github.com/repos/akka/akka/labels/t:cluster-tools","name":"t:cluster-tools","color":"CCCCCC"}],"state":"open","locked":false,"assignee":{"login":"johanandren","id":666915,"avatar_url":"https://avatars.githubusercontent.com/u/666915?v=3","gravatar_id":"","url":"https://api.github.com/users/johanandren","html_url":"https://github.com/johanandren","followers_url":"https://api.github.com/users/johanandren/followers","following_url":"https://api.github.com/users/johanandren/following{/other_user}","gists_url":"https://api.github.com/users/johanandren/gists{/gist_id}","starred_url":"https://api.github.com/users/johanandren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johanandren/subscriptions","organizations_url":"https://api.github.com/users/johanandren/orgs","repos_url":"https://api.github.com/users/johanandren/repos","events_url":"https://api.github.com/users/johanandren/events{/privacy}","received_events_url":"https://api.github.com/users/johanandren/received_events","type":"User","site_admin":false},"assignees":[{"login":"johanandren","id":666915,"avatar_url":"https://avatars.githubusercontent.com/u/666915?v=3","gravatar_id":"","url":"https://api.github.com/users/johanandren","html_url":"https://github.com/johanandren","followers_url":"https://api.github.com/users/johanandren/followers","following_url":"https://api.github.com/users/johanandren/following{/other_user}","gists_url":"https://api.github.com/users/johanandren/gists{/gist_id}","starred_url":"https://api.github.com/users/johanandren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johanandren/subscriptions","organizations_url":"https://api.github.com/users/johanandren/orgs","repos_url":"https://api.github.com/users/johanandren/repos","events_url":"https://api.github.com/users/johanandren/events{/privacy}","received_events_url":"https://api.github.com/users/johanandren/received_events","type":"User","site_admin":false}],"milestone":null,"comments":15,"created_at":"2016-07-12T11:45:52Z","updated_at":"2016-07-18T09:19:42Z","closed_at":null,"body":"Setup: three cluster nodes with singleton.\r\nSteps to reproduce: restart oldest node (where singleton is currently running).\r\nWhat happens: \r\non new oldest\r\n> 2016-07-12T10:13:35.497Z [cluster-test-akka.actor.default-dispatcher-39] [INFO ] [akka.cluster.Cluster(akka://cluster-test)] a.c.Cluster(akka://cluster-test) - Cluster Node [akka.tcp://cluster-test@node01:2552] - New incarnation of existing member [Member(address = akka.tcp://cluster-test@node03:2552, status = Down)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.\r\n2016-07-12T10:14:06.426Z [cluster-test-akka.actor.default-dispatcher-51] [INFO ] [akka.tcp://cluster-test@node01:2552/user/master] a.c.s.ClusterSingletonManager - Younger observed OldestChanged: [Some(akka.tcp://cluster-test@node03:2552) -> myself]\r\n2016-07-12T10:14:07.439Z [cluster-test-akka.actor.default-dispatcher-47] [INFO ] [akka.tcp://cluster-test@node01:2552/user/master] a.c.s.ClusterSingletonManager - Retry [1], sending HandOverToMe to [Some(akka.tcp://cluster-test@node03:2552)]\r\n...\r\n2016-07-12T10:14:35.660Z [cluster-test-akka.actor.default-dispatcher-51] [INFO ] [akka.tcp://cluster-test@node01:2552/user/master] a.c.s.ClusterSingletonManager - Retry [29], sending HandOverToMe to [Some(akka.tcp://cluster-test@node03:2552)]\r\n2016-07-12T10:14:36.429Z [cluster-test-akka.actor.default-dispatcher-51] [INFO ] [akka.tcp://cluster-test@node01:2552/user/master] a.c.s.ClusterSingletonManager - Previous oldest [akka.tcp://cluster-test@node03:2552] removed\r\n\r\non new incarnation of previously oldest\r\n> 2016-07-12T10:13:23.936Z [pool-1-thread-1] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Starting up...\r\n2016-07-12T10:13:23.962Z [pool-1-thread-1] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Registered cluster JMX MBean [akka:type=Cluster]\r\n2016-07-12T10:13:23.962Z [pool-1-thread-1] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Started up successfully\r\n2016-07-12T10:13:23.982Z [cluster-test-cluster-dispatcher-28] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriate platform-specific native libary to 'java.library.path'. Reason: java.lang.ClassNotFoundException: org.hyperic.sigar.Sigar\r\n2016-07-12T10:13:23.990Z [cluster-test-cluster-dispatcher-28] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Metrics collection has started successfully\r\n...\r\n2016-07-12T10:13:25.138Z [cluster-test-akka.actor.default-dispatcher-4] [INFO] [akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-1] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://cluster-test@node01:2552/system/cluster/core/daemon#-987111555] to Actor[akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-1#1591016585] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:13:25.138Z [cluster-test-akka.actor.default-dispatcher-4] [INFO] [akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-1] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://cluster-test@node02:2552/system/cluster/core/daemon#-1169302356] to Actor[akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-1#1591016585] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:14:06.430Z [cluster-test-akka.actor.default-dispatcher-4] [INFO] [akka://cluster-test/user/master] Message [akka.cluster.singleton.ClusterSingletonManager$Internal$HandOverToMe$] from Actor[akka.tcp://cluster-test@node01:2552/user/master#-131426289] to Actor[akka://cluster-test/user/master] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:14:06.493Z [cluster-test-akka.actor.default-dispatcher-4] [INFO] [akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-6] Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://cluster-test@node01:2552/system/cluster/core/daemon#-987111555] to Actor[akka://cluster-test/system/cluster/core/daemon/joinSeedNodeProcess-6#1534537651] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:14:06.511Z [cluster-test-cluster-dispatcher-30] [INFO] [akka.cluster.Cluster(akka://cluster-test)] Cluster Node [akka.tcp://cluster-test@node03:2552] - Welcome from [akka.tcp://cluster-test@node02:2552]\r\n2016-07-12T10:14:07.441Z [cluster-test-akka.actor.default-dispatcher-3] [INFO] [akka://cluster-test/user/master] Message [akka.cluster.singleton.ClusterSingletonManager$Internal$HandOverToMe$] from Actor[akka.tcp://cluster-test@node01:2552/user/master#-131426289] to Actor[akka://cluster-test/user/master] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:14:08.446Z [cluster-test-akka.actor.default-dispatcher-32] [INFO] [akka://cluster-test/user/master] Message [akka.cluster.singleton.ClusterSingletonManager$Internal$HandOverToMe$] from Actor[akka.tcp://cluster-test@node01:2552/user/master#-131426289] to Actor[akka://cluster-test/user/master] was not delivered. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.\r\n2016-07-12T10:14:09.210Z [cluster-test-lp-dispatcher-46] [INFO] [akka.tcp://cluster-test@node03:2552/user/master] ClusterSingletonManager state change [Start -> Younger]\r\n2016-07-12T10:14:09.456Z [cluster-test-lp-dispatcher-48] [WARN] [akka.tcp://cluster-test@node03:2552/user/master] unhandled event HandOverToMe in state Younger\r\n...\r\n2016-07-12T10:14:35.662Z [cluster-test-lp-dispatcher-48] [WARN] [akka.tcp://cluster-test@node03:2552/user/master] unhandled event HandOverToMe in state Younger\r\n\r\n\r\nWhat is confusing: a new incarnation of previously oldest is discovered and new oldest is still trying to do the handover, which leads to unhandled message in new incarnation and it takes 1 minute to remove previous oldest (and that `CleanupTimer` interval is not configurable at all)\r\nin general, it takes more time than in fact is needed to do the handover of the cluster singleton.\r\n\r\nI can see 3 options to solve that:\r\n1. handle HandOverToMe in state Younger, if previous oldest is the same address\r\n2. don't retry handover in case of new incarnation detected as it will anyway be in a wrong state\r\n3. do cleanup earlier, so that previous oldest is removed sooner, when new incarnation is started\r\n\r\nperhaps, all of them are worth doing..."},"comment":{"url":"https://api.github.com/repos/akka/akka/issues/comments/233281789","html_url":"https://github.com/akka/akka/issues/20942#issuecomment-233281789","issue_url":"https://api.github.com/repos/akka/akka/issues/20942","id":233281789,"user":{"login":"412b","id":12436202,"avatar_url":"https://avatars.githubusercontent.com/u/12436202?v=3","gravatar_id":"","url":"https://api.github.com/users/412b","html_url":"https://github.com/412b","followers_url":"https://api.github.com/users/412b/followers","following_url":"https://api.github.com/users/412b/following{/other_user}","gists_url":"https://api.github.com/users/412b/gists{/gist_id}","starred_url":"https://api.github.com/users/412b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/412b/subscriptions","organizations_url":"https://api.github.com/users/412b/orgs","repos_url":"https://api.github.com/users/412b/repos","events_url":"https://api.github.com/users/412b/events{/privacy}","received_events_url":"https://api.github.com/users/412b/received_events","type":"User","site_admin":false},"created_at":"2016-07-18T09:19:42Z","updated_at":"2016-07-18T09:19:42Z","body":"@patriknw waht are the drawback of reorganizing the ifs there? does it make sense to spin in retry, if `previousOldestOption forall removed.contains` is `true`?"}}
{"actor":{"display_login":"412b"}}