{"action":"created","issue":{"url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1458","labels_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1458/labels{/name}","comments_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1458/comments","events_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1458/events","html_url":"https://github.com/akkadotnet/akka.net/issues/1458","id":118430995,"number":1458,"title":"Member not being removed to allow new incarnation of that member to join.","user":{"login":"cgstevens","id":11180136,"avatar_url":"https://avatars.githubusercontent.com/u/11180136?v=3","gravatar_id":"","url":"https://api.github.com/users/cgstevens","html_url":"https://github.com/cgstevens","followers_url":"https://api.github.com/users/cgstevens/followers","following_url":"https://api.github.com/users/cgstevens/following{/other_user}","gists_url":"https://api.github.com/users/cgstevens/gists{/gist_id}","starred_url":"https://api.github.com/users/cgstevens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgstevens/subscriptions","organizations_url":"https://api.github.com/users/cgstevens/orgs","repos_url":"https://api.github.com/users/cgstevens/repos","events_url":"https://api.github.com/users/cgstevens/events{/privacy}","received_events_url":"https://api.github.com/users/cgstevens/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/akkadotnet/akka.net/labels/akka-cluster","name":"akka-cluster","color":"c7def8"},{"url":"https://api.github.com/repos/akkadotnet/akka.net/labels/confirmed%20bug","name":"confirmed bug","color":"e11d21"},{"url":"https://api.github.com/repos/akkadotnet/akka.net/labels/critical","name":"critical","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"Aaronontheweb","id":326939,"avatar_url":"https://avatars.githubusercontent.com/u/326939?v=3","gravatar_id":"","url":"https://api.github.com/users/Aaronontheweb","html_url":"https://github.com/Aaronontheweb","followers_url":"https://api.github.com/users/Aaronontheweb/followers","following_url":"https://api.github.com/users/Aaronontheweb/following{/other_user}","gists_url":"https://api.github.com/users/Aaronontheweb/gists{/gist_id}","starred_url":"https://api.github.com/users/Aaronontheweb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Aaronontheweb/subscriptions","organizations_url":"https://api.github.com/users/Aaronontheweb/orgs","repos_url":"https://api.github.com/users/Aaronontheweb/repos","events_url":"https://api.github.com/users/Aaronontheweb/events{/privacy}","received_events_url":"https://api.github.com/users/Aaronontheweb/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/akkadotnet/akka.net/milestones/6","html_url":"https://github.com/akkadotnet/akka.net/milestones/Akka.NET%20v1.1","labels_url":"https://api.github.com/repos/akkadotnet/akka.net/milestones/6/labels","id":1199488,"number":6,"title":"Akka.NET v1.1","description":"Per the [Akka.NET 2015-2016 roadmap](https://petabridge.com/blog/akkadotnet-2015-roadmap/), first minor version milestone is to ship a stable release of Akka.Cluster.","creator":{"login":"Aaronontheweb","id":326939,"avatar_url":"https://avatars.githubusercontent.com/u/326939?v=3","gravatar_id":"","url":"https://api.github.com/users/Aaronontheweb","html_url":"https://github.com/Aaronontheweb","followers_url":"https://api.github.com/users/Aaronontheweb/followers","following_url":"https://api.github.com/users/Aaronontheweb/following{/other_user}","gists_url":"https://api.github.com/users/Aaronontheweb/gists{/gist_id}","starred_url":"https://api.github.com/users/Aaronontheweb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Aaronontheweb/subscriptions","organizations_url":"https://api.github.com/users/Aaronontheweb/orgs","repos_url":"https://api.github.com/users/Aaronontheweb/repos","events_url":"https://api.github.com/users/Aaronontheweb/events{/privacy}","received_events_url":"https://api.github.com/users/Aaronontheweb/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":19,"state":"open","created_at":"2015-07-08T20:53:14Z","updated_at":"2016-01-09T00:19:16Z","due_on":"2015-10-01T07:00:00Z","closed_at":null},"comments":42,"created_at":"2015-11-23T17:31:08Z","updated_at":"2016-01-10T01:50:33Z","closed_at":null,"body":"I have a 2 websites that are part of my cluster. Sometimes one websites will become unreachable and my logic for all of the other members determine that after 120 seconds they should do a Cluster.Down(ThatWebsiteAddress). I can see that all of the members get a MemberRemoved for that website and then it is reported as Down.\r\nMy problem is that when my other service, Solarwinds, detects that this website is down and has been removed from the cluster it tries restarts that website. I can see see it trying to join and I get this message: [[akka://MyService/system/cluster/core/daemon]] - New incarnation of existing member [UniqueAddress: (akka.tcp://MyService@1.1.9.2:57771, 303375918)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.\r\nBut the website member never gets removed from the cluster so the member that is coming back up is  never able to join. I can try to have a member .Leave(ThatWebsiteAddress) and do another .Down(ThatWebsiteAddress) but it never gets removed. Basically I have a ClusterStatus actor that monitors the status of the cluster from its view and determines if it needs to restart itself or if a member has been unreachable for x seconds to down it. If so it shuts that service down and logs to the event log for Solarwinds to determine if the service or website needs to be started back up.\r\nIn order for me to recover from this issue I have to restart the entire cluster.  Note... This has happened to one of my services so it isn't website specific but happens more on the websites as they become unreachable more often.\r\n\r\nIf you are interested here is a snapshot of my cluster status page which shows the member being down and never gets removed. When I view the cluster information about the website there are no members or cluster leader.\r\n\r\n![image](https://cloud.githubusercontent.com/assets/11180136/11344141/8f451e34-91dd-11e5-848d-5fd9255e17ca.png)\r\n\r\nI can force a down on a ServiceWorker which will cause it to be removed from cluster just fine.  When it started back up it join the cluster just fine.\r\n\r\nHere you can see I downed the service worker.\r\n![image](https://cloud.githubusercontent.com/assets/11180136/11344167/a2d98624-91dd-11e5-8008-f9385ec4bb27.png)\r\n\r\nHere you can see the service worker joining the cluster after being detected that it was down.... still the website isn't joining...  \r\n![image](https://cloud.githubusercontent.com/assets/11180136/11344176/b3496830-91dd-11e5-885e-a175f20d2bb9.png)\r\n"},"comment":{"url":"https://api.github.com/repos/akkadotnet/akka.net/issues/comments/170300085","html_url":"https://github.com/akkadotnet/akka.net/issues/1458#issuecomment-170300085","issue_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1458","id":170300085,"user":{"login":"easuter","id":4330134,"avatar_url":"https://avatars.githubusercontent.com/u/4330134?v=3","gravatar_id":"","url":"https://api.github.com/users/easuter","html_url":"https://github.com/easuter","followers_url":"https://api.github.com/users/easuter/followers","following_url":"https://api.github.com/users/easuter/following{/other_user}","gists_url":"https://api.github.com/users/easuter/gists{/gist_id}","starred_url":"https://api.github.com/users/easuter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/easuter/subscriptions","organizations_url":"https://api.github.com/users/easuter/orgs","repos_url":"https://api.github.com/users/easuter/repos","events_url":"https://api.github.com/users/easuter/events{/privacy}","received_events_url":"https://api.github.com/users/easuter/received_events","type":"User","site_admin":false},"created_at":"2016-01-10T01:50:33Z","updated_at":"2016-01-10T01:50:33Z","body":"@Aaronontheweb I'm using the Akka+Akka.Remote nightly and so far things are looking good. It's still early to tell if the problem is completely solved, so I'll post an update later on when I can confirm that the fix works for sure!"}}