{"action":"created","issue":{"url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1670","repository_url":"https://api.github.com/repos/akkadotnet/akka.net","labels_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1670/labels{/name}","comments_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1670/comments","events_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1670/events","html_url":"https://github.com/akkadotnet/akka.net/issues/1670","id":128853835,"number":1670,"title":"Web application does not rejoin akka.cluster after restart","user":{"login":"voltcode","id":5236618,"avatar_url":"https://avatars.githubusercontent.com/u/5236618?v=3","gravatar_id":"","url":"https://api.github.com/users/voltcode","html_url":"https://github.com/voltcode","followers_url":"https://api.github.com/users/voltcode/followers","following_url":"https://api.github.com/users/voltcode/following{/other_user}","gists_url":"https://api.github.com/users/voltcode/gists{/gist_id}","starred_url":"https://api.github.com/users/voltcode/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voltcode/subscriptions","organizations_url":"https://api.github.com/users/voltcode/orgs","repos_url":"https://api.github.com/users/voltcode/repos","events_url":"https://api.github.com/users/voltcode/events{/privacy}","received_events_url":"https://api.github.com/users/voltcode/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-01-26T15:25:43Z","updated_at":"2016-02-19T23:01:58Z","closed_at":null,"body":"My setup:\r\n\r\na [wcfservice] role talks to [mainservice] role (a windows service) that talks to [webapp] role (an mvc app)\r\n\r\n1.I start up [mainservice], [wcfservice], [webapp] (in this order)\r\n\r\nall looks good in the logs, cluster is formed.\r\n\r\n2.message is send from wcfservice to mainservice. mainservice processes it and sends a message to webapp. All works as expected\r\n\r\n3.I kill [webapp]. gossip failures show up in [mainservice] logs (as expected).\r\n\r\n4.I restart webapp, there is no message confirming that it rejoined the cluster. Sending new message as in p.2. does not work.\r\n\r\nI would expect that the webapp would rejoin the cluster automatically on startup. \r\nIf I kill the [mainservice] together with [webapp], and start them both up (mainservice first) then cluster is formed correctly (as in situation from p.1).\r\n\r\nakka std out from [mainservice] logs edited to show when p.2,3,4 occur (look for !!!>>>)\r\n```\r\nConfiguration Result:\r\n[Success] Name AkkaPoC.Grid.Service\r\n[Success] ServiceName AkkaPoC.Grid.Service\r\nTopshelf v3.3.154.0, .NET Framework v4.0.30319.34209\r\n[INFO][2016-01-26 15:07:41][Thread 0001][remoting] Starting remoting\r\n[INFO][2016-01-26 15:07:41][Thread 0001][remoting] Remoting started; listening on addresses : [akka.tcp://akkapocsystem@localhost:4053]\r\n[INFO][2016-01-26 15:07:41][Thread 0001][remoting] Remoting now listens on addresses: [akka.tcp://akkapocsystem@localhost:4053]\r\n[INFO][2016-01-26 15:07:42][Thread 0001][Cluster] Cluster Node [akka.tcp://akkapocsystem@localhost:4053] - Starting up...\r\nThe AkkaPoC.Grid.Service service is now running, press Control+C to exit.\r\n[INFO][2016-01-26 15:07:42][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:4053] is JOINING, roles [mainservice]\r\n[INFO][2016-01-26 15:07:43][Thread 0017][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:4053] to [Up]\r\n[INFO][2016-01-26 15:08:00][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:60873] is JOINING, roles [webapp]\r\n[INFO][2016-01-26 15:08:01][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Leader is moving node [akka.tcp://akkapocsystem@localhost:60873] to [Up]\r\n!!!>>>HERE THE event is passed from mainservice to [webapp] CORRECTLY<<!!!\r\n!!!>>> AFTERWARDS I KILL the [webapp] <<<!!!\r\n[WARNING][2016-01-26 15:08:40][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2]] Association with remote system akka.tcp://akkapocsystem@localhost:60873 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointDisassociatedException: Disassociated\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.Unhandled(Object message)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)\r\n   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)]\r\n[ERROR][2016-01-26 15:08:40][Thread 0011][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2] Disassociated\r\nCause: Akka.Remote.EndpointDisassociatedException: Disassociated\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.Unhandled(Object message)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.ReceivedTerminated(Terminated t)\r\n   at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n[INFO][2016-01-26 15:08:40][Thread 0006][akka://akkapocsystem/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fakkapocsystem%40127.0.0.1%3a60874-2] Message DisassociateUnderlying from akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2/endpointWriter to akka://akkapocsystem/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fakkapocsystem%40127.0.0.1%3a60874-2 was not delivered. 1 dead letters encountered.\r\n[INFO][2016-01-26 15:08:40][Thread 0006][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2] Message DeathWatchNotification from akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2 to akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-2 was not delivered. 2 dead letters encountered.\r\n[INFO][2016-01-26 15:08:41][Thread 0004][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 3 dead letters encountered.\r\n[INFO][2016-01-26 15:08:41][Thread 0007][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 4 dead letters encountered.\r\n[INFO][2016-01-26 15:08:42][Thread 0019][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 5 dead letters encountered.\r\n[INFO][2016-01-26 15:08:42][Thread 0007][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 6 dead letters encountered.\r\n[INFO][2016-01-26 15:08:43][Thread 0019][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 7 dead letters encountered.\r\n[INFO][2016-01-26 15:08:43][Thread 0006][akka://akkapocsystem/deadLetters] Message Heartbeat from akka://akkapocsystem/system/cluster/core/daemon/heartbeatSender to akka://akkapocsystem/deadLetters was not delivered. 8 dead letters encountered.\r\n[INFO][2016-01-26 15:08:44][Thread 0019][akka://akkapocsystem/deadLetters] Message GossipStatus from akka://akkapocsystem/system/cluster/core/daemon to akka://akkapocsystem/deadLetters was not delivered. 9 dead letters encountered.\r\n[WARNING][2016-01-26 15:08:45][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Cluster Node [akka.tcp://akkapocsystem@localhost:4053] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://akkapocsystem@localhost:60873, status = Up]\r\n[WARNING][2016-01-26 15:08:47][Thread 0011][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-3/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:08:47][Thread 0009][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:08:47][Thread 0009][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-3] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n[WARNING][2016-01-26 15:08:54][Thread 0010][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-4/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:08:54][Thread 0009][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:08:54][Thread 0009][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-4] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n[INFO][2016-01-26 15:08:56][Thread 0018][[akka://akkapocsystem/system/cluster/core/daemon]] Node [akka.tcp://akkapocsystem@localhost:60928] is JOINING, roles [webapp]\r\n[WARNING][2016-01-26 15:09:02][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-6/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:09:02][Thread 0008][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:09:02][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-6] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n[INFO][2016-01-26 15:09:04][Thread 0016][[akka://akkapocsystem/system/cluster/core/daemon]] Leader can currently not perform its duties, reachability status: [akka.tcp://akkapocsystem@localhost:4053 -> UniqueAddress: (akka.tcp://akkapocsystem@localhost:60873, 1093223717): Unreachable [Unreachable] (1), ], member status: [$akka.tcp://akkapocsystem@localhost:4053 $Up seen=$True, $akka.tcp://akkapocsystem@localhost:60873 $Up seen=$False, $akka.tcp://akkapocsystem@localhost:60928 $Joining seen=$True]\r\n[WARNING][2016-01-26 15:09:08][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-7/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:09:08][Thread 0008][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:09:08][Thread 0008][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-7] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n[WARNING][2016-01-26 15:09:15][Thread 0009][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-8/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:09:15][Thread 0009][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:09:15][Thread 0009][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-8] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n!!!>>> HERE I RESPAWN the [webapp] <<<!!!\r\n!!!>>> HERE new  mainservice sends new message to [webapp] but it is not received <<<!!!\r\n[WARNING][2016-01-26 15:09:22][Thread 0010][[akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-9/endpointWriter]] AssociationError [akka.tcp://akkapocsystem@localhost:4053] -> akka.tcp://akkapocsystem@localhost:60873: Error [Invalid address: akka.tcp://akkapocsystem@localhost:60873] []\r\n[WARNING][2016-01-26 15:09:22][Thread 0010][remoting] Tried to associate with unreachable remote address [akka.tcp://akkapocsystem@localhost:60873]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Invalid address: akka.tcp://akkapocsystem@localhost:60873] Caused by: [Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---]\r\n[ERROR][2016-01-26 15:09:22][Thread 0010][akka://akkapocsystem/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fakkapocsystem%40localhost%3a60873-9] Invalid address: akka.tcp://akkapocsystem@localhost:60873\r\nCause: Akka.Remote.InvalidAssociation: Invalid address: akka.tcp://akkapocsystem@localhost:60873 ---> Akka.Remote.Transport.InvalidAssociationException: Association failure ---> Helios.Exceptions.HeliosConnectionException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873 ---> System.Net.Sockets.SocketException: Nie moľna nawiĄza† poĄczenia, poniewaľ komputer docelowy aktywnie go odmawia 127.0.0.1:60873\r\n   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   --- End of inner exception stack trace ---\r\n   at Helios.Net.Connections.TcpConnection.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosHelpers.Open()\r\n   at Akka.Remote.Transport.Helios.CommonHandlers.Open()\r\n   at Akka.Remote.Transport.Helios.HeliosTcpTransport.AssociateInternal(Address remoteAddress)\r\n   at Akka.Remote.Transport.Helios.HeliosTransport.<Associate>d__27.MoveNext()\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---\r\n   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)\r\n   at Akka.Remote.EndpointWriter.<Initializing>b__23_1(Failure failure)\r\n   at lambda_method(Closure , Object , Action`1 , Action`1 , Action`1 )\r\n   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`4.Handle(T value)\r\n   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)\r\n   at Akka.Actor.ReceiveActor.OnReceive(Object message)\r\n   at Akka.Actor.UntypedActor.Receive(Object message)\r\n   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)\r\n   at Akka.Actor.ActorCell.ReceiveMessage(Object message)\r\n   at Akka.Actor.ActorCell.Invoke(Envelope envelope)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at Akka.Actor.ActorCell.HandleFailed(Failed f)\r\n   at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope)\r\n```\r\n\r\n\r\n[mainservice] hocon\r\n```\r\n<hocon>\r\n            <![CDATA[\r\n\t\t\t\t\takka {\r\n\t\t\t\t\t\tactor {\r\n\t\t\t\t\t\t\tprovider = \"Akka.Cluster.ClusterActorRefProvider, Akka.Cluster\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t                                                        \r\n\t\t\t\t\t\t}\r\n\t\t\t\t\t\t\r\n\t\t\t\t\t\tremote {                            \r\n\t\t\t\t\t\t\tlog-remote-lifecycle-events = DEBUG\r\n\t\t\t\t\t\t\tlog-received-messages = on\r\n                            log-sent-messages = on\r\n\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\thelios.tcp {\r\n\t\t\t\t\t\t\t\ttransport-class = \"Akka.Remote.Transport.Helios.HeliosTcpTransport, Akka.Remote\"\r\n    \t    \t\t\t\t\thostname = localhost\r\n\t\t\t\t\t\t\t\tport = 4053\r\n\t\t\t\t\t\t\t}\r\n\t\t\t\t\t\t}            \r\n\r\n\t\t\t\t\t\tcluster {\r\n\t\t\t\t\t\t\t#will inject this node as a self-seed node at run-time\r\n\t\t\t\t\t\t\tseed-nodes = [\"akka.tcp://akkapocsystem@localhost:4053\"] \r\n\t\t\t\t\t\t\troles = [\"mainservice\"]\r\n\t\t\t\t\t\t}\r\n\t\t\t\t\t}\r\n\t\t\t]]>\r\n        </hocon>\r\n```"},"comment":{"url":"https://api.github.com/repos/akkadotnet/akka.net/issues/comments/186442375","html_url":"https://github.com/akkadotnet/akka.net/issues/1670#issuecomment-186442375","issue_url":"https://api.github.com/repos/akkadotnet/akka.net/issues/1670","id":186442375,"user":{"login":"voltcode","id":5236618,"avatar_url":"https://avatars.githubusercontent.com/u/5236618?v=3","gravatar_id":"","url":"https://api.github.com/users/voltcode","html_url":"https://github.com/voltcode","followers_url":"https://api.github.com/users/voltcode/followers","following_url":"https://api.github.com/users/voltcode/following{/other_user}","gists_url":"https://api.github.com/users/voltcode/gists{/gist_id}","starred_url":"https://api.github.com/users/voltcode/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voltcode/subscriptions","organizations_url":"https://api.github.com/users/voltcode/orgs","repos_url":"https://api.github.com/users/voltcode/repos","events_url":"https://api.github.com/users/voltcode/events{/privacy}","received_events_url":"https://api.github.com/users/voltcode/received_events","type":"User","site_admin":false},"created_at":"2016-02-19T23:01:58Z","updated_at":"2016-02-19T23:01:58Z","body":"exact log message (proof of Ungate unhanlded):\r\n\r\n2016-02-20 00:00:43,714 [11] DEBUG Akka.Actor.LocalActorRef [(null)] - Unhandled message from akka://voltcode/system/endpointManager : Akka.Remote.ReliableDeliverySupervisor+Ungate"}}