{"action":"created","comment":{"url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/comments/74927961","id":74927961,"diff_hunk":"@@ -592,6 +593,8 @@ private class FailureDetector(sourceEndpointId: String, logName: String, failure\n     case FailureDetected(cause) =>\n       causes = causes :+ cause\n     case FailureDetectionLimitReached(scheduledCount) if scheduledCount == counter =>\n+      log.error(causes.lastOption.getOrElse(Logging.Error.NoCause), \"replication failure detection limit reached ({}),\" +\n+        \" publishing Unavailable for {}/{} (last exception being reported)\", failureDetectionLimit, sourceEndpointId, logName)","path":"eventuate-core/src/main/scala/com/rbmhtechnology/eventuate/ReplicationEndpoint.scala","position":49,"original_position":49,"commit_id":"e926849f6fae42eeb368a6839a1943fe7fe1a58f","original_commit_id":"e926849f6fae42eeb368a6839a1943fe7fe1a58f","user":{"login":"volkerstampa","id":2352337,"avatar_url":"https://avatars.githubusercontent.com/u/2352337?v=3","gravatar_id":"","url":"https://api.github.com/users/volkerstampa","html_url":"https://github.com/volkerstampa","followers_url":"https://api.github.com/users/volkerstampa/followers","following_url":"https://api.github.com/users/volkerstampa/following{/other_user}","gists_url":"https://api.github.com/users/volkerstampa/gists{/gist_id}","starred_url":"https://api.github.com/users/volkerstampa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/volkerstampa/subscriptions","organizations_url":"https://api.github.com/users/volkerstampa/orgs","repos_url":"https://api.github.com/users/volkerstampa/repos","events_url":"https://api.github.com/users/volkerstampa/events{/privacy}","received_events_url":"https://api.github.com/users/volkerstampa/received_events","type":"User","site_admin":false},"body":"another nitpick: avoid splitting the string with `+` and start the new line before `\"replication...`","created_at":"2016-08-16T12:53:21Z","updated_at":"2016-08-16T12:53:21Z","html_url":"https://github.com/RBMHTechnology/eventuate/pull/298#discussion_r74927961","pull_request_url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298","_links":{"self":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/comments/74927961"},"html":{"href":"https://github.com/RBMHTechnology/eventuate/pull/298#discussion_r74927961"},"pull_request":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298"}}},"pull_request":{"url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298","id":79560711,"html_url":"https://github.com/RBMHTechnology/eventuate/pull/298","diff_url":"https://github.com/RBMHTechnology/eventuate/pull/298.diff","patch_url":"https://github.com/RBMHTechnology/eventuate/pull/298.patch","issue_url":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/298","number":298,"state":"open","locked":false,"title":"Change intermediate replication failure logs from error to warning","user":{"login":"magro","id":64285,"avatar_url":"https://avatars.githubusercontent.com/u/64285?v=3","gravatar_id":"","url":"https://api.github.com/users/magro","html_url":"https://github.com/magro","followers_url":"https://api.github.com/users/magro/followers","following_url":"https://api.github.com/users/magro/following{/other_user}","gists_url":"https://api.github.com/users/magro/gists{/gist_id}","starred_url":"https://api.github.com/users/magro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magro/subscriptions","organizations_url":"https://api.github.com/users/magro/orgs","repos_url":"https://api.github.com/users/magro/repos","events_url":"https://api.github.com/users/magro/events{/privacy}","received_events_url":"https://api.github.com/users/magro/received_events","type":"User","site_admin":false},"body":"...and add an ERROR log for the final failure when the failure detection limit is reached.\r\n\r\n**Motivation**\r\n\r\nWe're regularly observing \"replication read failed\" ERRORs logged by\r\n`com.rbmhtechnology.eventuate.Replicator`, caused by a\r\n\r\n    com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 30000 milliseconds\r\n\r\nWhile of cause this highlights an issue with the underlying network,\r\nit's nothing special because the network is just not reliable.\r\nThese cases are also gracefully handled by eventuate because it retries\r\nand only when the failure detection limit is reached the replication endpoint\r\nis considered and reported as unavailable.\r\n\r\nLogs with level ERROR would also require our ops team to take some action\r\nor at least it would need some attention, while we'd say this is not necessary.\r\n\r\nTherefore the log level for temporary failures is changed from ERROR\r\nto WARN. Additionally the exception's stack trace is omitted, because it's\r\nnot carrying much useful information, the exception + message should be sufficient\r\n(additionally akka's logger simply doesn't allow to provide an exception for\r\na warning).  For reference, the stack trace is s.th. like this:\r\n\r\n    com.rbmhtechnology.eventuate.ReplicationProtocol$ReplicationReadTimeoutException: Replication read timed out after 30000 milliseconds\r\n            at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:543) ~[com.rbmhtechnology.eventuate-core_2.11-0.7.1.jar:0.7.1]\r\n            at com.rbmhtechnology.eventuate.Replicator$$anonfun$com$rbmhtechnology$eventuate$Replicator$$read$1.applyOrElse(ReplicationEndpoint.scala:542) ~[com.rbmhtechnology.eventuate-core_2.11-0.7.1.jar:0.7.1]\r\n            at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.util.Try$.apply(Try.scala:192) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.util.Failure.recover(Try.scala:216) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.concurrent.impl.CallbackRunnable.run_aroundBody0(Promise.scala:32) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n            at scala.concurrent.impl.CallbackRunnable$AjcClosure1.run(Promise.scala:1) ~[org.scala-lang.scala-library-2.11.8.jar:na]\r\n\r\nThe final failure then is considered being an issue which should in fact\r\nget some attention, therefore an additional ERROR log is added for this case.\r\nThe last exception is also provided for this ERROR log, or akka's `Error.NoCause`\r\nif no last exception available (akka handles `Error.NoCause` in a way as if\r\n`error(message)` was called).","created_at":"2016-08-01T12:33:27Z","updated_at":"2016-08-16T12:53:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"942068ea40c92b5de06651ad83a3ec5aac26ab85","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298/commits","review_comments_url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298/comments","review_comment_url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/comments{/number}","comments_url":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/298/comments","statuses_url":"https://api.github.com/repos/RBMHTechnology/eventuate/statuses/e926849f6fae42eeb368a6839a1943fe7fe1a58f","head":{"label":"magro:adjust-replication-logs","ref":"adjust-replication-logs","sha":"e926849f6fae42eeb368a6839a1943fe7fe1a58f","user":{"login":"magro","id":64285,"avatar_url":"https://avatars.githubusercontent.com/u/64285?v=3","gravatar_id":"","url":"https://api.github.com/users/magro","html_url":"https://github.com/magro","followers_url":"https://api.github.com/users/magro/followers","following_url":"https://api.github.com/users/magro/following{/other_user}","gists_url":"https://api.github.com/users/magro/gists{/gist_id}","starred_url":"https://api.github.com/users/magro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magro/subscriptions","organizations_url":"https://api.github.com/users/magro/orgs","repos_url":"https://api.github.com/users/magro/repos","events_url":"https://api.github.com/users/magro/events{/privacy}","received_events_url":"https://api.github.com/users/magro/received_events","type":"User","site_admin":false},"repo":{"id":64664733,"name":"eventuate","full_name":"magro/eventuate","owner":{"login":"magro","id":64285,"avatar_url":"https://avatars.githubusercontent.com/u/64285?v=3","gravatar_id":"","url":"https://api.github.com/users/magro","html_url":"https://github.com/magro","followers_url":"https://api.github.com/users/magro/followers","following_url":"https://api.github.com/users/magro/following{/other_user}","gists_url":"https://api.github.com/users/magro/gists{/gist_id}","starred_url":"https://api.github.com/users/magro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magro/subscriptions","organizations_url":"https://api.github.com/users/magro/orgs","repos_url":"https://api.github.com/users/magro/repos","events_url":"https://api.github.com/users/magro/events{/privacy}","received_events_url":"https://api.github.com/users/magro/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/magro/eventuate","description":"Event-sourcing and distributed event collaboration with causal consistency","fork":true,"url":"https://api.github.com/repos/magro/eventuate","forks_url":"https://api.github.com/repos/magro/eventuate/forks","keys_url":"https://api.github.com/repos/magro/eventuate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/magro/eventuate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/magro/eventuate/teams","hooks_url":"https://api.github.com/repos/magro/eventuate/hooks","issue_events_url":"https://api.github.com/repos/magro/eventuate/issues/events{/number}","events_url":"https://api.github.com/repos/magro/eventuate/events","assignees_url":"https://api.github.com/repos/magro/eventuate/assignees{/user}","branches_url":"https://api.github.com/repos/magro/eventuate/branches{/branch}","tags_url":"https://api.github.com/repos/magro/eventuate/tags","blobs_url":"https://api.github.com/repos/magro/eventuate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/magro/eventuate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/magro/eventuate/git/refs{/sha}","trees_url":"https://api.github.com/repos/magro/eventuate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/magro/eventuate/statuses/{sha}","languages_url":"https://api.github.com/repos/magro/eventuate/languages","stargazers_url":"https://api.github.com/repos/magro/eventuate/stargazers","contributors_url":"https://api.github.com/repos/magro/eventuate/contributors","subscribers_url":"https://api.github.com/repos/magro/eventuate/subscribers","subscription_url":"https://api.github.com/repos/magro/eventuate/subscription","commits_url":"https://api.github.com/repos/magro/eventuate/commits{/sha}","git_commits_url":"https://api.github.com/repos/magro/eventuate/git/commits{/sha}","comments_url":"https://api.github.com/repos/magro/eventuate/comments{/number}","issue_comment_url":"https://api.github.com/repos/magro/eventuate/issues/comments{/number}","contents_url":"https://api.github.com/repos/magro/eventuate/contents/{+path}","compare_url":"https://api.github.com/repos/magro/eventuate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/magro/eventuate/merges","archive_url":"https://api.github.com/repos/magro/eventuate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/magro/eventuate/downloads","issues_url":"https://api.github.com/repos/magro/eventuate/issues{/number}","pulls_url":"https://api.github.com/repos/magro/eventuate/pulls{/number}","milestones_url":"https://api.github.com/repos/magro/eventuate/milestones{/number}","notifications_url":"https://api.github.com/repos/magro/eventuate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/magro/eventuate/labels{/name}","releases_url":"https://api.github.com/repos/magro/eventuate/releases{/id}","deployments_url":"https://api.github.com/repos/magro/eventuate/deployments","created_at":"2016-08-01T12:23:57Z","updated_at":"2016-08-01T12:23:58Z","pushed_at":"2016-08-01T12:31:10Z","git_url":"git://github.com/magro/eventuate.git","ssh_url":"git@github.com:magro/eventuate.git","clone_url":"https://github.com/magro/eventuate.git","svn_url":"https://github.com/magro/eventuate","homepage":"http://rbmhtechnology.github.io/eventuate/","size":8929,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"RBMHTechnology:master","ref":"master","sha":"45743b6d592164e80c939163275cef84acc3b15e","user":{"login":"RBMHTechnology","id":10194147,"avatar_url":"https://avatars.githubusercontent.com/u/10194147?v=3","gravatar_id":"","url":"https://api.github.com/users/RBMHTechnology","html_url":"https://github.com/RBMHTechnology","followers_url":"https://api.github.com/users/RBMHTechnology/followers","following_url":"https://api.github.com/users/RBMHTechnology/following{/other_user}","gists_url":"https://api.github.com/users/RBMHTechnology/gists{/gist_id}","starred_url":"https://api.github.com/users/RBMHTechnology/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RBMHTechnology/subscriptions","organizations_url":"https://api.github.com/users/RBMHTechnology/orgs","repos_url":"https://api.github.com/users/RBMHTechnology/repos","events_url":"https://api.github.com/users/RBMHTechnology/events{/privacy}","received_events_url":"https://api.github.com/users/RBMHTechnology/received_events","type":"Organization","site_admin":false},"repo":{"id":29150668,"name":"eventuate","full_name":"RBMHTechnology/eventuate","owner":{"login":"RBMHTechnology","id":10194147,"avatar_url":"https://avatars.githubusercontent.com/u/10194147?v=3","gravatar_id":"","url":"https://api.github.com/users/RBMHTechnology","html_url":"https://github.com/RBMHTechnology","followers_url":"https://api.github.com/users/RBMHTechnology/followers","following_url":"https://api.github.com/users/RBMHTechnology/following{/other_user}","gists_url":"https://api.github.com/users/RBMHTechnology/gists{/gist_id}","starred_url":"https://api.github.com/users/RBMHTechnology/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RBMHTechnology/subscriptions","organizations_url":"https://api.github.com/users/RBMHTechnology/orgs","repos_url":"https://api.github.com/users/RBMHTechnology/repos","events_url":"https://api.github.com/users/RBMHTechnology/events{/privacy}","received_events_url":"https://api.github.com/users/RBMHTechnology/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/RBMHTechnology/eventuate","description":"Event-sourcing and distributed event collaboration with causal consistency","fork":false,"url":"https://api.github.com/repos/RBMHTechnology/eventuate","forks_url":"https://api.github.com/repos/RBMHTechnology/eventuate/forks","keys_url":"https://api.github.com/repos/RBMHTechnology/eventuate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/RBMHTechnology/eventuate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/RBMHTechnology/eventuate/teams","hooks_url":"https://api.github.com/repos/RBMHTechnology/eventuate/hooks","issue_events_url":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/events{/number}","events_url":"https://api.github.com/repos/RBMHTechnology/eventuate/events","assignees_url":"https://api.github.com/repos/RBMHTechnology/eventuate/assignees{/user}","branches_url":"https://api.github.com/repos/RBMHTechnology/eventuate/branches{/branch}","tags_url":"https://api.github.com/repos/RBMHTechnology/eventuate/tags","blobs_url":"https://api.github.com/repos/RBMHTechnology/eventuate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/RBMHTechnology/eventuate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/RBMHTechnology/eventuate/git/refs{/sha}","trees_url":"https://api.github.com/repos/RBMHTechnology/eventuate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/RBMHTechnology/eventuate/statuses/{sha}","languages_url":"https://api.github.com/repos/RBMHTechnology/eventuate/languages","stargazers_url":"https://api.github.com/repos/RBMHTechnology/eventuate/stargazers","contributors_url":"https://api.github.com/repos/RBMHTechnology/eventuate/contributors","subscribers_url":"https://api.github.com/repos/RBMHTechnology/eventuate/subscribers","subscription_url":"https://api.github.com/repos/RBMHTechnology/eventuate/subscription","commits_url":"https://api.github.com/repos/RBMHTechnology/eventuate/commits{/sha}","git_commits_url":"https://api.github.com/repos/RBMHTechnology/eventuate/git/commits{/sha}","comments_url":"https://api.github.com/repos/RBMHTechnology/eventuate/comments{/number}","issue_comment_url":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/comments{/number}","contents_url":"https://api.github.com/repos/RBMHTechnology/eventuate/contents/{+path}","compare_url":"https://api.github.com/repos/RBMHTechnology/eventuate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/RBMHTechnology/eventuate/merges","archive_url":"https://api.github.com/repos/RBMHTechnology/eventuate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/RBMHTechnology/eventuate/downloads","issues_url":"https://api.github.com/repos/RBMHTechnology/eventuate/issues{/number}","pulls_url":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls{/number}","milestones_url":"https://api.github.com/repos/RBMHTechnology/eventuate/milestones{/number}","notifications_url":"https://api.github.com/repos/RBMHTechnology/eventuate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/RBMHTechnology/eventuate/labels{/name}","releases_url":"https://api.github.com/repos/RBMHTechnology/eventuate/releases{/id}","deployments_url":"https://api.github.com/repos/RBMHTechnology/eventuate/deployments","created_at":"2015-01-12T18:47:39Z","updated_at":"2016-08-15T12:16:00Z","pushed_at":"2016-08-12T11:25:11Z","git_url":"git://github.com/RBMHTechnology/eventuate.git","ssh_url":"git@github.com:RBMHTechnology/eventuate.git","clone_url":"https://github.com/RBMHTechnology/eventuate.git","svn_url":"https://github.com/RBMHTechnology/eventuate","homepage":"http://rbmhtechnology.github.io/eventuate/","size":9422,"stargazers_count":400,"watchers_count":400,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":49,"mirror_url":null,"open_issues_count":35,"forks":49,"open_issues":35,"watchers":400,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298"},"html":{"href":"https://github.com/RBMHTechnology/eventuate/pull/298"},"issue":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/298"},"comments":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/issues/298/comments"},"review_comments":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298/comments"},"review_comment":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/pulls/298/commits"},"statuses":{"href":"https://api.github.com/repos/RBMHTechnology/eventuate/statuses/e926849f6fae42eeb368a6839a1943fe7fe1a58f"}}}}
{"actor":{"display_login":"volkerstampa"}}