{"action":"created","comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments/59564707","id":59564707,"diff_hunk":"@@ -73,6 +76,7 @@\n     private final ILogger logger;\n     private final ScheduledExecutorService scheduler;\n     private final Address thisAddress;\n+    private final ConcurrentMap<Address, AtomicLong> lastHeartbeatPerMember = new ConcurrentHashMap<Address, AtomicLong>();","path":"hazelcast/src/main/java/com/hazelcast/spi/impl/operationservice/impl/InvocationMonitor.java","position":40,"original_position":17,"commit_id":"a52356db9d596b15b6e43a5f6ea2eb64232ad608","original_commit_id":"7722cb10e3417fd20d1a62dd932fb2a5a2ef60ba","user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"body":"Also other threads; any thread can make an investigation regarding the last heartbeat of that given member (for the invocationfuture error message for example).","created_at":"2016-04-13T15:02:57Z","updated_at":"2016-04-13T15:02:57Z","html_url":"https://github.com/hazelcast/hazelcast/pull/7952#discussion_r59564707","pull_request_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952","_links":{"self":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments/59564707"},"html":{"href":"https://github.com/hazelcast/hazelcast/pull/7952#discussion_r59564707"},"pull_request":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952"}}},"pull_request":{"url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952","id":66327079,"html_url":"https://github.com/hazelcast/hazelcast/pull/7952","diff_url":"https://github.com/hazelcast/hazelcast/pull/7952.diff","patch_url":"https://github.com/hazelcast/hazelcast/pull/7952.patch","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7952","number":7952,"state":"open","locked":false,"title":"Improved OperationTimeoutException","user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"body":"it includes the operation lastheartbeat of the member. This gives some insight what is happening\r\nwith the member communicaton. Imagine a fat packet is jamming the pipeline, then of\r\ncourse the operation will timeout, but the member heartbeat will also fall behind.\r\n\r\nSo if both member and invocation are very far behind; it is likely some kind of environmental\r\nissue. If the member is up to date and the invocation is not, then appearently something happened\r\nto the operation.","created_at":"2016-04-13T14:06:55Z","updated_at":"2016-04-13T15:02:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"12d1c31d4609d4899e7059e19cd2b05beb3b0ae1","assignee":null,"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/70","html_url":"https://github.com/hazelcast/hazelcast/milestones/3.7","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/70/labels","id":1346433,"number":70,"title":"3.7","description":"","creator":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"open_issues":194,"closed_issues":329,"state":"open","created_at":"2015-10-09T12:16:13Z","updated_at":"2016-04-13T14:06:55Z","due_on":"2016-05-02T21:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952/commits","review_comments_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952/comments","review_comment_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/7952/comments","statuses_url":"https://api.github.com/repos/hazelcast/hazelcast/statuses/a52356db9d596b15b6e43a5f6ea2eb64232ad608","head":{"label":"pveentjer:improvement/3.x/operationtimeoutexception-including-last-member-heartbeat","ref":"improvement/3.x/operationtimeoutexception-including-last-member-heartbeat","sha":"a52356db9d596b15b6e43a5f6ea2eb64232ad608","user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"repo":{"id":10990111,"name":"hazelcast","full_name":"pveentjer/hazelcast","owner":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pveentjer/hazelcast","description":"Open Source In-Memory Data Grid","fork":true,"url":"https://api.github.com/repos/pveentjer/hazelcast","forks_url":"https://api.github.com/repos/pveentjer/hazelcast/forks","keys_url":"https://api.github.com/repos/pveentjer/hazelcast/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pveentjer/hazelcast/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pveentjer/hazelcast/teams","hooks_url":"https://api.github.com/repos/pveentjer/hazelcast/hooks","issue_events_url":"https://api.github.com/repos/pveentjer/hazelcast/issues/events{/number}","events_url":"https://api.github.com/repos/pveentjer/hazelcast/events","assignees_url":"https://api.github.com/repos/pveentjer/hazelcast/assignees{/user}","branches_url":"https://api.github.com/repos/pveentjer/hazelcast/branches{/branch}","tags_url":"https://api.github.com/repos/pveentjer/hazelcast/tags","blobs_url":"https://api.github.com/repos/pveentjer/hazelcast/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pveentjer/hazelcast/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pveentjer/hazelcast/git/refs{/sha}","trees_url":"https://api.github.com/repos/pveentjer/hazelcast/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pveentjer/hazelcast/statuses/{sha}","languages_url":"https://api.github.com/repos/pveentjer/hazelcast/languages","stargazers_url":"https://api.github.com/repos/pveentjer/hazelcast/stargazers","contributors_url":"https://api.github.com/repos/pveentjer/hazelcast/contributors","subscribers_url":"https://api.github.com/repos/pveentjer/hazelcast/subscribers","subscription_url":"https://api.github.com/repos/pveentjer/hazelcast/subscription","commits_url":"https://api.github.com/repos/pveentjer/hazelcast/commits{/sha}","git_commits_url":"https://api.github.com/repos/pveentjer/hazelcast/git/commits{/sha}","comments_url":"https://api.github.com/repos/pveentjer/hazelcast/comments{/number}","issue_comment_url":"https://api.github.com/repos/pveentjer/hazelcast/issues/comments{/number}","contents_url":"https://api.github.com/repos/pveentjer/hazelcast/contents/{+path}","compare_url":"https://api.github.com/repos/pveentjer/hazelcast/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pveentjer/hazelcast/merges","archive_url":"https://api.github.com/repos/pveentjer/hazelcast/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pveentjer/hazelcast/downloads","issues_url":"https://api.github.com/repos/pveentjer/hazelcast/issues{/number}","pulls_url":"https://api.github.com/repos/pveentjer/hazelcast/pulls{/number}","milestones_url":"https://api.github.com/repos/pveentjer/hazelcast/milestones{/number}","notifications_url":"https://api.github.com/repos/pveentjer/hazelcast/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pveentjer/hazelcast/labels{/name}","releases_url":"https://api.github.com/repos/pveentjer/hazelcast/releases{/id}","deployments_url":"https://api.github.com/repos/pveentjer/hazelcast/deployments","created_at":"2013-06-27T07:15:08Z","updated_at":"2015-04-22T04:00:47Z","pushed_at":"2016-04-13T15:01:23Z","git_url":"git://github.com/pveentjer/hazelcast.git","ssh_url":"git@github.com:pveentjer/hazelcast.git","clone_url":"https://github.com/pveentjer/hazelcast.git","svn_url":"https://github.com/pveentjer/hazelcast","homepage":"www.hazelcast.com","size":90367,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"hazelcast:master","ref":"master","sha":"fd499349aba2dfd63fba0956f5eaebd7aee389c5","user":{"login":"hazelcast","id":1453152,"avatar_url":"https://avatars.githubusercontent.com/u/1453152?v=3","gravatar_id":"","url":"https://api.github.com/users/hazelcast","html_url":"https://github.com/hazelcast","followers_url":"https://api.github.com/users/hazelcast/followers","following_url":"https://api.github.com/users/hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazelcast/subscriptions","organizations_url":"https://api.github.com/users/hazelcast/orgs","repos_url":"https://api.github.com/users/hazelcast/repos","events_url":"https://api.github.com/users/hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/hazelcast/received_events","type":"Organization","site_admin":false},"repo":{"id":3786237,"name":"hazelcast","full_name":"hazelcast/hazelcast","owner":{"login":"hazelcast","id":1453152,"avatar_url":"https://avatars.githubusercontent.com/u/1453152?v=3","gravatar_id":"","url":"https://api.github.com/users/hazelcast","html_url":"https://github.com/hazelcast","followers_url":"https://api.github.com/users/hazelcast/followers","following_url":"https://api.github.com/users/hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazelcast/subscriptions","organizations_url":"https://api.github.com/users/hazelcast/orgs","repos_url":"https://api.github.com/users/hazelcast/repos","events_url":"https://api.github.com/users/hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/hazelcast/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hazelcast/hazelcast","description":"Open Source In-Memory Data Grid","fork":false,"url":"https://api.github.com/repos/hazelcast/hazelcast","forks_url":"https://api.github.com/repos/hazelcast/hazelcast/forks","keys_url":"https://api.github.com/repos/hazelcast/hazelcast/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hazelcast/hazelcast/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hazelcast/hazelcast/teams","hooks_url":"https://api.github.com/repos/hazelcast/hazelcast/hooks","issue_events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/events{/number}","events_url":"https://api.github.com/repos/hazelcast/hazelcast/events","assignees_url":"https://api.github.com/repos/hazelcast/hazelcast/assignees{/user}","branches_url":"https://api.github.com/repos/hazelcast/hazelcast/branches{/branch}","tags_url":"https://api.github.com/repos/hazelcast/hazelcast/tags","blobs_url":"https://api.github.com/repos/hazelcast/hazelcast/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hazelcast/hazelcast/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hazelcast/hazelcast/git/refs{/sha}","trees_url":"https://api.github.com/repos/hazelcast/hazelcast/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hazelcast/hazelcast/statuses/{sha}","languages_url":"https://api.github.com/repos/hazelcast/hazelcast/languages","stargazers_url":"https://api.github.com/repos/hazelcast/hazelcast/stargazers","contributors_url":"https://api.github.com/repos/hazelcast/hazelcast/contributors","subscribers_url":"https://api.github.com/repos/hazelcast/hazelcast/subscribers","subscription_url":"https://api.github.com/repos/hazelcast/hazelcast/subscription","commits_url":"https://api.github.com/repos/hazelcast/hazelcast/commits{/sha}","git_commits_url":"https://api.github.com/repos/hazelcast/hazelcast/git/commits{/sha}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/comments{/number}","issue_comment_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments{/number}","contents_url":"https://api.github.com/repos/hazelcast/hazelcast/contents/{+path}","compare_url":"https://api.github.com/repos/hazelcast/hazelcast/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hazelcast/hazelcast/merges","archive_url":"https://api.github.com/repos/hazelcast/hazelcast/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hazelcast/hazelcast/downloads","issues_url":"https://api.github.com/repos/hazelcast/hazelcast/issues{/number}","pulls_url":"https://api.github.com/repos/hazelcast/hazelcast/pulls{/number}","milestones_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones{/number}","notifications_url":"https://api.github.com/repos/hazelcast/hazelcast/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/labels{/name}","releases_url":"https://api.github.com/repos/hazelcast/hazelcast/releases{/id}","deployments_url":"https://api.github.com/repos/hazelcast/hazelcast/deployments","created_at":"2012-03-21T12:25:18Z","updated_at":"2016-04-13T03:16:42Z","pushed_at":"2016-04-13T15:01:27Z","git_url":"git://github.com/hazelcast/hazelcast.git","ssh_url":"git@github.com:hazelcast/hazelcast.git","clone_url":"https://github.com/hazelcast/hazelcast.git","svn_url":"https://github.com/hazelcast/hazelcast","homepage":"www.hazelcast.com","size":122275,"stargazers_count":1515,"watchers_count":1515,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":716,"mirror_url":null,"open_issues_count":610,"forks":716,"open_issues":610,"watchers":1515,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952"},"html":{"href":"https://github.com/hazelcast/hazelcast/pull/7952"},"issue":{"href":"https://api.github.com/repos/hazelcast/hazelcast/issues/7952"},"comments":{"href":"https://api.github.com/repos/hazelcast/hazelcast/issues/7952/comments"},"review_comments":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952/comments"},"review_comment":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hazelcast/hazelcast/pulls/7952/commits"},"statuses":{"href":"https://api.github.com/repos/hazelcast/hazelcast/statuses/a52356db9d596b15b6e43a5f6ea2eb64232ad608"}}}}