{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/12278","repository_url":"https://api.github.com/repos/hazelcast/hazelcast","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/12278/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/12278/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/12278/events","html_url":"https://github.com/hazelcast/hazelcast/issues/12278","id":295391380,"number":12278,"title":"split-heal brain / quorum OOME","user":{"login":"Danny-Hazelcast","id":5988678,"avatar_url":"https://avatars1.githubusercontent.com/u/5988678?v=4","gravatar_id":"","url":"https://api.github.com/users/Danny-Hazelcast","html_url":"https://github.com/Danny-Hazelcast","followers_url":"https://api.github.com/users/Danny-Hazelcast/followers","following_url":"https://api.github.com/users/Danny-Hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/Danny-Hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/Danny-Hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Danny-Hazelcast/subscriptions","organizations_url":"https://api.github.com/users/Danny-Hazelcast/orgs","repos_url":"https://api.github.com/users/Danny-Hazelcast/repos","events_url":"https://api.github.com/users/Danny-Hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/Danny-Hazelcast/received_events","type":"User","site_admin":false},"labels":[{"id":137752575,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"109def","default":false},{"id":85325332,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Critical","name":"Type: Critical","color":"e11d21","default":false},{"id":1740690,"url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Defect","name":"Type: Defect","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/98","html_url":"https://github.com/hazelcast/hazelcast/milestone/98","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/98/labels","id":2624978,"number":98,"title":"3.10","description":null,"creator":{"login":"jerrinot","id":158619,"avatar_url":"https://avatars1.githubusercontent.com/u/158619?v=4","gravatar_id":"","url":"https://api.github.com/users/jerrinot","html_url":"https://github.com/jerrinot","followers_url":"https://api.github.com/users/jerrinot/followers","following_url":"https://api.github.com/users/jerrinot/following{/other_user}","gists_url":"https://api.github.com/users/jerrinot/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrinot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrinot/subscriptions","organizations_url":"https://api.github.com/users/jerrinot/orgs","repos_url":"https://api.github.com/users/jerrinot/repos","events_url":"https://api.github.com/users/jerrinot/events{/privacy}","received_events_url":"https://api.github.com/users/jerrinot/received_events","type":"User","site_admin":false},"open_issues":165,"closed_issues":407,"state":"open","created_at":"2017-07-06T10:25:45Z","updated_at":"2018-02-14T19:01:54Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2018-02-08T05:36:28Z","updated_at":"2018-02-15T06:37:27Z","closed_at":null,"author_association":"MEMBER","body":"\r\nsplit heal quorum jenkins test job. \r\n\r\nthis run https://hazelcast-l337.ci.cloudbees.com/view/split/job/split-quorum/42/console \r\nFailed the multi Member OOME\r\n\r\nhttp://54.82.84.143/~jenkins/workspace/split-quorum/3.10-SNAPSHOT/2018_02_07-03_19_23/quorum/gc.html\r\n```\r\noutput/HZ/HzMember3HZBB/HzMember3HZBB.hprof\r\noutput/HZ/HzMember1HZBB/HzMember1HZBB.hprof\r\noutput/HZ/HzMember2HZBB/HzMember2HZBB.hprof\r\n```\r\n\r\nin the failing run case, all 3 members of the \"big\" cluster oome'd,\r\nand the small cluster GC is not looking to great also.\r\n\r\nThis is the first time I have seen an OOME from this test job. \r\n\r\nThe main action needed is to check the hprof, to make sure we have not encountered a leak of some \r\nkind. \r\n\r\n\r\n\r\nand an example of a normal passing run,  the next job number using the same hazelcast jars. \r\nhttps://hazelcast-l337.ci.cloudbees.com/view/split/job/split-quorum/43/console\r\nhttp://54.82.84.143/~jenkins/workspace/split-quorum/3.10-SNAPSHOT/2018_02_07-12_47_20/quorum/gc.html\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/365838265","html_url":"https://github.com/hazelcast/hazelcast/issues/12278#issuecomment-365838265","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/12278","id":365838265,"user":{"login":"Danny-Hazelcast","id":5988678,"avatar_url":"https://avatars1.githubusercontent.com/u/5988678?v=4","gravatar_id":"","url":"https://api.github.com/users/Danny-Hazelcast","html_url":"https://github.com/Danny-Hazelcast","followers_url":"https://api.github.com/users/Danny-Hazelcast/followers","following_url":"https://api.github.com/users/Danny-Hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/Danny-Hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/Danny-Hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Danny-Hazelcast/subscriptions","organizations_url":"https://api.github.com/users/Danny-Hazelcast/orgs","repos_url":"https://api.github.com/users/Danny-Hazelcast/repos","events_url":"https://api.github.com/users/Danny-Hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/Danny-Hazelcast/received_events","type":"User","site_admin":false},"created_at":"2018-02-15T06:37:27Z","updated_at":"2018-02-15T06:37:27Z","author_association":"MEMBER","body":"for this run https://hazelcast-l337.ci.cloudbees.com/view/split/job/split-quorum/50/consoleFull\r\n\r\nI disabled all Durable executor test.\r\n\r\nGC charts still show a leak pattern.  but very slow\r\nhttp://54.82.84.143/~jenkins/workspace/split-quorum/3.10-SNAPSHOT/2018_02_15-00_35_09/quorum/gc.html\r\n\r\n\r\nthe heap sizes used was\r\n```\r\nhz memberOps \"-Xms300M -Xmx300M ${hzProps}\"\r\nhz clientOps \"-Xms100M -Xmx100M ${hzProps}\"\r\n```\r\n\r\nvery small heap for the 50 iteration loop test.\r\n\r\n"}}
{"actor":{"display_login":"Danny-Hazelcast"}}