{"action":"created","issue":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/233","repository_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster","labels_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/233/labels{/name}","comments_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/233/comments","events_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/233/events","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/233","id":362943283,"node_id":"MDU6SXNzdWUzNjI5NDMyODM=","number":233,"title":"es-data nodes exceeding Xmx memory","user":{"login":"fortuneFelix","id":2322703,"node_id":"MDQ6VXNlcjIzMjI3MDM=","avatar_url":"https://avatars0.githubusercontent.com/u/2322703?v=4","gravatar_id":"","url":"https://api.github.com/users/fortuneFelix","html_url":"https://github.com/fortuneFelix","followers_url":"https://api.github.com/users/fortuneFelix/followers","following_url":"https://api.github.com/users/fortuneFelix/following{/other_user}","gists_url":"https://api.github.com/users/fortuneFelix/gists{/gist_id}","starred_url":"https://api.github.com/users/fortuneFelix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fortuneFelix/subscriptions","organizations_url":"https://api.github.com/users/fortuneFelix/orgs","repos_url":"https://api.github.com/users/fortuneFelix/repos","events_url":"https://api.github.com/users/fortuneFelix/events{/privacy}","received_events_url":"https://api.github.com/users/fortuneFelix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-23T13:37:31Z","updated_at":"2018-10-01T15:42:24Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\nWe are running version 5.6.0 in our K8 1.10.3 cluster with docker version 1.13.1.\r\nThe es-data nodes, constantly exceed the configured the Xmx value.\r\nSettings:\r\nES_JAVA_OPTS=-Xms2048m -Xmx2048m\r\nKubernetes memory limit: 4096M\r\n\r\nFull JVM arguments:\r\n-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms2048m, -Xmx2048m, -Des.path.home=/elasticsearch\r\n\r\nThe es-data nodes grow beyond 4GB and then are getting killed.\r\n\r\nThe container we run is:\r\nquay.io/pires/docker-elasticsearch-kubernetes:5.6.0\r\n\r\nAny idea why this is happening? Any advice on how to improve this so es-data nodes stay within their limits?\r\n\r\nThanks a ton for any advice/help!\r\n\r\n- Felix"},"comment":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/comments/425955850","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/233#issuecomment-425955850","issue_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/233","id":425955850,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTk1NTg1MA==","user":{"login":"msmaverick2018","id":12723741,"node_id":"MDQ6VXNlcjEyNzIzNzQx","avatar_url":"https://avatars2.githubusercontent.com/u/12723741?v=4","gravatar_id":"","url":"https://api.github.com/users/msmaverick2018","html_url":"https://github.com/msmaverick2018","followers_url":"https://api.github.com/users/msmaverick2018/followers","following_url":"https://api.github.com/users/msmaverick2018/following{/other_user}","gists_url":"https://api.github.com/users/msmaverick2018/gists{/gist_id}","starred_url":"https://api.github.com/users/msmaverick2018/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/msmaverick2018/subscriptions","organizations_url":"https://api.github.com/users/msmaverick2018/orgs","repos_url":"https://api.github.com/users/msmaverick2018/repos","events_url":"https://api.github.com/users/msmaverick2018/events{/privacy}","received_events_url":"https://api.github.com/users/msmaverick2018/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T15:42:24Z","updated_at":"2018-10-01T15:42:24Z","author_association":"NONE","body":"Are you running into issue where JVM does not run under cgroup limitations set by the container and looks at host memory for limitations. Check the link below:\r\nhttps://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/\r\n\r\n-XX:+UnlockExperimentalVMOptions \\\r\n  -XX:+UseCGroupMemoryLimitForHeap \\\r\n  -XshowSettings:vm -version"}}
{"actor":{"display_login":"msmaverick2018"}}