{"action":"created","issue":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/204","repository_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster","labels_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/204/labels{/name}","comments_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/204/comments","events_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/204/events","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/204","id":339344196,"node_id":"MDU6SXNzdWUzMzkzNDQxOTY=","number":204,"title":"Data node Crash Loop ","user":{"login":"ivanovaleksandar","id":803369,"node_id":"MDQ6VXNlcjgwMzM2OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/803369?v=4","gravatar_id":"","url":"https://api.github.com/users/ivanovaleksandar","html_url":"https://github.com/ivanovaleksandar","followers_url":"https://api.github.com/users/ivanovaleksandar/followers","following_url":"https://api.github.com/users/ivanovaleksandar/following{/other_user}","gists_url":"https://api.github.com/users/ivanovaleksandar/gists{/gist_id}","starred_url":"https://api.github.com/users/ivanovaleksandar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivanovaleksandar/subscriptions","organizations_url":"https://api.github.com/users/ivanovaleksandar/orgs","repos_url":"https://api.github.com/users/ivanovaleksandar/repos","events_url":"https://api.github.com/users/ivanovaleksandar/events{/privacy}","received_events_url":"https://api.github.com/users/ivanovaleksandar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-07-09T08:02:42Z","updated_at":"2018-07-09T08:53:53Z","closed_at":null,"author_association":"NONE","body":"The data node is constantly restarting in the initialization phase after working well for few days. \r\n\r\n```\r\nkubectl logs es-data-0 -f\r\n\r\n[2018-07-09T07:53:35,420][INFO ][o.e.n.Node               ] [es-data-0] initializing ...\r\n[2018-07-09T07:53:35,572][INFO ][o.e.e.NodeEnvironment    ] [es-data-0] using [1] data paths, mounts [[/data (<ip-addr>:vol_016257cd4643b18711209d769f93e979)]], net usable_space [1.9gb], net total_space [2.9gb], types [fuse.glusterfs]\r\n[2018-07-09T07:53:35,573][INFO ][o.e.e.NodeEnvironment    ] [es-data-0] heap size [2.9gb], compressed ordinary object pointers [true]\r\n```\r\nAfter this it crashes and starts a new init once again.\r\n\r\nThis the the yaml that I am using. I've set up requests and limits accordingly (the java heap is smaller that the pod limits, as people suggested), but with no avail.\r\n\r\n```\r\napiVersion: apps/v1beta1\r\nkind: StatefulSet\r\nmetadata:\r\n  name: es-data\r\n  labels:\r\n    component: elasticsearch\r\n    role: data\r\nspec:\r\n  serviceName: elasticsearch-data\r\n  replicas: 2\r\n  template:\r\n    metadata:\r\n      labels:\r\n        component: elasticsearch\r\n        role: data\r\n    spec:\r\n      initContainers:\r\n      - name: init-sysctl\r\n        image: busybox:1.27.2\r\n        command:\r\n        - sysctl\r\n        - -w\r\n        - vm.max_map_count=262144\r\n        securityContext:\r\n          privileged: true\r\n      containers:\r\n      - name: es-data\r\n        image: quay.io/pires/docker-elasticsearch-kubernetes:6.3.0\r\n        env:\r\n        - name: NAMESPACE\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.namespace\r\n        - name: NODE_NAME\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.name\r\n        - name: CLUSTER_NAME\r\n          value: myesdb\r\n        - name: NODE_MASTER\r\n          value: \"false\"\r\n        - name: NODE_INGEST\r\n          value: \"false\"\r\n        - name: HTTP_ENABLE\r\n          value: \"true\"\r\n        - name: ES_JAVA_OPTS\r\n          value: -Xms3g -Xmx3g\r\n        - name: PROCESSORS\r\n          valueFrom:\r\n            resourceFieldRef:\r\n              resource: limits.cpu\r\n        resources:\r\n          requests:\r\n            memory: 3Gi\r\n          limits:\r\n            cpu: 2\r\n            memory: 4Gi\r\n        ports:\r\n        - containerPort: 9200\r\n          name: http\r\n        - containerPort: 9300\r\n          name: transport\r\n        livenessProbe:\r\n          tcpSocket:\r\n            port: transport\r\n          initialDelaySeconds: 20\r\n          periodSeconds: 10\r\n        readinessProbe:\r\n          httpGet:\r\n            path: /_cluster/health\r\n            port: http\r\n          initialDelaySeconds: 20\r\n          timeoutSeconds: 5\r\n        volumeMounts:\r\n        - name: storage\r\n          mountPath: /data\r\n  volumeClaimTemplates:\r\n  - metadata:\r\n      name: storage\r\n    spec:\r\n      storageClassName: gluster-heketi-external\r\n      accessModes: [ ReadWriteOnce ]\r\n      resources:\r\n        requests:\r\n          storage: 3Gi\r\n```\r\n\r\nAny idea or suggestion?\r\n"},"comment":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/comments/403408187","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/204#issuecomment-403408187","issue_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/204","id":403408187,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzQwODE4Nw==","user":{"login":"ivanovaleksandar","id":803369,"node_id":"MDQ6VXNlcjgwMzM2OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/803369?v=4","gravatar_id":"","url":"https://api.github.com/users/ivanovaleksandar","html_url":"https://github.com/ivanovaleksandar","followers_url":"https://api.github.com/users/ivanovaleksandar/followers","following_url":"https://api.github.com/users/ivanovaleksandar/following{/other_user}","gists_url":"https://api.github.com/users/ivanovaleksandar/gists{/gist_id}","starred_url":"https://api.github.com/users/ivanovaleksandar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivanovaleksandar/subscriptions","organizations_url":"https://api.github.com/users/ivanovaleksandar/orgs","repos_url":"https://api.github.com/users/ivanovaleksandar/repos","events_url":"https://api.github.com/users/ivanovaleksandar/events{/privacy}","received_events_url":"https://api.github.com/users/ivanovaleksandar/received_events","type":"User","site_admin":false},"created_at":"2018-07-09T08:53:53Z","updated_at":"2018-07-09T08:53:53Z","author_association":"NONE","body":"Hey, that worked. :)\r\nThat was a trivial overlook from my side. But, as the cluster gets in some data to process and startup the services, the readiness/liveliness need to be adjusted probably.\r\n\r\nThank you @mat1010 ! I will close the issue now."}}
{"actor":{"display_login":"ivanovaleksandar"}}