{"action":"created","issue":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/205","repository_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster","labels_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/205/labels{/name}","comments_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/205/comments","events_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/205/events","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/205","id":339710437,"node_id":"MDU6SXNzdWUzMzk3MTA0Mzc=","number":205,"title":"Pods elasticsearch failed with \"Back-off restarting failed container\"","user":{"login":"maryxu","id":13269986,"node_id":"MDQ6VXNlcjEzMjY5OTg2","avatar_url":"https://avatars0.githubusercontent.com/u/13269986?v=4","gravatar_id":"","url":"https://api.github.com/users/maryxu","html_url":"https://github.com/maryxu","followers_url":"https://api.github.com/users/maryxu/followers","following_url":"https://api.github.com/users/maryxu/following{/other_user}","gists_url":"https://api.github.com/users/maryxu/gists{/gist_id}","starred_url":"https://api.github.com/users/maryxu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maryxu/subscriptions","organizations_url":"https://api.github.com/users/maryxu/orgs","repos_url":"https://api.github.com/users/maryxu/repos","events_url":"https://api.github.com/users/maryxu/events{/privacy}","received_events_url":"https://api.github.com/users/maryxu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-10T06:32:48Z","updated_at":"2018-07-10T07:20:36Z","closed_at":null,"author_association":"NONE","body":"Can you help to guide us about why Pods elasticsearch failed with \"Back-off restarting failed container\"? Thanks a lot!\r\n\r\n[root@####### ~]# kubectl describe pod efk3-elasticsearch-2 --namespace=efk --insecure-skip-tls-verify=true\r\nName:           efk3-elasticsearch-2\r\nNamespace:      efk\r\nNode:           lvdevk8sw23/10.219.161.3\r\nStart Time:     Tue, 03 Jul 2018 14:58:28 +0800\r\nLabels:         app=elasticsearch\r\n                component=master\r\n                controller-revision-hash=efk3-elasticsearch-569cf776f\r\n                release=efk3\r\n                statefulset.kubernetes.io/pod-name=efk3-elasticsearch-2\r\nAnnotations:    <none>\r\nStatus:         Running\r\nIP:             10.42.6.19\r\nControlled By:  StatefulSet/efk3-elasticsearch\r\nInit Containers:\r\n  sysctl:\r\n    Container ID:  docker://bed338fd0e395678abbbd1c49be7d14e1636faacdadba334c0e5607c4eb07251\r\n    Image:         busybox\r\n    Image ID:      docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47\r\n    Port:          <none>\r\n    Command:\r\n      sysctl\r\n      -w\r\n      vm.max_map_count=262144\r\n    State:          Terminated\r\n      Reason:       Completed\r\n      Exit Code:    0\r\n      Started:      Tue, 03 Jul 2018 14:58:32 +0800\r\n      Finished:     Tue, 03 Jul 2018 14:58:32 +0800\r\n    Ready:          True\r\n    Restart Count:  0\r\n    Environment:    <none>\r\n    Mounts:\r\n      /var/run/secrets/kubernetes.io/serviceaccount from efk3-elasticsearch-token-5vqzr (ro)\r\nContainers:\r\n  elasticsearch:\r\n    Container ID:   docker://336bfce82124136d78de952552de2f2688c5f4249c9c440b1259fd7b3e230046\r\n    Image:          docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4\r\n    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch-oss@sha256:2d9c774c536bd1f64abc4993ebc96a2344404d780cbeb81a8b3b4c3807550e57\r\n    Ports:          9300/TCP, 9200/TCP\r\n    State:          Waiting\r\n      Reason:       CrashLoopBackOff\r\n    Last State:     Terminated\r\n      Reason:       Error\r\n      Exit Code:    1\r\n      Started:      Tue, 10 Jul 2018 13:20:21 +0800\r\n      Finished:     Tue, 10 Jul 2018 13:20:26 +0800\r\n    Ready:          False\r\n    Restart Count:  1929\r\n    Limits:\r\n      cpu:  1\r\n    Requests:\r\n      cpu:      25m\r\n      memory:   512Mi\r\n    Readiness:  http-get http://:9200/_cluster/health%3Flocal=true delay=5s timeout=1s period=10s #success=1 #failure=3\r\n    Environment:\r\n      cluster.name:                        efk3-cluster\r\n      discovery.zen.ping.unicast.hosts:    efk3-elasticsearch\r\n      discovery.zen.minimum_master_nodes:  2\r\n      KUBERNETES_NAMESPACE:                efk (v1:metadata.namespace)\r\n      discovery.zen.ping.unicast.hosts:    efk3-elasticsearch\r\n      PROCESSORS:                          1 (limits.cpu)\r\n      ES_JAVA_OPTS:                        -Djava.net.preferIPv4Stack=true -Xms512m -Xmx512m\r\n    Mounts:\r\n      /usr/share/elasticsearch/data from data (rw)\r\n      /var/run/secrets/kubernetes.io/serviceaccount from efk3-elasticsearch-token-5vqzr (ro)\r\nConditions:\r\n  Type           Status\r\n  Initialized    True\r\n  Ready          False\r\n  PodScheduled   True\r\nVolumes:\r\n  data:\r\n    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\r\n    ClaimName:  data-efk3-elasticsearch-2\r\n    ReadOnly:   false\r\n  efk3-elasticsearch-token-5vqzr:\r\n    Type:        Secret (a volume populated by a Secret)\r\n    SecretName:  efk3-elasticsearch-token-5vqzr\r\n    Optional:    false\r\nQoS Class:       Burstable\r\nNode-Selectors:  <none>\r\nTolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\r\n                 node.kubernetes.io/unreachable:NoExecute for 300s\r\nEvents:\r\n  Type     Reason   Age                  From                  Message\r\n  ----     ------   ----                 ----                  -------\r\n  Normal   Pulled   48m (x1921 over 6d)  kubelet, lvdevk8sw23  Container image \"docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4\" already present on machine\r\n  Warning  BackOff  3m (x44220 over 6d)  kubelet, lvdevk8sw23  Back-off restarting failed container\r\n"},"comment":{"url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/comments/403725920","html_url":"https://github.com/pires/kubernetes-elasticsearch-cluster/issues/205#issuecomment-403725920","issue_url":"https://api.github.com/repos/pires/kubernetes-elasticsearch-cluster/issues/205","id":403725920,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzcyNTkyMA==","user":{"login":"mat1010","id":5108320,"node_id":"MDQ6VXNlcjUxMDgzMjA=","avatar_url":"https://avatars0.githubusercontent.com/u/5108320?v=4","gravatar_id":"","url":"https://api.github.com/users/mat1010","html_url":"https://github.com/mat1010","followers_url":"https://api.github.com/users/mat1010/followers","following_url":"https://api.github.com/users/mat1010/following{/other_user}","gists_url":"https://api.github.com/users/mat1010/gists{/gist_id}","starred_url":"https://api.github.com/users/mat1010/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mat1010/subscriptions","organizations_url":"https://api.github.com/users/mat1010/orgs","repos_url":"https://api.github.com/users/mat1010/repos","events_url":"https://api.github.com/users/mat1010/events{/privacy}","received_events_url":"https://api.github.com/users/mat1010/received_events","type":"User","site_admin":false},"created_at":"2018-07-10T07:20:36Z","updated_at":"2018-07-10T07:20:36Z","author_association":"NONE","body":"It looks like a permission issue\r\n\r\n```\r\nCaused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0/node.lock\r\n        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]\r\n        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]\r\n        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]\r\n        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[?:?]\r\n        at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[?:1.8.0_161]\r\n        at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_161]\r\n        at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n        at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n        at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]\r\n        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:209) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]\r\n        ... 6 more\r\n```\r\nElasticsearch seems to not be able to write into ` /usr/share/elasticsearch/data/`"}}
{"actor":{"display_login":"mat1010"}}