<p>I deployed Kubernetes on a bare metal dedicated server using <code>conjure-up kubernetes</code> on Ubuntu 18.04 LTS. This also means the nodes are LXD containers.</p>

<p>I need persistent volumes for Elasticsearch and MongoDB, and after some research I decided that the simplest way of getting that to work in my deployment was an NFS share.
I created an NFS share in the host OS, with the following configuration:</p>

<blockquote>
  <p>/srv/volumes 127.0.0.1(rw) 10.78.69.*(rw,no_root_squash)</p>
</blockquote>

<p><code>10.78.69.*</code> appears to be the bridge network used by Kubernetes, at least looking at ifconfig there's nothing else.</p>

<p>Then I proceeded to create two folders, /srv/volumes/1 and /srv/volumes/2
I created two PVs from these folders with this configuration for the first (the second is similar):</p>

<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: elastic-pv1
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /srv/volumes/1
    server: 10.78.69.1
</code></pre>

<p>Then I deploy the Elasticsearch helm chart (<a href="https://github.com/helm/charts/tree/master/incubator/elasticsearch" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/incubator/elasticsearch</a>) and it creates two claims which successfully bind to my PVs.</p>

<p>The issue is that afterwards the containers seem to encounter errors:</p>

<blockquote>
  <p>Error: failed to start container "sysctl": Error response from daemon: linux runtime spec devices: lstat /dev/.lxc/proc/17848/fdinfo/24: no such file or directory
  Back-off restarting failed container</p>
</blockquote>

<p><a href="https://i.stack.imgur.com/NckA5.png" rel="nofollow noreferrer">Pods view</a></p>

<p><a href="https://i.stack.imgur.com/Yqogu.png" rel="nofollow noreferrer">Persistent Volume Claims view</a></p>

<p>I'm kinda stuck here. I've tried searching for the error but I haven't been able to find a solution to this issue. </p>

<p>Previously before I set the allowed IP in <code>/etc/exports</code> to <code>10.78.69.*</code> Kubernetes would tell me it got "permission denied" from the NFS server while trying to mount, so I assume that now mounting succeeded, since that error disappeared.</p>

<p><strong>EDIT:</strong> </p>

<p>I decided to purge the helm deployment and try again, this time with a different storage type, local-storage volumes. I created them following the guide from Canonical, and I know they work because I set up one for MongoDB this way and it works perfectly.</p>

<p>The configuration for the elasticsearch helm deployment changed since now I have to set affinity for the nodes on which the persistent volumes were created:</p>

<p><code>values.yaml</code>:</p>

<pre><code>data:
    replicas: 1,
    nodeSelector:
        elasticsearch: data
master:
    replicas: 1,
    nodeSelector:
        elasticsearch: master
client:
    replicas: 1,
cluster:
    env: {MINIMUM_MASTER_NODES: "1"}
</code></pre>

<p>I deployed using </p>

<blockquote>
  <p>helm install --name site-search -f values.yaml incubator/elasticsearch</p>
</blockquote>

<p>These are the only changes, however elasticsearch still presents the same issues.</p>

<p>Additional information:</p>

<p><code>kubectl version</code>:</p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>The elasticsearch image is the default one in the helm chart:</p>

<blockquote>
  <p>docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.1</p>
</blockquote>

<p>The various pods' (master, client, data) logs are empty.
The error is the same.</p>

<p>I was able to solve the issue by running <code>sysctl -w vm.max_map_count=262144</code> myself on the host machine, and removing the "sysctl" init container which was trying to do this unsuccessfully.</p>
