<p>I am trying to run indy-nodes in kubernetes. These indy nodes are sandbox nodes and write data in <code>/var/lib/indy</code> directory inside the container. When I run the pod with a volume mounted, it does not write anything in the volume directory. Although it creates a directory inside the volume, it is empty all the time. However, when I create a pod without a volume mount option, the container writes data inside <code>/var/lib/indy</code>. </p>

<p>Following is the Dockerfile:<br>
Hastebin: <a href="https://hastebin.com/hitinefizi.nginx" rel="noreferrer">https://hastebin.com/hitinefizi.nginx</a></p>

<p>Kubernetes Deployment:</p>

<pre><code>{{- $root := .}}
{{- range .Values.indy}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  # namespace: {{$root.Values.namespace}}
  name: {{.name}}
spec:
  selector:
    matchLabels:
      name: {{.name}}
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: {{.name}}
    spec:
      containers:
        - name: {{.name}}
          image: {{.image}}
          volumeMounts:
            - name: {{$root.Values.pv.metadata.name}}
              mountPath: "/var/lib/indy/sandbox"
              subPath: "volume/indy/{{.name}}/sandbox"
          ports:
            - containerPort: {{ index .ports 0 }}
            - containerPort: {{ index .ports 1 }}
      nodeSelector:
        nodeType: {{$root.Values.hosts.blockchain}}
      volumes:
        - name: {{$root.Values.pv.metadata.name}}
          {{- if eq $root.Values.storage.type "nfs" }}
          persistentVolumeClaim:
            claimName: {{$root.Values.pvc.metadata.name}}
          {{- else }}
          hostPath:
            path: /var/kubeshare/
          {{- end }}
{{- end}}
</code></pre>

<p>The directory inside volume:</p>

<pre><code>[root@centos1 kubeshare]# tree volume/indy/
volume/indy/
|-- indy-node1
|-- indy-node2
|-- indy-node3
`-- indy-node
</code></pre>

<p>The directory <code>/var/lib/indy</code> inside the container without volume:</p>

<pre><code>root@indy-node1-587c4758bf-2hpp6:/var/lib/indy# tree -L 3
.
|-- plugins
`-- sandbox
    |-- data
    |   `-- Node1
    |-- domain_transactions_genesis
    |-- keys
    |   |-- Node1
    |   |-- Node1C
    |   |-- Node2
    |   |-- Node3
    |   `-- Node4
    |-- node1_additional_info.json
    |-- node1_info.json
    |-- node1_version_info.json
    `-- pool_transactions_genesis
</code></pre>

<p>I am not sure why it is happening. Any help/suggestions would be appreciated.</p>

<p>Update: This is the same thing happening with docker-compose when I try to use local volume.</p>

<p>Mounting in docker is consistent with standard behaviour of mounting on Linux. 
Linux <code>mount</code> command <a href="https://linux.die.net/man/8/mount" rel="nofollow noreferrer">docs say</a> </p>

<p><code>The previous contents (if any) and owner and mode of dir become invisible, and as long as this filesystem remains mounted</code></p>

<p>This is as well the way things work in Docker. If you mount a local directory, or an <strong>existing</strong> named docker volume, the content of filesystem in the container on the location of the mount will be shadowed (or we can call it "overriden").</p>

<h1>Simplified example of what is going on</h1>

<p>Having dockerfile</p>

<pre><code>FROM alpine:3.9.6

WORKDIR /home/root/greetings
RUN echo "hello world" &gt; /home/root/greetings/english.txt
CMD sleep 60000
</code></pre>

<p>And build it <code>docker build -t greetings:1.0 .</code></p>

<p>Now create following <code>docker-compose.yml</code>:</p>

<pre><code>version: '3.7'

services:
  greetings:
    container_name: greetings
    image: greetings:1.0
    volumes:
      - ./empty:/home/root/greetings
</code></pre>

<p>and create empty directory <code>empty</code> next to it.</p>

<p>Start it <code>docker-compose up -d</code>. While the container is running, let's get into container and see what the filestructure inside looks like. <code>docker exec -ti greetings sh</code>. Now when we are inside, if you run <code>ls /home/root/greetings</code> you'll see that the directory is empty - even though in the Dockerfile we have baked file <code>/home/root/greetings/english.txt</code> into the image's filesystem.</p>

<p>Named docker containers behave more desirably, if the named docker container is new and doesn't contain any data. If you mount such container on location in  container where there already is some data, the named volume will get this data copied on it.</p>

<p>You can try this by adjusting the <code>docker-compose.yml</code> to this</p>

<pre><code>version: '3.7'

services:
  greetings:
    container_name: greetings
    image: greetings:1.0
    volumes:
      - greetingsvol:/home/root/greetings

volumes:
  greetingsvol:
    driver: local
</code></pre>

<p>and if you repeat the procedure and exec yourself into the container, you'll see that file <code>/home/root/greetings/english.txt</code> is still there.</p>

<p>That's because when you <code>cd</code> yourself into <code>/home/root/greetings</code>, you are not looking at actual container's filesystem, but at mounted device - the name docker volume - which has been initialized by copy of container's original files on that given location. (Assuming docker volume <code>greetingsvol</code> did not previously exist.)</p>

<h1>Solution to your problem</h1>

<p>You are mounting directory <code>/var/kubeshare</code> on your host to container's <code>/var/lib/indy/sandbox</code>. Let's see what the container stores on that location on startup (<code>indypool</code> is how I named built indy sandbox image on my localhost)</p>

<pre><code>docker run --rm indypool ls /var/lib/indy/sandbox
domain_transactions_genesis
keys
pool_transactions_genesis
</code></pre>

<p>So if you mount your local directory onto <code>/var/lib/indy/sandbox</code>, it will shadow these files and the pool will fail to start up (and therefore consequently won't create files such as <code>node1_additional_info.json</code> etc). </p>

<p>So I think you have 2 options:</p>

<ol>
<li>Unless you have a good reason not to, use named docker volumes.</li>
<li>Copy the original image data from container's <code>/var/lib/indy/sandbox</code> into your <code>/var/kubeshare</code>. Then you keep everything else as was. That way, the directory will be shadowed by new filesystem containing exactly the same data as the container expects to find there.</li>
</ol>
