<p>I am working through the persistent disks tutorial found <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/persistent-disk" rel="noreferrer">here</a> while also creating it as a StatefulSet instead of a deployment.</p>

<p>When I run the yaml file into GKE the database fails to start, looking at the logs it has the following error.</p>

<blockquote>
  <p>[ERROR] --initialize specified but the data directory has files in it. Aborting.</p>
</blockquote>

<p>Is it possible to inspect the volume created to see what is in the directory? Otherwise, what am I doing wrong that is causing the disk to be non empty?</p>

<p>Thanks</p>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: datalayer-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: datalayer-svc
  labels:
    app: myapplication
spec:
  ports:
  - port: 80
    name: dbadmin
  clusterIP: None
  selector:
    app: database
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: datalayer
spec:
  selector:
    matchLabels:
      app: myapplication
  serviceName: "datalayer-svc"
  replicas: 1
  template:
    metadata:
      labels:
        app: myapplication
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: database
        image: mysql:5.7.22
        env:
        - name: "MYSQL_ROOT_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: mysql-root-password
              key: password
        - name: "MYSQL_DATABASE"
          value: "appdatabase"
        - name: "MYSQL_USER"
          value: "app_user"
        - name: "MYSQL_PASSWORD"
          value: "app_password"
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: datalayer-pv
          mountPath: /var/lib/mysql
      volumes:
      - name: datalayer-pv
        persistentVolumeClaim:
          claimName: datalayer-pvc
</code></pre>

<p>This issue could be caused by the <code>lost+found</code> directory on the filesystem of the PersistentVolume.</p>

<p>I was able to verify this by adding a <code>k8s.gcr.io/busybox</code> container (in PVC set <code>accessModes: [ReadWriteMany]</code>, OR comment out the <code>database</code> container):</p>

<pre><code>- name: init
  image: "k8s.gcr.io/busybox"
  command: ["/bin/sh","-c","ls -l /var/lib/mysql"]
  volumeMounts:
  - name: database
    mountPath: "/var/lib/mysql"
</code></pre>

<p>There are a few potential workarounds...</p>

<p>Most preferable is to use a <a href="https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath" rel="noreferrer"><code>subPath</code></a> on the <code>volumeMounts</code> object. This uses a subdirectory of the PersistentVolume, which should be empty at creation time, instead of the volume root:</p>

<pre><code>volumeMounts:
- name: database
  mountPath: "/var/lib/mysql"
  subPath: mysql
</code></pre>

<p>Less preferable workarounds include:</p>

<ul>
<li>Use a one-time container to <code>rm -rf /var/lib/mysql/lost+found</code> (not a great solution, because the directory is managed by the filesystem and is likely to re-appear)</li>
<li>Use <code>mysql:5</code> image, and add <code>args: ["--ignore-db-dir=lost+found"]</code> to the container (this option was removed in mysql 8)</li>
<li>Use <code>mariadb</code> image instead of <code>mysql</code></li>
</ul>

<p>More details might be available at docker-library/mysql issues: <a href="https://github.com/docker-library/mysql/issues/69" rel="noreferrer">#69</a> and <a href="https://github.com/docker-library/mysql/issues/186" rel="noreferrer">#186</a></p>
