<p>I have deployed a MySQL database (statefulset) on Kubernetes zonal cluster, running as a service (GKE) in Google Cloud Platform.</p>
<p>The zonal cluster consist of 3 instances of type e2-medium.</p>
<p>The MySQL container cannot start due to the following error.</p>
<pre><code>kubectl logs mysql-statefulset-0
2022-02-07 05:55:38+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.35-1debian10 started.
find: '/var/lib/mysql/': Input/output error
</code></pre>
<p>Last seen events.</p>
<pre><code>4m57s   Warning   Ext4Error   gke-cluster-default-pool-rnfh   kernel-monitor, gke-cluster-default-pool-rnfh   EXT4-fs error (device sdb): __ext4_find_entry:1532: inode #2: comm mysqld: reading directory lblock 0   40d   8062   gke-cluster-default-pool-rnfh
3m22s   Warning   BackOff     pod/mysql-statefulset-0   spec.containers{mysql}   kubelet, gke-cluster-default-pool-rnfh   Back-off restarting failed container
</code></pre>
<p>Nodes.</p>
<pre><code>kubectl get node -owide
gke-cluster-default-pool-ayqo   Ready    &lt;none&gt;   54d   v1.21.5-gke.1302   So.Me.I.P   So.Me.I.P    Container-Optimized OS from Google   5.4.144+         containerd://1.4.8
gke-cluster-default-pool-rnfh   Ready    &lt;none&gt;   54d   v1.21.5-gke.1302   So.Me.I.P   So.Me.I.P   Container-Optimized OS from Google   5.4.144+         containerd://1.4.8
gke-cluster-default-pool-sc3p   Ready    &lt;none&gt;   54d   v1.21.5-gke.1302   So.Me.I.P   So.Me.I.P     Container-Optimized OS from Google   5.4.144+         containerd://1.4.8
</code></pre>
<p>I also noticed that rnfh node is out of memory.</p>
<pre><code>kubectl top node
NAME                            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
gke-cluster-default-pool-ayqo   117m         12%    992Mi           35%
gke-cluster-default-pool-rnfh   180m         19%    2953Mi          104%
gke-cluster-default-pool-sc3p   179m         19%    1488Mi          52%
</code></pre>
<p>MySql mainfest</p>
<pre><code># HEADLESS SERVICE
apiVersion: v1
kind: Service
metadata:
  name: mysql-headless-service
  labels:
    kind: mysql-headless-service
spec:
  clusterIP: None
  selector:
    tier: mysql-db
  ports:
    - name: 'mysql-http'
      protocol: 'TCP'
      port: 3306
---
# STATEFUL SET
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-statefulset
spec:
  selector:
    matchLabels:
      tier: mysql-db
  serviceName: mysql-statefulset
  replicas: 1
  template:
    metadata:
      labels:
        tier: mysql-db
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: my-mysql
          image: my-mysql:latest
          imagePullPolicy: Always
          args:
            - &quot;--ignore-db-dir=lost+found&quot;
          ports:
            - name: 'http'
              protocol: 'TCP'
              containerPort: 3306
          volumeMounts:
            - name: mysql-pvc
              mountPath: /var/lib/mysql
          env:
            - name: MYSQL_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: mysql-root-username
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: mysql-root-password
            - name: MYSQL_USER
              valueFrom:
                configMapKeyRef:
                  name: mysql-config
                  key: mysql-username
            - name: MYSQL_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: mysql-config
                  key: mysql-password
            - name: MYSQL_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: mysql-config
                  key: mysql-database
  volumeClaimTemplates:
    - metadata:
        name: mysql-pvc
      spec:
        storageClassName: 'mysql-fast'
        resources:
          requests:
            storage: 120Gi
        accessModes:
          - ReadWriteOnce
          - ReadOnlyMany
</code></pre>
<p>MySQL storage class manifest:</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mysql-fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate
</code></pre>
<p>Why Kubernetes is trying to schedule pod in out of memory node?</p>
<p><strong>UPDATES</strong></p>
<p>I've added requests and limits to <code>MySQL</code> manifest to improve the <code>Qos Class</code>. Now the <code>Qos Class</code> is <code>Guaranteed</code>.</p>
<p>Unfortunately, Kubernetes still trying to schedule to out of memory <code>rnfh</code> node.</p>
<pre><code>kubectl describe po mysql-statefulset-0 | grep node -i
Node: gke-cluster-default-pool-rnfh/So.Me.I.P

kubectl describe po mysql-statefulset-0 | grep qos -i
QoS Class: Guaranteed
</code></pre>

Mikolaj
<p>I ran a few more tests but I couldn't replicate this.</p>
<p>To answer this one correctly, we would need much more logs. Not sure if you still have them. If I could guess which was the root cause of this issue I would say it was connected with the PersistentVolume.</p>
<p>In one of the <a href="https://github.com/kubernetes-retired/external-storage/issues/752" rel="nofollow noreferrer">Github issue - Volume was remounted as read only after error #752</a> I found very similar behavior to OP's behavior.</p>
<p>You have created a <code>special</code> storageclass for your MySQL. You've set <code>reclaimPolicy: Retain</code> so PV was not removed. When <code>Statefulset</code> pod (with the same suffix <code>-0</code>) has been recreated (restarted due to error with connectivity, some issues on DB, hard to say) it tried to re-claim this Volume. In the mentioned Github issue, user had very similar situation. Also got <code>inode #262147: comm mysqld: reading directory lblock</code> issue, but in the bellow there was also entry <code>[ +0.003695] EXT4-fs (sda): Remounting filesystem read-only</code>. Maybe it changed permissions when re-mounted?</p>
<p>Another thing that your <code>volumeClaimTemplates</code> contained</p>
<pre><code>        accessModes:
          - ReadWriteOnce
          - ReadOnlyMany
</code></pre>
<p>So one <code>PersistentVolume</code> could be used as <code>ReadWriteOnce</code> by one node or only <code>ReadOnlyMany</code> by many nodes. There is a possibility that POD was recreated in different node with <code>Read-Only</code> assessMode.</p>
<pre><code>[ +35.912075] EXT4-fs warning (device sda): htree_dirblock_to_tree:977: inode #2: lblock 0: comm mysqld: error -5 reading directory block
[  +6.294232] EXT4-fs error (device sda): ext4_find_entry:1436: inode #262147: comm mysqld: reading directory lblock ...
[  +0.005226] EXT4-fs error (device sda): ext4_find_entry:1436: inode #2: comm mysqld: reading directory lblock 0
[  +1.666039] EXT4-fs error (device sda): ext4_journal_check_start:61: Detected aborted journal
[ +0.003695] EXT4-fs (sda): Remounting filesystem read-only
</code></pre>
<p>It would fit to OP's comment:</p>
<blockquote>
<p>Two days ago for reasons unknown to me Kubernetes restarted the container and was keep trying to run it on rnfa machine. The container was probably evicted from another node.</p>
</blockquote>
<p>Another thing is that node or cluster might be updated (depending if the auto update option was turned on) which might enforce restart of the pod.</p>
<p>Issue with <code>'/var/lib/mysql/': Input/output error</code> might point to database corruption like mentioned <a href="https://dba.stackexchange.com/questions/155251/possibility-of-fixing-corrupt-mysql-database">here</a>.</p>
<p>In general, the issue has been resolved by <code>cordoning</code> affected node. Additional information about the difference between <code>cordon</code> and <code>drain</code> can be found <a href="https://intl.cloud.tencent.com/document/product/457/30654" rel="nofollow noreferrer">here</a>.</p>
<p>Just as an addition, to assign pods to specific node or node with specified label, you can use <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">Affinity</a></p>
