<p>I am currently working with Apache Pulsar, installed from a helm chart on a local Minikube cluster. The install goes just fine and Apache Pulsar runs well. However, whenever I shutdown/restart my laptop, I can never get the pods all running again. I always get the <code>CrashLoopBackOff</code> status. I try and restart the Pulsar cluster using the following command upon restarting my machine (<code>minikube start</code>):</p>
<pre><code>xyz-MBP:~ xyz$ minikube start
😄  minikube v1.23.2 on Darwin 11.4
🆕  Kubernetes 1.22.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.22.2
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for &quot;minikube&quot; ...
🐳  Preparing Kubernetes v1.19.0 on Docker 20.10.8 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.3.1
    ▪ Using image kubernetesui/metrics-scraper:v1.0.7
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard

❗  /usr/local/bin/kubectl is version 1.22.0, which may have incompatibilites with Kubernetes 1.19.0.
    ▪ Want kubectl v1.19.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use &quot;minikube&quot; cluster and &quot;default&quot; namespace by default
</code></pre>
<p>Now, it looks like it started okay but then when I go to query the status of the pods sometime later, I get the following:</p>
<pre><code>xyz-MBP:pulsar xyz$ kubectl get pods -n pulsar
NAME                                         READY   STATUS             RESTARTS   AGE
pulsar-mini-bookie-0                         0/1     CrashLoopBackOff   8          25h
pulsar-mini-bookie-init-kqx6j                0/1     Completed          0          25h
pulsar-mini-broker-0                         0/1     CrashLoopBackOff   8          25h
pulsar-mini-grafana-555cf54cf-jl5xp          1/1     Running            1          25h
pulsar-mini-prometheus-5556dbb8b8-k5v2v      1/1     Running            1          25h
pulsar-mini-proxy-0                          0/1     Init:1/2           1          25h
pulsar-mini-pulsar-init-h78xk                0/1     Completed          0          25h
pulsar-mini-pulsar-manager-6c6889dff-r6tmk   1/1     Running            1          25h
pulsar-mini-toolset-0                        1/1     Running            1          25h
pulsar-mini-zookeeper-0                      1/1     Running            1          25h
</code></pre>
<p>The mini-proxy never gets out of the init stage, and the bookie and broker keep retrying and instantly going into <code>CrashLoopBackOff</code>. Then, when digging into the logs for the Bookie pod I see the following unfamiliar exception:</p>
<pre><code>01:15:10.164 [main] ERROR org.apache.bookkeeper.bookie.Bookie - Cookie for this bookie is not stored in metadata store. Bookie failing to come up
01:15:10.170 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
</code></pre>
<p>Additionally, I get an exception from the broker pod:</p>
<pre><code>01:21:44.733 [main-EventThread] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to pulsar-mini-bookie-0.pulsar-mini-bookie.pulsar.svc.cluster.local:3181 as endpopint resolution failed
</code></pre>
<p>There is more to the above error but didn't want to dump the entire log here. The above error is the first one that shows up, I believe anything that follows is just fallout from the above... let me know if I'm mistaken about that!</p>

Snoop
<p><strong>Solution:</strong></p>
<ol>
<li>You can check whether your application is exiting due the application crashing which is resulting in the issue. Running the following command should give you a sense of whether that is happening in the container logs:</li>
</ol>
<p><strong>kubectl  logs pod-name --all-containers=true</strong></p>
<p>If you enable the stackdriver logging the following filters can be used to obtain container logs:</p>
<p><strong>Stackdriver V1:</strong></p>
<p>resource.type=&quot;container&quot;</p>
<p>resource.labels.pod_id=&quot;$POD_NAME&quot;</p>
<p><strong>Stackdriver V2:</strong></p>
<p>resource.type=&quot;k8s_container&quot;</p>
<p>resource.labels.pod_name=&quot;$POD_NAME&quot;</p>
<ol start="2">
<li>You can check whether your liveness probes are failing which is resulting in crashes. These would be in pod events. Running the following commands should give you a sense of whether that is happening:</li>
</ol>
<p><strong>kubectl describe pod &quot;$POD_NAME&quot;</strong></p>
<p>If the you have stackdriver logging the following filters can be used to get pod event logs :</p>
<p><strong>Stackdriver V1:</strong></p>
<p>resource.type=&quot;gke_cluster&quot;</p>
<p>logName=&quot;projects/$PROJECT_ID/logs/events&quot;</p>
<p>jsonPayload.reason=&quot;Unhealthy&quot;</p>
<p>jsonPayload.involvedObject.name=&quot;$POD_NAME&quot;</p>
<p><strong>Stackdriver V2:</strong></p>
<p>resource.type=&quot;k8s_pod&quot;</p>
<p>logName=&quot;projects/$PROJECT_ID/logs/events&quot;</p>
<p>jsonPayload.reason=&quot;Unhealthy&quot;</p>
<p>resource.labels.pod_name=&quot;$POD_NAME&quot;</p>
<p><strong>Root cause of this issue:</strong>
The pod simply stuck in a loop of starting and crashing.</p>
