title: Kubernetes liveness probe logging recovery
body: <p>I am trying to test a liveness probe while learning kubernetes.
  I have set up a minikube and configured a pod with a liveness probe.</p>
  
  <p>Testing the script (E.g. via docker exec) it seems to report success and failure as required.</p>
  
  <p>The probe leads to failures events which I can view via <em>kubectl describe podname</em>
  but it does not report recovery from failures.</p>
  
  <p><a href="https://stackoverflow.com/a/34599554/1569204">This answer</a> says that liveness probe successes are not reported by default.</p>
  
  <p>I have been trying to increase the log level with no success by running variations like:</p>
  
  <pre><code>minikube start --extra-config=apiserver.v=4
  minikube start --extra-config=kubectl.v=4
  minikube start --v=4
  </code></pre>
  
  <p>As suggested <a href="https://stackoverflow.com/questions/55929593/run-kubernetes-api-server-in-minikube-in-verbose-mode">here</a> &amp; <a href="https://minikube.sigs.k8s.io/docs/reference/configuration/kubernetes/" rel="nofollow noreferrer">here</a>.</p>
  
  <p><strong>What is the proper way to configure the logging level for a kubelet?</strong></p>
  
  <p>can it be modified without restarting the pod or minikube?</p>
  
  <p>An event will be reported if a failure causes the pod to be restarted.
  For kubernetes itself I understand that using it to decide whether to restart the pod is sufficient.</p>
  
  <p>Why aren't events recorded for recovery from a failure which does not require a restart?
  This is how I would expect probes to work in a health monitoring system.</p>
  
  <p>How would recovery be visible if the same probe was used in prometheus or similar?
  For an expensive probe I would not want it to be run multiple times.
  (granted one probe could cache the output to a file making the second probe cheaper)</p>