<p>I have an application deployed to Kubernetes that depends on an outside application. Sometimes the connection between these 2 goes to an invalid state, and that can only be fixed by restarting my application.</p>
<p>To do automatic restarts, I have configured a liveness probe that will verify the connection.</p>
<p>This has been working great, however, I'm afraid that if that outside application goes down (such that the connection error isn't just due to an invalid pod state), all of my pods will immediately restart, and my application will become completely unavailable. I want it to remain running so that functionality not depending on the bad service can continue.</p>
<p>I'm wondering if a pod disruption budget would prevent this scenario, as it limits the # of pods down due to a &quot;voluntary&quot; disruption. However, the K8s docs don't state whether liveness probe failure are a voluntary disruption. Are they?</p>

roim
<p>I would say, accordingly to the documentation:</p>
<blockquote>
<h3>Voluntary and involuntary disruptions</h3>
<p>Pods do not disappear until someone (a person or a controller) destroys them, or there is an unavoidable hardware or system software error.</p>
<p>We call these unavoidable cases  <em>involuntary disruptions</em>  to an application. Examples are:</p>
<ul>
<li>a hardware failure of the physical machine backing the node</li>
<li>cluster administrator deletes VM (instance) by mistake</li>
<li>cloud provider or hypervisor failure makes VM disappear</li>
<li>a kernel panic</li>
<li>the node disappears from the cluster due to cluster network partition</li>
<li>eviction of a pod due to the node being  <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">out-of-resources</a>.</li>
</ul>
<p>Except for the out-of-resources condition, all these conditions should be familiar to most users; they are not specific to Kubernetes.</p>
<p>We call other cases <em>voluntary disruptions</em>. These include both actions initiated by the application owner and those initiated by a Cluster Administrator. Typical application owner actions include:</p>
<ul>
<li>deleting the deployment or other controller that manages the pod</li>
<li>updating a deployment's pod template causing a restart</li>
<li>directly deleting a pod (e.g. by accident)</li>
</ul>
<p>Cluster administrator actions include:</p>
<ul>
<li><a href="https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/" rel="nofollow noreferrer">Draining a node</a>  for repair or upgrade.</li>
<li>Draining a node from a cluster to scale the cluster down (learn about  <a href="https://github.com/kubernetes/autoscaler/#readme" rel="nofollow noreferrer">Cluster Autoscaling</a>  ).</li>
<li>Removing a pod from a node to permit something else to fit on that node.</li>
</ul>
<p>-- <em><a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/" rel="nofollow noreferrer">Kubernetes.io: Docs: Concepts: Workloads: Pods: Disruptions</a></em></p>
</blockquote>
<p><strong>So your example is quite different and according to my knowledge it's neither voluntary or involuntary disruption.</strong></p>
<hr />
<p>Also taking a look on another Kubernetes documentation:</p>
<blockquote>
<h3>Pod lifetime</h3>
<p>Like individual application containers, Pods are considered to be relatively ephemeral (rather than durable) entities. Pods are created, assigned a unique ID (<a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids" rel="nofollow noreferrer">UID</a>), and scheduled to nodes where they remain until termination (according to restart policy) or deletion. If a <a href="https://kubernetes.io/docs/concepts/architecture/nodes/" rel="nofollow noreferrer">Node</a> dies, the Pods scheduled to that node are <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection" rel="nofollow noreferrer">scheduled for deletion</a> after a timeout period.</p>
<p>Pods do not, by themselves, self-heal. If a Pod is scheduled to a <a href="https://kubernetes.io/docs/concepts/architecture/nodes/" rel="nofollow noreferrer">node</a> that then fails, the Pod is deleted; likewise, a Pod won't survive an eviction due to a lack of resources or Node maintenance. Kubernetes uses a higher-level abstraction, called a <a href="https://kubernetes.io/docs/concepts/architecture/controller/" rel="nofollow noreferrer">controller</a>, that handles the work of managing the relatively disposable Pod instances.</p>
<p>-- <em><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-lifetime" rel="nofollow noreferrer">Kubernetes.io: Docs: Concepts: Workloads: Pods: Pod lifecycle: Pod lifetime</a></em></p>
</blockquote>
<blockquote>
<h3>Container probes</h3>
<p>The kubelet can optionally perform and react to three kinds of probes on running containers (focusing on a <code>livenessProbe</code>):</p>
<ul>
<li><code>livenessProbe</code>: Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its  <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy" rel="nofollow noreferrer">restart policy</a>. If a Container does not provide a liveness probe, the default state is  <code>Success</code>.</li>
</ul>
<p>-- <em><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes" rel="nofollow noreferrer">Kubernetes.io: Docs: Concepts: Workloads: Pods: Pod lifecycle: Container probes</a></em></p>
<h3>When should you use a liveness probe?</h3>
<p>If the process in your container is able to crash on its own whenever it encounters an issue or becomes unhealthy, you do not necessarily need a liveness probe; the kubelet will automatically perform the correct action in accordance with the Pod's  <code>restartPolicy</code>.</p>
<p>If you'd like your container to be killed and restarted if a probe fails, then specify a liveness probe, and specify a  <code>restartPolicy</code>  of Always or OnFailure.</p>
<p>-- <em><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#when-should-you-use-a-startup-probe" rel="nofollow noreferrer">Kubernetes.io: Docs: Concepts: Workloads: Pods: Pod lifecycle: When should you use a startup probe</a></em></p>
</blockquote>
<p>According to those information it would be better to create custom liveness probe which should consider internal process health checks and external dependency(liveness) health check. In the first scenario your container should stop/terminate your process unlike the the second case with external dependency.</p>
<p>Answering following question:</p>
<blockquote>
<p>I'm wondering if a pod disruption budget would prevent this scenario.</p>
</blockquote>
<p><strong>In this particular scenario PDB will not help.</strong></p>
<hr />
<p>I'd reckon giving more visibility to the comment, I've made with additional resources on the matter could prove useful to other community members:</p>
<ul>
<li><em><a href="https://blog.risingstack.com/designing-microservices-architecture-for-failure/" rel="nofollow noreferrer">Blog.risingstack.com: Designing microservices architecture for failure</a></em></li>
<li><em><a href="https://loft.sh/blog/kubernetes-readiness-probes-examples-common-pitfalls/#external-dependencies" rel="nofollow noreferrer">Loft.sh: Blog: Kubernetes readiness probles examples common pitfalls: External depenedencies</a></em></li>
<li><em><a href="https://cloud.google.com/architecture/scalable-and-resilient-apps#resilience_designing_to_withstand_failures" rel="nofollow noreferrer">Cloud.google.com: Archiecture: Scalable and resilient apps: Resilience designing to withstand failures</a></em></li>
</ul>
