title: Liveness/Readiness set of health indicators for Spring Boot service running on top of Kafka Streams
body: <p>How health indicators should be properly configured for Spring Boot service running on top of Kafka Streams with DB connection? We use Spring Cloud Streams and Kafka Streams binding, Spring-Data JPA, Kubernetes as a container hypervisor. We have let say 3 service replicas and 9 partitions for each topic. A typical service usually joins messages from two topics and persist data in a database and publish data back to another kafka topic.</p>
  <p>After switching to Spring Boot 2.3.1 and changing K8s liveness/readiness endpoints to the new ones:</p>
  <ul>
  <li>/actuator/health/liveness</li>
  <li>/actuator/health/readiness</li>
  </ul>
  <p>we discovered that by default they do not have any health indicators included.</p>
  <p>According to <a href="https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-kubernetes-probes-external-state" rel="nofollow noreferrer">documentation</a>:</p>
  <blockquote>
  <p>Actuator configures the &quot;liveness&quot; and &quot;readiness&quot; probes as Health
  Groups; this means that all the Health Groups features are available
  for them. (...) By default, Spring Boot does not add other Health
  Indicators to these groups.</p>
  </blockquote>
  <p>I believe that this is the right approach, but I have not tested that:</p>
  <pre><code>management.endpoint.health.group.readiness.include: readinessState,db,binders
  management.endpoint.health.group.liveness.include: livenessState,ping,diskSpace
  </code></pre>
  <p>We try to cover the following use cases:</p>
  <ul>
  <li>rolling update: not available consumption slot (idle instance) when new replica is added</li>
  <li>stream has died (runtime exception has been thrown)</li>
  <li>DB is not available during container start up / when service is running</li>
  <li>broker is not available</li>
  </ul>
  <p>I have found a similar <a href="https://stackoverflow.com/questions/54267765/spring-boot-app-kubernetes-liveness-readiness-checks">question</a>, however I believe the current one is specifically related to Kafka services. They are different in it's nature from REST services.</p>
  <p><strong>Update:</strong>
  In spring boot 2.3.1 binders health indicator checks if streams are in RUNNING or REBALANCING state for Kafka 2.5 (before only RUNNING), so I guess that rolling update case with idle instance is handled by its logic.</p>