<p>I am running a Java webapp on Tomcat in a Docker image on a Kubernetes cluster. The service runs fine, I am trying to set up the liveness and readiness probes.</p>

<p>I haven't found documentation on best practices wrt Tomcat and Kubernetes specifically, but I considered a HTTP GET request following the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request" rel="nofollow noreferrer">documentation</a> to be appropriate like this:</p>

<pre><code>    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 20
      periodSeconds: 20
      successThreshold: 1
      timeoutSeconds: 3
</code></pre>

<p>When accessing the Tomcat base URL, it returns a welcome page and code <code>200 -- OK</code>. However, the liveness probe fails, this is in the pod description:</p>

<pre><code>Events:
  Type     Reason                 Age               From               Message
  ----     ------                 ----              ----               -------
  Normal   Scheduled              3m                default-scheduler  Successfully assigned xxxxx-service-7f8f76988-lkxdf to kube-03
  Normal   SuccessfulMountVolume  3m                kubelet, kube-03   MountVolume.SetUp succeeded for volume "default-token-b6tps"
  Normal   Created                1m (x3 over 3m)   kubelet, kube-03   Created container
  Normal   Started                1m (x3 over 3m)   kubelet, kube-03   Started container
  Warning  Unhealthy              42s (x7 over 3m)  kubelet, kube-03   Liveness probe failed: Get http://10.233.96.19:8080/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Normal   Pulling                40s (x4 over 3m)  kubelet, kube-03   pulling image "xxxxx:4999/xxxxx/xxxxxservice:v1.3.0"
  Normal   Pulled                 40s (x4 over 3m)  kubelet, kube-03   Successfully pulled image "xxxxx:4999/xxxxx/xxxxxservice:v1.3.0"
  Normal   Killing                40s (x3 over 2m)  kubelet, kube-03   Killing container with id docker://xxxxx-service:Container failed liveness probe.. Container will be killed and recreated.
</code></pre>

<p>The same goes for the readiness probe when set up in the same way.
However, when I deactivate the probes, the service runs fine. I can access the Tomcat welcome page with <code>/</code> as well as the actual webapp.</p>

<p>My question is hence: how should I correctly set up Kubernetes liveness/readiness probes for a Tomcat webapp? Why does the simple <code>HTTP GET</code> approach fail?</p>

<p>Related issues seem to be due to startup times longer than what is set in the <code>initialDelaySeconds</code> parameter (e.g. <a href="https://github.com/kubernetes/kubernetes/issues/51096" rel="nofollow noreferrer">this</a>), just like the error message indicates. However, Tomcat and the webapp really are accessible after a few seconds in this case, so the startup time is not the issue here.</p>

<p>Here are the deployment specs:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: xxxxx-service
  namespace: xxxxx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: xxxxx-service
    spec:
      imagePullSecrets:
      - name: regsecret
      containers:
      - image: xxxxxservice:v1.3.0
        imagePullPolicy: Always
        name: xxxxx-service
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          limits:
            cpu: "0.2"
            memory: 4Gi
</code></pre>

<p>I had the same issue. After a while investigating, I found out that tomcat server in my application took about 35 seconds to start. Thus, I changed initialDelaySeconds to greater than 35 seconds. I also increased time-out seconds to 10 and periods_second to 10 seconds. Kubernetes probe then worked perfectly.</p>
