<p>My team has a deployment process that involves using Spring's <code>bootBuildImage</code> to create the application image, then we deploy it to our Kubernetes cluster. Last week the deployment was working correctly but it suddenly stopped working.</p>
<p>When we try to deploy a new version, the image is created correctly but the pod doesn't get <code>Ready</code> because the actuator endpoints return a 404 error.</p>
<pre class="lang-bash prettyprint-override"><code>curl localhost:8080/actuator/health/liveness
</code></pre>
<pre class="lang-html prettyprint-override"><code>&lt;!doctype html&gt;
&lt;html lang=&quot;en&quot;&gt;
    &lt;head&gt;
        &lt;title&gt;HTTP Status 404 – Not Found&lt;/title&gt;
        &lt;style type=&quot;text/css&quot;&gt;body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}&lt;/style&gt;
    &lt;/head&gt;
    &lt;body&gt;
        &lt;h1&gt;HTTP Status 404 – Not Found&lt;/h1&gt;
    &lt;/body&gt;
&lt;/html&gt;
</code></pre>
<p>I already added the health properties (even though we hadn't needed them before)</p>
<pre><code>management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true
</code></pre>
<p>This is what Kubernetes says</p>
<pre><code>Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 07 Feb 2023 01:23:27 -0600
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/actuator/health/liveness delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:8080/actuator/health/readiness delay=60s timeout=1s period=10s #success=1 #failure=3
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m30s              default-scheduler  Successfully assigned podname to ip-10-8-x-x.ap-northeast-1.compute.internal
  Normal   Pulled     2m29s              kubelet            Container image &quot;registry.gitlab.com/integration-image&quot; already present on machine
  Normal   Created    2m29s              kubelet            Created container ui-backend
  Normal   Started    2m29s              kubelet            Started container ui-backend
  Warning  Unhealthy  30s (x2 over 60s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 404
  Warning  Unhealthy  10s (x8 over 80s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 404
</code></pre>
<p>I've also tried:</p>
<ul>
<li>extending the checks time</li>
<li>changing the port of the actuator</li>
<li>disabling spring tasks</li>
</ul>
<p>In all of these cases spring starts but gets restarted by kubernetes after failing the readiness checks</p>
<pre><code>Setting Active Processor Count to 2
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
Calculating JVM memory based on 5201M available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx4528339K -XX:MaxMetaspaceSize=285484K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 5201M, Thread Count: 250, Loaded Class Count: 47989, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 124 container CA certificates to JVM truststore
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=2 -Dspring.profiles.active=dev -Dspring.config.additional-location=optional:file:/config/,optional:file:/secret/ -Dsentry.environment=dev -Dhoneycomb.config.file=/secret/application.properties -javaagent:/workspace/WEB-INF/lib/honeycomb-opentelemetry-javaagent-1.0.0.jar -XX:MaxDirectMemorySize=10M -Xmx4528339K -XX:MaxMetaspaceSize=285484K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics
[otel.javaagent 2023-02-02 03:27:59:129 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: honeycomb-1.0.0-otel-1.12.0
[CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Initializing ProtocolHandler [&quot;http-nio-8080&quot;]
[CONTAINER] org.apache.catalina.startup.Catalina               INFO    Server initialization in [968] milliseconds
[CONTAINER] org.apache.catalina.core.StandardService           INFO    Starting service [Catalina]
[CONTAINER] org.apache.catalina.core.StandardEngine            INFO    Starting Servlet engine: [Apache Tomcat/9.0.71]
[CONTAINER] org.apache.catalina.startup.HostConfig             INFO    Deploying web application directory [/layers/paketo-buildpacks_apache-tomcat/catalina-base/webapps/ROOT]
[CONTAINER] org.apache.jasper.servlet.TldScanner               INFO    At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
[CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] INFO    2 Spring WebApplicationInitializers detected on classpath
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:15.334Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app&quot;,&quot;message&quot;:&quot;Starting UiApplication using Java 11.0.18 on ui-backend-7b6ddf8f58-m8vtq with PID 1 (/workspace/WEB-INF/classes started by cnb in /workspace)&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377774090677176}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:15.388Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app&quot;,&quot;message&quot;:&quot;Running with Spring Boot v2.5.8, Spring v5.3.14&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377774144285955}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:15.388Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app&quot;,&quot;message&quot;:&quot;The following profiles are active: dev&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377774145291720}
[CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] INFO    Initializing Spring embedded WebApplicationContext
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:22.755Z&quot;,&quot;level&quot;:&quot;WARN&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.boot.autoconfigure.orm.jpa.JpaBaseConfiguration$JpaWebConfiguration&quot;,&quot;message&quot;:&quot;spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377781511405545}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:38.660Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.report.ReportTemplateRegistry&quot;,&quot;message&quot;:&quot;report template registry -&gt; templates -&gt; {receipt_ja=app.route.ui.report.ReceiptReportTemplate@6dc0e992, invoice_ja=app.route.ui.report.InvoiceReportTemplate@6e221246}&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377797416315839}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:38.675Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.service.ReportService&quot;,&quot;message&quot;:&quot;report service -&gt; initialize -&gt; start&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377797431868400}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:38.721Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.service.ReportService&quot;,&quot;message&quot;:&quot;report service -&gt; initialize -&gt; cache created succefully&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377797477131943}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:38.721Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.service.ReportService&quot;,&quot;message&quot;:&quot;report service -&gt; initialize -&gt; end&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377797481925043}
[CONTAINER] freemarker.configuration                           SEVERE  DefaultObjectWrapper.incompatibleImprovements was set to the object returned by Configuration.getVersion(). That defeats the purpose of incompatibleImprovements, and makes upgrading FreeMarker a potentially breaking change. Also, this probably won't be allowed starting from 2.4.0. Instead, set incompatibleImprovements to the highest concrete version that's known to be compatible with your application.
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:40.289Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.entity.storage.EntityStorageRegistry&quot;,&quot;message&quot;:&quot;entity storage registry -&gt; interceptors -&gt; [app.route.ui.entity.interceptor.UserInterceptor@3a94d628]&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377799045261311}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:41.091Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.security.web.DefaultSecurityFilterChain&quot;,&quot;message&quot;:&quot;Will secure Ant [pattern='/actuator/health/**'] with []&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377799847478125}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:41.092Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.security.web.DefaultSecurityFilterChain&quot;,&quot;message&quot;:&quot;Will secure Ant [pattern='/h2-console/**'] with []&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377799848958832}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:41.182Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.security.web.DefaultSecurityFilterChain&quot;,&quot;message&quot;:&quot;Will secure Or [Ant [pattern='/admin/user/webhook'], Ant [pattern='/v1/backups/webhoook/**']] with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@7843b634, org.springframework.security.web.context.SecurityContextPersistenceFilter@2f514909, org.springframework.security.web.header.HeaderWriterFilter@22c59a39, org.springframework.security.web.authentication.logout.LogoutFilter@395a9cdb, app.route.ui.security.internal.ApiKeyAuthFilter@3d042a50, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@1404f97, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@50d8cc71, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@63b040d5, org.springframework.security.web.session.SessionManagementFilter@75c3b5ab, org.springframework.security.web.access.ExceptionTranslationFilter@763630ac, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@7753b2e8]&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377799938730993}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:41.194Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.security.web.DefaultSecurityFilterChain&quot;,&quot;message&quot;:&quot;Will secure any request with [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1c2cee82, org.springframework.security.web.context.SecurityContextPersistenceFilter@3c3e87b8, org.springframework.security.web.header.HeaderWriterFilter@58e130bc, org.springframework.web.filter.CorsFilter@7c067793, org.keycloak.adapters.springsecurity.filter.KeycloakPreAuthActionsFilter@43c49d4c, org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticationProcessingFilter@233b20ed, org.springframework.security.web.authentication.logout.LogoutFilter@4eb259f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@465b750a, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@176ed9ad, org.keycloak.adapters.springsecurity.filter.KeycloakSecurityContextRequestFilter@1de33861, org.keycloak.adapters.springsecurity.filter.KeycloakAuthenticatedActionsFilter@41dfea3, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@25cb293e, org.springframework.security.web.session.SessionManagementFilter@76a5b602, org.springframework.security.web.access.ExceptionTranslationFilter@160d925f, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@3a5e6480]&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377799956811708}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:42.964Z&quot;,&quot;level&quot;:&quot;INFO&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app&quot;,&quot;message&quot;:&quot;Started UiApplication in 29.288 seconds (JVM running for 44.237)&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377801720328782}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:43.156Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.interceptor.RequestLogFilter&quot;,&quot;message&quot;:&quot;Filter 'requestLogFilter' configured for use&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377801912945502}
{&quot;timestamp&quot;:&quot;2023-02-02T03:28:43.160Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;app.route.ui.security.UiAuthenticationFilter&quot;,&quot;message&quot;:&quot;Filter 'uiAuthenticationFilter' configured for use&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:27377801916847463}
[CONTAINER] org.apache.catalina.startup.HostConfig             INFO    Deployment of web application directory [/layers/paketo-buildpacks_apache-tomcat/catalina-base/webapps/ROOT] has finished in [38,936] ms
[CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Starting ProtocolHandler [&quot;http-nio-8080&quot;]
[CONTAINER] org.apache.catalina.startup.Catalina               INFO    Server startup in [39138] milliseconds
[CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Pausing ProtocolHandler [&quot;http-nio-8080&quot;]
[CONTAINER] org.apache.catalina.core.StandardService           INFO    Stopping service [Catalina]
[CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] INFO    Closing Spring root WebApplicationContext
</code></pre>
<p>I'm not sure if this is a spring or a Kubernetes problem. Thank you.</p>
<p>Edit:
Other things that I've tried are:</p>
<ul>
<li>draining the nodes</li>
<li>running the images with <code>kubectl run</code> which starts spring, and I can reach the actuator endpoints but can't do anything else.</li>
</ul>
<p>Nodes description.</p>
<ul>
<li>Node where at least the previous deployment still works</li>
</ul>
<pre><code>Name:               goodnode.ap-northeast-1.compute.internal
Roles:              &lt;none&gt;
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5.large
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=ap-northeast-1
                    failure-domain.beta.kubernetes.io/zone=ap-northeast-1a
                    k8s.io/cloud-provider-aws=86af69e954aa5c8fc905ce14364ec305
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=goodnode.ap-northeast-1.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5.large
                    topology.kubernetes.io/region=ap-northeast-1
                    topology.kubernetes.io/zone=ap-northeast-1a
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 22 Mar 2022 00:32:53 -0600
Taints:             &lt;none&gt;
Unschedulable:      false
Lease:
  HolderIdentity:  goodnode.ap-northeast-1.compute.internal
  AcquireTime:     &lt;unset&gt;
  RenewTime:       Tue, 07 Feb 2023 13:13:59 -0600
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 07 Feb 2023 13:11:07 -0600   Tue, 22 Mar 2022 00:32:51 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 07 Feb 2023 13:11:07 -0600   Tue, 22 Mar 2022 00:32:51 -0600   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 07 Feb 2023 13:11:07 -0600   Tue, 22 Mar 2022 00:32:51 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 07 Feb 2023 13:11:07 -0600   Tue, 22 Mar 2022 00:33:23 -0600   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   goodnodeip
  Hostname:     goodnode.ap-northeast-1.compute.internal
  InternalDNS:  goodnode.ap-northeast-1.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         2
  ephemeral-storage:           52416492Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7934440Ki
  pods:                        29
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         1930m
  ephemeral-storage:           47233297124
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7244264Ki
  pods:                        29
System Info:
  Machine ID:                 ec26c540fc98771f
  System UUID:                ec26c540-fc98-77
  Boot ID:                    034cb48b-8129-40
  Kernel Version:             5.4.181-99.354.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.21.5-eks-9017834
  Kube-Proxy Version:         v1.21.5-eks-9017834
ProviderID:                   aws:///ap-northeast-1a/i-01a03da4ffb56c6f0
Non-terminated Pods:          (11 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           cloudwatch-agent-b2cxj                                             200m (10%)    200m (10%)  200Mi (2%)       200Mi (2%)     322d
  amazon-cloudwatch           fluent-bit-f825k                                                   500m (25%)    0 (0%)      100Mi (1%)       200Mi (2%)     322d
  webapp                application A                            500m (25%)    1 (51%)     2G (26%)         4G (53%)       12h
  webapp                application B                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  webapp                APP_WITH_PROBLEMS                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         11h
  kube-system                 alb-ingress-controller-aws-load-balancer-controller-76df65d6qrx    0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  kube-system                 alb-ingress-controller-aws-load-balancer-controller-76df65kwmss    0 (0%)        0 (0%)      0 (0%)           0 (0%)         12h
  kube-system                 aws-node-g7bp7                                                     10m (0%)      0 (0%)      0 (0%)           0 (0%)         322d
  kube-system                 coredns-54bc78bc49-6bdv6                                           100m (5%)     0 (0%)      70Mi (0%)        170Mi (2%)     12h
  kube-system                 coredns-54bc78bc49-vh2rv                                           100m (5%)     0 (0%)      70Mi (0%)        170Mi (2%)     12h
  kube-system                 kube-proxy-lt9bq                                                   100m (5%)     0 (0%)      0 (0%)           0 (0%)         322d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests         Limits
  --------                    --------         ------
  cpu                         1510m (78%)      1200m (62%)
  memory                      2403685Ki (33%)  4664010Ki (64%)
  ephemeral-storage           0 (0%)           0 (0%)
  hugepages-1Gi               0 (0%)           0 (0%)
  hugepages-2Mi               0 (0%)           0 (0%)
  attachable-volumes-aws-ebs  0                0
Events:                       &lt;none&gt;
</code></pre>
<ul>
<li>Node where neither the previous deployment nor the new ones work.</li>
</ul>
<pre><code>Name:               badnode.ap-northeast-1.compute.internal
Roles:              &lt;none&gt;
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=m5.large
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=ap-northeast-1
                    failure-domain.beta.kubernetes.io/zone=ap-northeast-1c
                    k8s.io/cloud-provider-aws=86af69e954aa5c8fc905ce14364ec305
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=badnode.ap-northeast-1.compute.internal
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=m5.large
                    topology.kubernetes.io/region=ap-northeast-1
                    topology.kubernetes.io/zone=ap-northeast-1c
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 22 Mar 2022 00:25:27 -0600
Taints:             &lt;none&gt;
Unschedulable:      false
Lease:
  HolderIdentity:  badnode.ap-northeast-1.compute.internal
  AcquireTime:     &lt;unset&gt;
  RenewTime:       Tue, 07 Feb 2023 13:14:01 -0600
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 07 Feb 2023 13:12:59 -0600   Tue, 22 Mar 2022 00:25:25 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 07 Feb 2023 13:12:59 -0600   Tue, 22 Mar 2022 00:25:25 -0600   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Tue, 07 Feb 2023 13:12:59 -0600   Tue, 22 Mar 2022 00:25:25 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Tue, 07 Feb 2023 13:12:59 -0600   Tue, 22 Mar 2022 00:25:57 -0600   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   badnodeip
  Hostname:     badnode.ap-northeast-1.compute.internal
  InternalDNS:  badnode.ap-northeast-1.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         2
  ephemeral-storage:           52416492Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7934440Ki
  pods:                        29
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         1930m
  ephemeral-storage:           47233297124
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7244264Ki
  pods:                        29
System Info:
  Machine ID:                 ec28188afbf50b034
  System UUID:                ec28188a-fbf5-0b0
  Boot ID:                    316e2549-9735-4fe4
  Kernel Version:             5.4.181-99.354.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.21.5-eks-9017834
  Kube-Proxy Version:         v1.21.5-eks-9017834
ProviderID:                   aws:///ap-northeast-1c/i-056ee5ae8f99b7076
Non-terminated Pods:          (4 in total)
  Namespace                   Name                      CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                      ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           cloudwatch-agent-v7kk2    200m (10%)    200m (10%)  200Mi (2%)       200Mi (2%)     322d
  amazon-cloudwatch           fluent-bit-mfhxk          500m (25%)    0 (0%)      100Mi (1%)       200Mi (2%)     322d
  kube-system                 aws-node-qsgkm            10m (0%)      0 (0%)      0 (0%)           0 (0%)         322d
  kube-system                 kube-proxy-nlgxk          100m (5%)     0 (0%)      0 (0%)           0 (0%)         322d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests    Limits
  --------                    --------    ------
  cpu                         810m (41%)  200m (10%)
  memory                      300Mi (4%)  400Mi (5%)
  ephemeral-storage           0 (0%)      0 (0%)
  hugepages-1Gi               0 (0%)      0 (0%)
  hugepages-2Mi               0 (0%)      0 (0%)
  attachable-volumes-aws-ebs  0           0
Events:                       &lt;none&gt;
</code></pre>
<pre class="lang-json prettyprint-override"><code>resource &quot;kubernetes_deployment&quot; &quot;ui_backend&quot; {
  metadata {
    name      = local.name
    namespace = local.namespace
    labels = {
      &quot;app.kubernetes.io/name&quot;       = local.name
      &quot;app.kubernetes.io/managed-by&quot; = local.managed_by
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        &quot;app.kubernetes.io/name&quot;       = local.name
        &quot;app.kubernetes.io/managed-by&quot; = local.managed_by
      }
    }

    template {
      metadata {
        labels = {
          &quot;app.kubernetes.io/name&quot;       = local.name
          &quot;app.kubernetes.io/managed-by&quot; = local.managed_by
        }
      }

      spec {
        service_account_name = local.name

        image_pull_secrets {
          name = &quot;gitlab-auth&quot;
        }

        container {
          image = var.docker_image
          name  = local.name
          env {
            name  = &quot;JAVA_OPTS&quot;
            value = &quot;-Dspring.profiles.active=${var.environment} &quot;
          }

          port {
            container_port = local.application_port
          }

          liveness_probe {
            http_get {
              path = &quot;/actuator/health/liveness&quot;
              port = local.application_port
            }

            initial_delay_seconds = 60
            period_seconds        = 30
          }

          readiness_probe {
            http_get {
              path = &quot;/actuator/health/readiness&quot;
              port = local.application_port
            }

            initial_delay_seconds = 60
            period_seconds        = 10
          }

        }
      }
    }
  }
}
</code></pre>
<p>logs with <code>logging.level.root=DEBUG</code></p>
<pre><code>{&quot;timestamp&quot;:&quot;2023-02-08T21:28:40.448Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.boot.availability.ApplicationAvailabilityBean&quot;,&quot;message&quot;:&quot;Application availability state LivenessState changed to CORRECT&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:142674280192537}
{&quot;timestamp&quot;:&quot;2023-02-08T21:28:40.451Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.boot.devtools.restart.Restarter&quot;,&quot;message&quot;:&quot;Creating new Restarter for thread Thread[main,5,main]&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:142674282973006}
{&quot;timestamp&quot;:&quot;2023-02-08T21:28:40.454Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.boot.availability.ApplicationAvailabilityBean&quot;,&quot;message&quot;:&quot;Application availability state ReadinessState changed to ACCEPTING_TRAFFIC&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:142674286139388}
{&quot;timestamp&quot;:&quot;2023-02-08T21:28:40.496Z&quot;,&quot;level&quot;:&quot;DEBUG&quot;,&quot;thread&quot;:&quot;main&quot;,&quot;logger&quot;:&quot;org.springframework.boot.web.servlet.filter.OrderedRequestContextFilter&quot;,&quot;message&quot;:&quot;Filter 'requestContextFilter' configured for use&quot;,&quot;context&quot;:&quot;default&quot;,&quot;nanotime&quot;:142674328227052}
</code></pre>

Ricardo Rubik Ruiz
<p>remove <code>-javaagent:/workspace/WEB-INF/lib/honeycomb-opentelemetry-javaagent-1.0.0.jar</code></p>
<p>edit:
Looks like adding <code>-Dotel.instrumentation.tomcat.enabled=false</code> will also fix the issue.</p>
