<p>I have implemented a gRPC service, build it into a container, and deployed it using k8s, in particular AWS EKS, as a DaemonSet.</p>
<p>The Pod starts and turns to be in Running status very soon, but it takes very long, typically 300s, for the actual service to be accessible.</p>
<p>In fact, when I run <code>kubectl logs</code> to print the log of the Pod, it is empty for a long time.</p>
<p>I have logged something at the very starting of the service. In fact, my code looks like</p>
<pre class="lang-golang prettyprint-override"><code>package main

func init() {
    log.Println(&quot;init&quot;)
}

func main() {
  // ...
}
</code></pre>
<p>So I am pretty sure when there are no logs, the service is not started yet.</p>
<p>I understand that there may be a time gap between the Pod is running and the actual process inside it is running. However, 300s looks too long for me.</p>
<p>Furthermore, this happens randomly, sometimes the service is ready almost immediately. By the way, my runtime image is based on <a href="https://hub.docker.com/r/chromedp/headless-shell/" rel="nofollow noreferrer">chromedp headless-shell</a>, not sure if it is relevant.</p>
<p>Could anyone provide some advice for how to debug and locate the problem? Many thanks!</p>
<hr />
<p>Update</p>
<p>I did not set any readiness probes.</p>
<p>Running <code>kubectl get -o yaml</code> of my DaemonSet gives</p>
<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: &quot;1&quot;
  creationTimestamp: &quot;2021-10-13T06:30:16Z&quot;
  generation: 1
  labels:
    app: worker
    uuid: worker
  name: worker
  namespace: collection-14f45957-e268-4719-88c3-50b533b0ae66
  resourceVersion: &quot;47265945&quot;
  uid: 88e4671f-9e33-43ef-9c49-b491dcb578e4
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: worker
      uuid: worker
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: &quot;2112&quot;
        prometheus.io/scrape: &quot;true&quot;
      creationTimestamp: null
      labels:
        app: worker
        uuid: worker
    spec:
      containers:
      - env:
        - name: GRPC_PORT
          value: &quot;22345&quot;
        - name: DEBUG
          value: &quot;false&quot;
        - name: TARGET
          value: localhost:12345
        - name: TRACKER
          value: 10.100.255.31:12345
        - name: MONITOR
          value: 10.100.125.35:12345
        - name: COLLECTABLE_METHODS
          value: shopping.ShoppingService.GetShop
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: DISTRIBUTABLE_METHODS
          value: collection.CollectionService.EnumerateShops
        - name: PERFORM_TASK_INTERVAL
          value: 0.000000s
        image: xxx
        imagePullPolicy: Always
        name: worker
        ports:
        - containerPort: 22345
          protocol: TCP
        resources:
          requests:
            cpu: 1800m
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - env:
        - name: CAPTCHA_PARALLEL
          value: &quot;32&quot;
        - name: HTTP_PROXY
          value: http://10.100.215.25:8080
        - name: HTTPS_PROXY
          value: http://10.100.215.25:8080
        - name: API
          value: 10.100.111.11:12345
        - name: NO_PROXY
          value: 10.100.111.11:12345
        - name: POD_IP
        image: xxx
        imagePullPolicy: Always
        name: source
        ports:
        - containerPort: 12345
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/ssl/certs/api.crt
          name: ca
          readOnly: true
          subPath: tls.crt
      dnsPolicy: ClusterFirst
      nodeSelector:
        api/nodegroup-app: worker
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: ca
        secret:
          defaultMode: 420
          secretName: ca
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 2
  desiredNumberScheduled: 2
  numberAvailable: 2
  numberMisscheduled: 0
  numberReady: 2
  observedGeneration: 1
  updatedNumberScheduled: 2
</code></pre>
<p>Furthermore, there are two containers in the Pod. Only one of them is exceptionally slow to start, and the other one is always fine.</p>

<p>When you use HTTP_PROXY for your solution, watchout how it may route differently from your underlying cluster network - which often result to unexpected timeout.</p>
