title: Argo workflow stuck in pending due to liveness probe fail?
body: <p>I am trying to set up a Hyperledger Fabric network on Kubernetes by using <a href="https://github.com/hyfen-nl/PIVT" rel="nofollow noreferrer">this</a>.</p>
  <p>I am at the step where I am trying to create channels. I run the command <code>argo submit output.yaml -v</code> where <code>output.yaml</code> is the output of the command <code>helm template channel-flow/ -f samples/simple/network.yaml -f samples/simple/crypto-config.yaml</code> but with <code>spec.securityContext</code> added as follows:</p>
  <pre><code>...
  spec:
    securityContext:
      runAsNonRoot: true
      #runAsUser: 8737 (I commented out this because I don't know my user ID; not sure if this could cause a problem)
  
    entrypoint: channels
  ...
  </code></pre>
  <p>My argo workflow ends up getting stuck in the pending state. I say this because I check my orderer and peer logs but I see no movement in their logs.</p>
  <p>I referenced <a href="https://stackoverflow.com/questions/61799013/argo-sample-workflows-stuck-in-the-pending-state">Argo sample workflows stuck in the pending state</a> and I start with getting the argo logs:</p>
  <pre><code>[user@vmmock3 fabric-kube]$ kubectl logs -n argo -l app=workflow-controller
  time=&quot;2021-05-31T05:02:41.145Z&quot; level=info msg=&quot;Get leases 200&quot;
  time=&quot;2021-05-31T05:02:41.150Z&quot; level=info msg=&quot;Update leases 200&quot;
  time=&quot;2021-05-31T05:02:46.162Z&quot; level=info msg=&quot;Get leases 200&quot;
  time=&quot;2021-05-31T05:02:46.168Z&quot; level=info msg=&quot;Update leases 200&quot;
  time=&quot;2021-05-31T05:02:51.179Z&quot; level=info msg=&quot;Get leases 200&quot;
  time=&quot;2021-05-31T05:02:51.185Z&quot; level=info msg=&quot;Update leases 200&quot;
  time=&quot;2021-05-31T05:02:56.193Z&quot; level=info msg=&quot;Get leases 200&quot;
  time=&quot;2021-05-31T05:02:56.199Z&quot; level=info msg=&quot;Update leases 200&quot;
  time=&quot;2021-05-31T05:03:01.213Z&quot; level=info msg=&quot;Get leases 200&quot;
  time=&quot;2021-05-31T05:03:01.219Z&quot; level=info msg=&quot;Update leases 200&quot;
  </code></pre>
  <p>I try to describe the workflow controller pod:</p>
  <pre><code>[user@vmmock3 fabric-kube]$ kubectl -n argo describe pod workflow-controller-57fcfb5df8-qvn74
  Name:         workflow-controller-57fcfb5df8-qvn74
  Namespace:    argo
  Priority:     0
  Node:         hlf-pool1-8rnem/10.104.0.8
  Start Time:   Tue, 25 May 2021 13:44:56 +0800
  Labels:       app=workflow-controller
                pod-template-hash=57fcfb5df8
  Annotations:  &lt;none&gt;
  Status:       Running
  IP:           10.244.0.158
  IPs:
    IP:           10.244.0.158
  Controlled By:  ReplicaSet/workflow-controller-57fcfb5df8
  Containers:
    workflow-controller:
      Container ID:  containerd://78c7f8dcb0f3a3b861293559ae0a11b92ce6843065e6f9459556a6b7099c8961
      Image:         argoproj/workflow-controller:v3.0.5
      Image ID:      docker.io/argoproj/workflow-controller@sha256:740dca63b11168490d9cc7b2d1b08c1364f4a4064e1d9b7a778ca2ab12a63158
      Ports:         9090/TCP, 6060/TCP
      Host Ports:    0/TCP, 0/TCP
      Command:
        workflow-controller
      Args:
        --configmap
        workflow-controller-configmap
        --executor-image
        argoproj/argoexec:v3.0.5
        --namespaced
      State:          Running
        Started:      Mon, 31 May 2021 13:08:11 +0800
      Last State:     Terminated
        Reason:       Error
        Exit Code:    2
        Started:      Mon, 31 May 2021 12:59:05 +0800
        Finished:     Mon, 31 May 2021 13:03:04 +0800
      Ready:          True
      Restart Count:  1333
      Liveness:       http-get http://:6060/healthz delay=90s timeout=1s period=60s #success=1 #failure=3
      Environment:
        LEADER_ELECTION_IDENTITY:  workflow-controller-57fcfb5df8-qvn74 (v1:metadata.name)
      Mounts:
        /var/run/secrets/kubernetes.io/serviceaccount from argo-token-hflpb (ro)
  Conditions:
    Type              Status
    Initialized       True
    Ready             True
    ContainersReady   True
    PodScheduled      True
  Volumes:
    argo-token-hflpb:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  argo-token-hflpb
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  kubernetes.io/os=linux
  Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                   node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
    Type     Reason     Age                        From     Message
    ----     ------     ----                       ----     -------
    Warning  Unhealthy  7m44s (x3994 over 5d23h)   kubelet  Liveness probe failed: Get &quot;http://10.244.0.158:6060/healthz&quot;: dial tcp 10.244.0.158:6060: connect: connection refused
    Warning  BackOff    3m46s (x16075 over 5d22h)  kubelet  Back-off restarting failed container
  </code></pre>
  <p>Could this failure be why my argo workflow is stuck in the pending state? How should I go about troubleshooting this?</p>
  <hr />
  <p>EDIT: Output of <code>kubectl get pods --all-namespaces</code> (FYI these are being run on Digital Ocean):</p>
  <pre><code>[user@vmmock3 fabric-kube]$ kubectl get pods --all-namespaces
  NAMESPACE     NAME                                    READY   STATUS             RESTARTS   AGE
  argo          argo-server-5695555c55-867bx            1/1     Running            1          6d19h
  argo          minio-58977b4b48-r2m2h                  1/1     Running            0          6d19h
  argo          postgres-6b5c55f477-7swpp               1/1     Running            0          6d19h
  argo          workflow-controller-57fcfb5df8-qvn74    0/1     CrashLoopBackOff   1522       6d19h
  default       hlf-ca--atlantis-58bbd79d9d-x4mz4       1/1     Running            0          21h
  default       hlf-ca--karga-547dbfddc8-7w6b5          1/1     Running            0          21h
  default       hlf-ca--nevergreen-7ffb98484c-nlg4j     1/1     Running            0          21h
  default       hlf-orderer--groeifabriek--orderer0-0   1/1     Running            0          21h
  default       hlf-peer--atlantis--peer0-0             2/2     Running            0          21h
  default       hlf-peer--karga--peer0-0                2/2     Running            0          21h
  default       hlf-peer--nevergreen--peer0-0           2/2     Running            0          21h
  kube-system   cilium-2kjfz                            1/1     Running            3          26d
  kube-system   cilium-operator-84bdd6f7b6-kp9vb        1/1     Running            1          6d20h
  kube-system   cilium-operator-84bdd6f7b6-pkkf9        1/1     Running            1          6d20h
  kube-system   coredns-55ff57f948-jb5jc                1/1     Running            0          6d20h
  kube-system   coredns-55ff57f948-r2q4g                1/1     Running            0          6d20h
  kube-system   csi-do-node-4r9gj                       2/2     Running            0          26d
  kube-system   do-node-agent-sbc8b                     1/1     Running            0          26d
  kube-system   kube-proxy-hpsc7                        1/1     Running            0          26d
  </code></pre>