<p>I have a node.js service that I want to build with Jenkins in Kubernetes, using a Jenkins agent pod specified by the node.js project. I am trying to eliminate manual touch to Jenkins UI. Everything is running in one kubernetes cluster.</p>
<p>I am following this <a href="https://www.jenkins.io/blog/2018/09/14/kubernetes-and-secret-agents/" rel="nofollow noreferrer">blog</a> and adapting it slightly, but running into problems:</p>
<ol>
<li>I get an error <code>‘Jenkins’ doesn’t have label ‘test-pod’</code></li>
<li>The job loops infinitely.</li>
</ol>
<p>The build agent is successfully created in Kubernetes. The <code>test-pod</code> label is specified by the <code>Jenkinsfile</code> so I don't know why I get this error. And how is it infinitely looping?</p>
<pre><code>podTemplate(
    name: 'test-pod',
    label: 'test-pod',
    containers: [
        containerTemplate(name: 'node14', image: 'node:14-alpine'),
        containerTemplate(name: 'docker', image:'trion/jenkins-docker-client'),
    ],
    {
        node('test-pod') {
            stage('Build'){
                container('node14') {
                    // do nothing just yet
                }
            }
        }
    }
)
</code></pre>
<p>Here is part of the Jenkins console output:</p>
<pre><code>Started by user admin
Obtained Jenkinsfile from git ssh://git@kube-master.cluster.dev/git/hello.git
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes jenkins/test-pod-2hdfp-9kcjj
[Normal][jenkins/test-pod-2hdfp-9kcjj][Scheduled] Successfully assigned jenkins/test-pod-2hdfp-9kcjj to kube-worker2.cluster.dev
[Normal][jenkins/test-pod-2hdfp-9kcjj][Pulled] Container image &quot;node:14-alpine&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-9kcjj][Created] Created container node14
[Normal][jenkins/test-pod-2hdfp-9kcjj][Started] Started container node14
[Normal][jenkins/test-pod-2hdfp-9kcjj][Pulled] Container image &quot;trion/jenkins-docker-client&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-9kcjj][Created] Created container docker
[Normal][jenkins/test-pod-2hdfp-9kcjj][Started] Started container docker
[Normal][jenkins/test-pod-2hdfp-9kcjj][Pulled] Container image &quot;jenkins/inbound-agent:4.3-4&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-9kcjj][Created] Created container jnlp
[Normal][jenkins/test-pod-2hdfp-9kcjj][Started] Started container jnlp
jenkins/test-pod-2hdfp-9kcjj Container node14 was terminated (Exit Code: 0, Reason: Completed)
[Normal][jenkins/test-pod-2hdfp-9kcjj][Killing] Stopping container docker
Created Pod: kubernetes jenkins/test-pod-2hdfp-gc2qb
[Normal][jenkins/test-pod-2hdfp-gc2qb][Scheduled] Successfully assigned jenkins/test-pod-2hdfp-gc2qb to kube-worker2.cluster.dev
[Normal][jenkins/test-pod-2hdfp-gc2qb][Pulled] Container image &quot;node:14-alpine&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-gc2qb][Created] Created container node14
[Normal][jenkins/test-pod-2hdfp-gc2qb][Started] Started container node14
[Normal][jenkins/test-pod-2hdfp-gc2qb][Pulled] Container image &quot;trion/jenkins-docker-client&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-gc2qb][Created] Created container docker
[Normal][jenkins/test-pod-2hdfp-gc2qb][Started] Started container docker
[Normal][jenkins/test-pod-2hdfp-gc2qb][Pulled] Container image &quot;jenkins/inbound-agent:4.3-4&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-gc2qb][Created] Created container jnlp
[Normal][jenkins/test-pod-2hdfp-gc2qb][Started] Started container jnlp
jenkins/test-pod-2hdfp-gc2qb Container node14 was terminated (Exit Code: 0, Reason: Completed)
Still waiting to schedule task
‘Jenkins’ doesn’t have label test-pod’
[Normal][jenkins/test-pod-2hdfp-gc2qb][Killing] Stopping container docker
Created Pod: kubernetes jenkins/test-pod-2hdfp-xwkm2
[Normal][jenkins/test-pod-2hdfp-xwkm2][Scheduled] Successfully assigned jenkins/test-pod-2hdfp-xwkm2 to kube-worker2.cluster.dev
[Normal][jenkins/test-pod-2hdfp-xwkm2][Pulled] Container image &quot;node:14-alpine&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-xwkm2][Created] Created container node14
[Normal][jenkins/test-pod-2hdfp-xwkm2][Started] Started container node14
[Normal][jenkins/test-pod-2hdfp-xwkm2][Pulled] Container image &quot;trion/jenkins-docker-client&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-xwkm2][Created] Created container docker
[Normal][jenkins/test-pod-2hdfp-xwkm2][Started] Started container docker
[Normal][jenkins/test-pod-2hdfp-xwkm2][Pulled] Container image &quot;jenkins/inbound-agent:4.3-4&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-xwkm2][Created] Created container jnlp
[Normal][jenkins/test-pod-2hdfp-xwkm2][Started] Started container jnlp
jenkins/test-pod-2hdfp-xwkm2 Container node14 was terminated (Exit Code: 0, Reason: Completed)
[Normal][jenkins/test-pod-2hdfp-xwkm2][Killing] Stopping container docker
Created Pod: kubernetes jenkins/test-pod-2hdfp-4ltq3
[Normal][jenkins/test-pod-2hdfp-4ltq3][Scheduled] Successfully assigned jenkins/test-pod-2hdfp-4ltq3 to kube-worker2.cluster.dev
[Normal][jenkins/test-pod-2hdfp-4ltq3][Pulled] Container image &quot;node:14-alpine&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-4ltq3][Created] Created container node14
[Normal][jenkins/test-pod-2hdfp-4ltq3][Started] Started container node14
[Normal][jenkins/test-pod-2hdfp-4ltq3][Pulled] Container image &quot;trion/jenkins-docker-client&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-4ltq3][Created] Created container docker
[Normal][jenkins/test-pod-2hdfp-4ltq3][Started] Started container docker
[Normal][jenkins/test-pod-2hdfp-4ltq3][Pulled] Container image &quot;jenkins/inbound-agent:4.3-4&quot; already present on machine
[Normal][jenkins/test-pod-2hdfp-4ltq3][Created] Created container jnlp
[Normal][jenkins/test-pod-2hdfp-4ltq3][Started] Started container jnlp
jenkins/test-pod-2hdfp-4ltq3 Container node14 was terminated (Exit Code: 0, Reason: Completed)
[Normal][jenkins/test-pod-2hdfp-4ltq3][Killing] Stopping container docker
Created Pod: kubernetes jenkins/test-pod-2hdfp-0216w
...
</code></pre>
<p><strong>Update with latest findings</strong></p>
<p>Master log (see <a href="https://github.com/jenkinsci/kubernetes-plugin/#debugging" rel="nofollow noreferrer">debugging</a>) doesn't provide much else:</p>
<pre><code>...
2021-04-30 11:52:42.715+0000 [id=4660]  INFO    hudson.slaves.NodeProvisioner#lambda$update$6: test-pod-gb4vq-hf3d4 provisioning successfully completed. We have now 2 computer(s)
2021-04-30 11:52:42.741+0000 [id=4659]  INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins/test-pod-gb4vq-hf3d4
2021-04-30 11:52:42.847+0000 [id=4680]  WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: test-pod-gb4vq-pdd69, template=PodTemplate{id='f29ecbdd-9c1d-468f-86ff-dd46ff40f306', name='test-pod-gb4vq', namespace='jenkins', label='test-pod', containers=[ContainerTemplate{name='node14', image='node:14-alpine'}, ContainerTemplate{name='docker', image='trion/jenkins-docker-client'}], annotations=[PodAnnotation{key='buildUrl', value='http://172.16.1.12/job/hello/14/'}, PodAnnotation{key='runUrl', value='job/hello/14/'}]}
java.lang.IllegalStateException: Pod is no longer available: jenkins/test-pod-gb4vq-pdd69
...
</code></pre>
<p>except that it suggests the container is starting up, then failing. It appears the loop is because the error handling in the Kubernetes plug-in isn't properly catching it and failing the job.</p>
<p>By watching for the build pod (using k9s) I am able to capture the pod's log, and <code>Unknown client name</code> also sounds like it is caused by fast container termination:</p>
<pre><code> jnlp INFO: [JNLP4-connect connection to 172.16.1.12/172.16.1.12:50000] Local headers refused by remote: Unknown client name: test-pod-34sd7-5xhs2                                                                                              
 jnlp Apr 29, 2021 10:42:15 PM hudson.remoting.jnlp.Main$CuiListener status                                                                                                                                                                      
 jnlp INFO: Protocol JNLP4-connect encountered an unexpected exception                                                                                                                                                                           
 jnlp java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: test-pod-34sd7-5xhs2
 jnlp     at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)                                                                                                                                                             
 jnlp     at hudson.remoting.Engine.innerRun(Engine.java:743)                                                                                                                                                                                    
 jnlp     at hudson.remoting.Engine.run(Engine.java:518)                                                                                                                                                                                         
 jnlp Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: test-pod-34sd7-5xhs2                                                                                                                     
 jnlp     at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378)
</code></pre>
<p>Just found a similar <a href="https://stackoverflow.com/questions/63097836/pods-terminated-immediately-from-jenkins-to-kubernetes-cluster">issue</a></p>
<p>This is useful: I added <code>podRetention: always(),</code> to <code>podTemplate()</code> after <code>label</code> so the agent pods don't terminate, and they show <code>Error</code>.</p>
<p><strong>Good finding</strong></p>
<p>With the above retaining the pod on error, I can now find <code>/var/log/containers/&lt;failed pod&gt;.log</code> and it has led me to a root cause.</p>
<p><code>2021-04-30T08:59:36.047989534-04:00 stderr F java.net.UnknownHostException: updates.jenkins.io</code></p>
<p>This is because of <code>dnsPolicy</code> that limits DNS to cluster-only lookups. The fix for this is to add <code>hostNetwork: true</code> to <code>podTemplate()</code> next to <code>label</code>.</p>
<p>Next, the image <code>trion/jenkins-docker-client</code> as recommended by the blog is a client AND a server, so it is the wrong image.</p>
<p>Switching to <code>jenkins/agent</code> creates a new problem. The pod now goes up and down doing nothing, not even logging. I suspect this is a launch parameter issue.</p>
<p>Now it is clear I shouldn't even have a Jenkins container in the Jenkinsfile, because the Kubernetes plug-in will automatically start a JNLP container.</p>
<p>And that means the problem is, at last, the node14 container - which either is immediately erroring, or immediately finding nothing to do and terminating.</p>

jws
<p>The error handling is difficult to understand and troubleshoot, and the blog is wrong.</p>
<p>Start with a bare minimum working agent Jenkinsfile:</p>
<pre><code>podTemplate(
    name: 'build-pod',
    namespace: 'jenkins',
    podRetention: always(),  // for debugging
    {
        node(POD_LABEL) {
            stage('Build') {
                sh &quot;echo hello&quot;
            }
        }
    }
)
</code></pre>
<p>From there, extend it with <code>containers</code>, <code>volumes</code>, <code>container</code> build sections, etc. one step at a time.</p>
<p>Troubleshoot using the logs:</p>
<p><code>kubectl get pods -n jenkins</code> to list the pod name, and then
<code>kubectl logs -f &lt;jenkins-pod&gt; -n jenkins</code></p>
<p>(assuming <code>jenkins</code> is your Kubernetes namespace)</p>
