title: Jenkins Kubernetes plugin failing to provision jnlp-slave pods
body: <p>I have a Kubernetes 1.10.0, Docker 17.03.2-ce, and Jenkins 2.107.1 running on an Ubuntu 17.04 VM with Kubernetes Plugin 1.5 installed in Jenkins.  I have 4 other Ubuntu VM(s) successfully set up as nodes in the cluster, including the untainted master.  I can deploy nginx-based services directly and have unfettered access to the dashboard.  So, Kubernetes itself seems happy enough.</p>
  
  <p>Before you mention it, let me say that we don't have short term plans to run Jenkins master inside Kubernetes itself.  So, I'd prefer to get this strategy working.</p>
  
  <p>The plugin config for a Kubernetes Cloud is thus:</p>
  
  <p>"Name": kubernetes</p>
  
  <p>"Kubernetes URL":  <a href="https://172.20.43.30:6443" rel="noreferrer">https://172.20.43.30:6443</a></p>
  
  <p>from</p>
  
  <pre><code># kubectl describe pods/kube-apiserver-jenkins-kube-master --namespace=kube-system | grep Liveness
  Liveness:     http-get https://172.20.43.30:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
  </code></pre>
  
  <p>after accepting the insecure cert, a browser to <a href="https://172.20.43.30:6443/" rel="noreferrer">https://172.20.43.30:6443/</a> will show</p>
  
  <pre><code>{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {
  
    },
    "status": "Failure",
    "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
    "reason": "Forbidden",
    "details": {
  
    },
    "code": 403
  }
  </code></pre>
  
  <p>"Kubernetes server certificate key" obtained from</p>
  
  <pre><code># kubectl get pods/kube-apiserver-jenkins-kube-master -o yaml --namespace=kube-system | grep tls
      - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
      - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
  
  # cat /etc/kubernetes/pki/apiserver.crt
  -----BEGIN CERTIFICATE-----
  MIIDZ******
  *******************
  ****PP5wigl
  -----END CERTIFICATE-----
  </code></pre>
  
  <p>"Kubernetes Namespace": jenkins-slaves</p>
  
  <p>the jenkins-slaves namespace setup like this ...</p>
  
  <p>create jenkins-namespace.yaml and add this:</p>
  
  <pre><code>apiVersion: v1
  kind: Namespace
  metadata:
    name: jenkins-slaves
    labels:
      name: jenkins-slaves
  spec:
    finalizers:
    - kubernetes
  </code></pre>
  
  <p>then</p>
  
  <pre><code># kubectl create -f jenkins-namespace.yaml
  namespace "jenkins-slaves" created
  
  # kubectl -n jenkins-slaves create sa jenkins
  serviceaccount "jenkins" created
  
  # kubectl create role jenkins --verb=get,list,watch,create,patch,delete --resource=pods
  role.rbac.authorization.k8s.io "jenkins" created
  
  # kubectl create rolebinding jenkins --role=jenkins --serviceaccount=jenkins-slaves:jenkins
  rolebinding.rbac.authorization.k8s.io "jenkins" created
  
  # kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=jenkins-slaves:jenkins
  clusterrolebinding.rbac.authorization.k8s.io "jenkins" created
  </code></pre>
  
  <p>added a Jenkins credential of "secret text" using the token spit out from</p>
  
  <pre><code># kubectl get -n jenkins-slaves sa/jenkins --template='{{range .secrets}}{{ .name }} {{end}}' | xargs -n 1 kubectl -n jenkins-slaves get secret --template='{{ if .data.token }}{{ .data.token }}{{end}}' | head -n 1 | base64 -d -
  </code></pre>
  
  <p>a "Test Connection" shows "Connection test successful"</p>
  
  <p>It should be noted that that same token can be used to login to the Kubernetes dashboard with full access rights.</p>
  
  <p>"Jenkins URL": <a href="http://172.20.43.30:8080" rel="noreferrer">http://172.20.43.30:8080</a></p>
  
  <p>"Kubernetes Pod Template:Name": jnlp slave</p>
  
  <p>"Kubernetes Pod Template:Namespace": jenkins-slaves</p>
  
  <p>"Kubernetes Pod Template:Labels": jenkins-slaves</p>
  
  <p>"Kubernetes Pod Template:Usage": Only build jobs with label expressions matching this node</p>
  
  <p>"Kubernetes Pod Template:Container Template:Name": jnlp-slave</p>
  
  <p>"Kubernetes Pod Template:Container Template:Docker image": jenkins/jnlp-slave</p>
  
  <p>"Kubernetes Pod Template:Container Template:Working directory": ./.jenkins-agent</p>
  
  <p>At this point, if I create a job and "Restrict where this project can be run" to a "Label Expression" of "jenkins-slaves", I get:</p>
  
  <pre><code>Label jenkins-slaves is serviced by no nodes and 1 cloud. Permissions or other restrictions provided by plugins may prevent this job from running on those nodes.
  </code></pre>
  
  <p>If I try to build the job, it will sit in the build queue and the "Build Executor Status" will periodically say "jnlp-slave-#####  (offline) (suspended)" and then disappear a couple seconds later.</p>
  
  <p>The system log says:</p>
  
  <pre><code>Apr 03, 2018 12:16:21 PM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher logLastLines
  Error in provisioning; agent=KubernetesSlave name: jnlp-slave-t8004, template=PodTemplate{inheritFrom='', name='jnlp slave', namespace='jenkins-slaves', label='jenkins-slaves', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume@44dcba2d, containers=[ContainerTemplate{name='jnlp-slave', image='jenkins/jnlp-slave', workingDir='./.jenkins-agent', command='/bin/sh -c', args='cat', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@58f0ceec}]}. Container jnlp exited with error 255. Logs: Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
  Warning: SECRET is defined twice in command-line arguments and the environment variable
  Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
  Apr 03, 2018 4:16:16 PM hudson.remoting.jnlp.Main createEngine
  INFO: Setting up agent: jnlp-slave-t8004
  Apr 03, 2018 4:16:16 PM hudson.remoting.jnlp.Main$CuiListener &lt;init&gt;
  INFO: Jenkins agent is running in headless mode.
  Apr 03, 2018 4:16:16 PM hudson.remoting.Engine startEngine
  INFO: Using Remoting version: 3.19
  Apr 03, 2018 4:16:16 PM hudson.remoting.Engine startEngine
  WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
  Apr 03, 2018 4:16:17 PM hudson.remoting.jnlp.Main$CuiListener status
  INFO: Locating server among [http://172.20.43.30:8080/]
  Apr 03, 2018 4:16:17 PM hudson.remoting.jnlp.Main$CuiListener error
  SEVERE: http://172.20.43.30:8080/tcpSlaveAgentListener/ is invalid: 404 Not Found
  java.io.IOException: http://172.20.43.30:8080/tcpSlaveAgentListener/ is invalid: 404 Not Found
      at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:197)
      at hudson.remoting.Engine.innerRun(Engine.java:518)
      at hudson.remoting.Engine.run(Engine.java:469)
  Apr 03, 2018 12:16:21 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
  Terminating Kubernetes instance for agent jnlp-slave-t8004
  Apr 03, 2018 12:16:21 PM WARNING io.fabric8.kubernetes.client.Config tryServiceAccount
  Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
  Apr 03, 2018 12:16:21 PM INFO okhttp3.internal.platform.Platform log
  ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
  Apr 03, 2018 12:16:21 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
  Terminated Kubernetes instance for agent jenkins-slaves/jnlp-slave-t8004
  Apr 03, 2018 12:16:21 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
  Disconnected computer jnlp-slave-t8004
  Apr 03, 2018 12:16:25 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
  Excess workload after pending Kubernetes agents: 1
  Apr 03, 2018 12:16:25 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
  Template: Kubernetes Pod Template
  Apr 03, 2018 12:16:25 PM WARNING io.fabric8.kubernetes.client.Config tryServiceAccount
  Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
  Apr 03, 2018 12:16:25 PM INFO okhttp3.internal.platform.Platform log
  ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
  Apr 03, 2018 12:16:25 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
  Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
  Apr 03, 2018 12:16:35 PM WARNING io.fabric8.kubernetes.client.Config tryServiceAccount
  Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
  Apr 03, 2018 12:16:35 PM INFO hudson.slaves.NodeProvisioner$2 run
  Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
  Apr 03, 2018 12:16:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
  Excess workload after pending Kubernetes agents: 0
  Apr 03, 2018 12:16:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
  Template: Kubernetes Pod Template
  Apr 03, 2018 12:16:35 PM INFO okhttp3.internal.platform.Platform log
  ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?
  Apr 03, 2018 12:16:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
  Created Pod: jnlp-slave-bnz94 in namespace jenkins-slaves
  Apr 03, 2018 12:16:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
  </code></pre>
  
  <p>-Steve Maring</p>
  
  <p>Orlando, FL</p>