<p>Hey I have a kubernetes cluster for a gitlab ci/cd pipeline.
There is a gitlab runner (kubernetes executor) running on it.</p>
<p>Sometimes the pipeline passes, but sometime I get</p>
<pre><code>Waiting for pod gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z to be running, status is Pending
    ContainersNotInitialized: &quot;containers with incomplete status: [init-permissions]&quot;
    ContainersNotReady: &quot;containers with unready status: [build helper]&quot;
    ContainersNotReady: &quot;containers with unready status: [build helper]&quot;
ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
</code></pre>
<p>I checked the link but it says the kubernetes executor should not cause any problem with shell profiles.
So I ran <code>kubectl describe pod gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z</code></p>
<pre><code>...
Events:
  Type     Reason                    Age   From               Message
  ----     ------                    ----  ----               -------
  Normal   Scheduled                 40s   default-scheduler  Successfully assigned gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z to bloxberg
  Warning  FailedCreatePodContainer  5s    kubelet            unable to ensure pod container exists: failed to create container for [kubepods besteffort pod6fe2669a-ae7f-47e3-8794-814767c14895] : Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
</code></pre>
<p>Why does the runner fail to start <code>systemd</code> is there a way to fix that?</p>
<h3>EDIT:</h3>
<p>I looked at the pods running on the cluster and it seems that there is an issue with the runner. with <code>kubectl describe  pod gitlab-runner-gitlab-runner-6b7bf4d766-9t4k6 -n gitlab-runner</code> I get:</p>
<pre><code>  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  7m42s (x52196 over 21d)  kubelet  Readiness probe failed:
  Warning  BackOff    2m42s (x91155 over 21d)  kubelet  Back-off restarting failed container
</code></pre>
<p>So there is an issue with the runner, but the error message says nothing about the cause</p>

iaquobe
<p>According to the <a href="https://docs.gitlab.com/runner/executors/kubernetes.html#troubleshooting" rel="nofollow noreferrer">troubleshooting docs</a> for the kubernetes executor, you should increase your <code>poll_timeout</code> value in your <code>config.toml</code>:</p>
<blockquote>
<p>The following errors are commonly encountered when using the Kubernetes executor.</p>
<p><code>Job failed (system failure): timed out waiting for pod to start</code><br />
If the cluster cannot schedule the build pod before the timeout defined by poll_timeout, the build pod returns an error. The Kubernetes Scheduler should be able to delete it.</p>
<p>To fix this issue, increase the poll_timeout value in your config.toml file.</p>
</blockquote>
