<p>Hey I have a kubernetes cluster for a gitlab ci/cd pipeline.
There is a gitlab runner (kubernetes executor) running on it.</p>
<p>Sometimes the pipeline passes, but sometime I get</p>
<pre><code>Waiting for pod gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z to be running, status is Pending
    ContainersNotInitialized: &quot;containers with incomplete status: [init-permissions]&quot;
    ContainersNotReady: &quot;containers with unready status: [build helper]&quot;
    ContainersNotReady: &quot;containers with unready status: [build helper]&quot;
ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
</code></pre>
<p>I checked the link but it says the kubernetes executor should not cause any problem with shell profiles.
So I ran <code>kubectl describe pod gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z</code></p>
<pre><code>...
Events:
  Type     Reason                    Age   From               Message
  ----     ------                    ----  ----               -------
  Normal   Scheduled                 40s   default-scheduler  Successfully assigned gitlab-runner/runner-wyplq6-h-project-7180-concurrent-0lr66z to bloxberg
  Warning  FailedCreatePodContainer  5s    kubelet            unable to ensure pod container exists: failed to create container for [kubepods besteffort pod6fe2669a-ae7f-47e3-8794-814767c14895] : Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
</code></pre>
<p>Why does the runner fail to start <code>systemd</code> is there a way to fix that?</p>
<h3>EDIT:</h3>
<p>I looked at the pods running on the cluster and it seems that there is an issue with the runner. with <code>kubectl describe  pod gitlab-runner-gitlab-runner-6b7bf4d766-9t4k6 -n gitlab-runner</code> I get:</p>
<pre><code>  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  7m42s (x52196 over 21d)  kubelet  Readiness probe failed:
  Warning  BackOff    2m42s (x91155 over 21d)  kubelet  Back-off restarting failed container
</code></pre>
<p>So there is an issue with the runner, but the error message says nothing about the cause</p>

iaquobe
<p><strong>Job failed (system failure): timed out waiting for pod to start</strong></p>
<p>The following error occurs if the cluster cannot schedule the build pod before the timeout defined by poll_timeout, the build pod returns an error. The <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-lifetime" rel="nofollow noreferrer">Kubernetes Scheduler</a> should be able to delete it.
To fix this issue, increase the <code>poll_timeout</code> value in your <code>config.toml</code> file.</p>
<p><strong>Error cleaning up pod and Job failed (system failure): prepare environment: waiting for pod running</strong></p>
<p>The following error occurs when Kubernetes fails to schedule the job pod in a timely manner. GitLab Runner waits for the pod to be ready, but it fails and then tries to clean up the pod, which can also fail.</p>
<p>To troubleshoot, check the Kubernetes primary node and all nodes that run a <code>kube-apiserver</code> instance. Ensure they have all of the resources needed to manage the target number of pods that you hope to scale up to on the cluster.</p>
<p>To change the time GitLab Runner waits for a pod to reach its <strong>Ready</strong> status, use the <a href="https://docs.gitlab.com/runner/executors/kubernetes.html#other-configtoml-settings" rel="nofollow noreferrer">poll_timeout</a> setting.</p>
<p>To better understand how pods are scheduled or why they might not get scheduled on time, read about the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/" rel="nofollow noreferrer">Kubernetes Scheduler</a>.</p>
<p>Refer to the troubleshooting <a href="https://docs.gitlab.com/runner/executors/kubernetes.html#job-failed-system-failure-timed-out-waiting-for-pod-to-start" rel="nofollow noreferrer">documentation</a> for more details.</p>
