<p>When using <a href="https://docs.gitlab.com/ee/topics/autodevops/" rel="nofollow noreferrer">GitLab Auto DevOps</a> to build and deploy application from my repository to <a href="https://microk8s.io/" rel="nofollow noreferrer">microk8s</a>, the build jobs often take a long time to run, eventually timing out. The issue happens 99% of the time, but some builds run through. Often, the build stops at a different time in the build script.</p>
<p>The projects do not contain a <code>.gitlab-ci.yml</code> file and fully rely on the Auto DevOps feature to do its magic.</p>
<p>For Spring Boot/Java projects, the build often fails when downloading the Gradle via the Gradle wrapper, other times it fails while downloading the dependencies itself. The error message is very vague and not helpful at all:</p>
<pre><code>Step 5/11 : RUN /bin/herokuish buildpack build
 ---&gt; Running in e9ec110c0dfe
       -----&gt; Gradle app detected
-----&gt; Spring Boot detected
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 35
</code></pre>
<p>Sometimes, if you get lucky, the error is different:</p>
<pre><code>Step 5/11 : RUN /bin/herokuish buildpack build
 ---&gt; Running in fe284971a79c
       -----&gt; Gradle app detected
-----&gt; Spring Boot detected
-----&gt; Installing JDK 11... done
-----&gt; Building Gradle app...
-----&gt; executing ./gradlew build -x check
       Downloading https://services.gradle.org/distributions/gradle-7.0-bin.zip
       ..........10%...........20%...........30%..........40%...........50%...........60%...........70%..........80%...........90%...........100%
       To honour the JVM settings for this build a single-use Daemon process will be forked. See https://docs.gradle.org/7.0/userguide/gradle_daemon.html#sec:disabling_the_daemon.
       Daemon will be stopped at the end of the build
       &gt; Task :compileJava
       &gt; Task :compileJava FAILED
       
       FAILURE: Build failed with an exception.
       
       * What went wrong:
       Execution failed for task ':compileJava'.
       &gt; Could not download netty-resolver-dns-native-macos-4.1.65.Final-osx-x86_64.jar (io.netty:netty-resolver-dns-native-macos:4.1.65.Final)
       &gt; Could not get resource 'https://repo.maven.apache.org/maven2/io/netty/netty-resolver-dns-native-macos/4.1.65.Final/netty-resolver-dns-native-macos-4.1.65.Final-osx-x86_64.jar'.
       &gt; Could not GET 'https://repo.maven.apache.org/maven2/io/netty/netty-resolver-dns-native-macos/4.1.65.Final/netty-resolver-dns-native-macos-4.1.65.Final-osx-x86_64.jar'.
       &gt; Read timed out
</code></pre>
<p>For React/TypeScript problems, the symptoms are similar but the error itself manifests in a different way:</p>
<pre><code>[INFO] Using npm v8.1.0 from package.json
/cnb/buildpacks/heroku_nodejs-npm/0.4.4/lib/build.sh: line 179: /layers/heroku_nodejs-engine/toolbox/bin/yj: Permission denied
ERROR: failed to build: exit status 126
ERROR: failed to build: executing lifecycle: failed with status code: 145
</code></pre>
<p>The problem seems to occur mostly when the GitLab runners itself are deplyoed in Kubernetes. microk8s uses <a href="https://projectcalico.docs.tigera.io/getting-started/kubernetes/" rel="nofollow noreferrer">Project Calico</a> to implement virtual networks.</p>
<p>What gives? Why are the error messages to unhelpful? Is there a way to turn up verbose build logs or debug the build steps?</p>

<p>This seems to be a networking problem caused by incompatbile MTU settings between the Calico network layer and Docker's network configuration (and an inability to autoconfige the MTU correctly?) When the MTU values don't match, network packets get fragmented and the Docker runners fail to complete TLS handshakes. As far as I understand, this only affects DIND (docker-in-docker) runners.</p>
<p>Even finding this out requires jumping through a few hoops. You have to:</p>
<ol>
<li>Start a CI pipeline and wait for the job to &quot;hang&quot;</li>
<li><code>kubectl exec</code> into the current/active GitLab runner pod</li>
<li>Find out the correct value for the <code>DOCKER_HOST</code> environment variable (e.g. by grepping through <code>/proc/$pid/environ</code>. Very likely, this will be <code>tcp://localhost:2375</code>.</li>
<li>Export the value to be used by the <code>docker</code> client: <code>export DOCKER_HOST=tcp://localhost:2375</code></li>
<li><code>docker ps</code> and then <code>docker exec</code> into the actual CI job container</li>
<li>Use ping and other tools to find proper MTU values (but MTU for what? Docker, Calico, OS, router, …?). Use curl/openssl to verify that (certain) https sites cause problems from inside the DIND container.</li>
</ol>
<p>Execute</p>
<pre><code>microk8s kubectl get -n kube-system cm calico-config -o yaml
</code></pre>
<p>and look for the <code>veth_mtu</code> value, which will very likely be set to <code>1440</code>. DIND uses the same MTU and thus fails send or receive certain network packages (each virtual network needs to add its own header to the network packet, which adds a few bytes at every layer).</p>
<p>The naïve fix would be to change the Calico settings to a higher or lower value, but somehow this did not really work, even after the Calico deployment. Furthermore, the value seems to be reset to its original value from time to time; probably caused by automatic updates to microk8s (which comes as a <a href="https://snapcraft.io/" rel="nofollow noreferrer">Snap</a>).</p>
<p>So what is a solution that actually works and is permanent? It is possible to override DIND settings for Auto DevOps by writing a custom <code>.gitlab-ci.yml</code> file and simply includes the Auto DevOps template:</p>
<pre class="lang-yaml prettyprint-override"><code>build:
  services:
    - name: docker:20.10.6-dind # make sure to update version
      command: ['--tls=false', '--host=tcp://0.0.0.0:2375', '--mtu=1240']

include:
    - template: Auto-DevOps.gitlab-ci.yml
</code></pre>
<p>The <code>build.services</code> definition is copied from the <a href="https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml" rel="nofollow noreferrer"><code>Jobs/Build.gitlab-ci</code></a> template and extended with an additional <code>--mtu</code> option.</p>
<p>I've had good experience so far by setting the DIND MTU to 1240, which is 200 bytes lower than Calico's MTU. As an added bonus, it doesn't affect any other pods' network settings. And for CI builds I can live with non-optimal network settings.</p>
<p>References:</p>
<ul>
<li><a href="https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27300" rel="nofollow noreferrer">https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27300</a></li>
<li><a href="https://projectcalico.docs.tigera.io/networking/mtu" rel="nofollow noreferrer">https://projectcalico.docs.tigera.io/networking/mtu</a></li>
<li><a href="https://liejuntao001.medium.com/fix-docker-in-docker-network-issue-in-kubernetes-cc18c229d9e5" rel="nofollow noreferrer">https://liejuntao001.medium.com/fix-docker-in-docker-network-issue-in-kubernetes-cc18c229d9e5</a></li>
<li><a href="https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router" rel="nofollow noreferrer">https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router</a></li>
</ul>
