<p>I am having a problem with my GitLab pipeline when trying to mount a docker volume inside a container. Before I explain the problem, first I will describe my whole setup, because I think that is very essential to understand the problem, because I think that this is the reason why I am having this problem.</p>
<h2>Setup</h2>
<p>Okay, so to start off, I have a kubernetes cluster. This cluster runs my <code>gitlab/gitlab-ee:15.8.0-ee.0</code> image. I installed a GitLab runner in this cluster as well, so that I am able to run pipelines of course. Then the last thing I installed is a docker instance, because I saw that you can mount the <code>docker.sock</code> from your host machine to the gitlab pipeline, but this is not recommended, because the entire cluster relies on that <code>docker.sock</code>, so I have another instance of docker running and I am mounting that <code>docker.sock</code> for pipelines only. These 3 deployments are used by me to run GitLab pipelines.</p>
<h2>The problem</h2>
<p>I am happy with the way everything is setup, but I think I am still missing some configuration, because the mounting of docker volumes are not working properly in pipelines. I have this script to test this, which contains this code:</p>
<pre><code>image: docker:20.10.16-dind

variables:
  DOCKER_HOST: &quot;tcp://docker-service:2375&quot; # &lt;-- Address to reach the docker instance from my cluster
  DOCKER_COMPOSE_CMD: &quot;docker-compose -f docker-compose-test.yml&quot;

  
stages:
  - test

test:
  stage: test
  script:
    - $DOCKER_COMPOSE_CMD down --volumes --remove-orphans
    - $DOCKER_COMPOSE_CMD build
    - $DOCKER_COMPOSE_CMD --env-file .env.pipeline up -d
    - $DOCKER_COMPOSE_CMD exec -T -e APP_ENV=testing laravel-api-test sh -c &quot;ls&quot;
</code></pre>
<p>With the following docker-compose-test.yml:</p>
<pre><code>version: '3.7'
services:
  laravel-api-test:
    build:
      context: .
      dockerfile: docker/development/Dockerfile
    volumes:
      - .:/var/www/html
    environment:
    - COMPOSER_MEMORY_LIMIT=-1
    depends_on:
    - database-test
  database-test:
    image: postgres:15.1-alpine
    ports:
    - ${DB_PORT}:5432
    environment:
      POSTGRES_DB: ${DB_DATABASE}
      POSTGRES_PASSWORD: ${DB_PASSWORD_SECRET}
      POSTGRES_USER: ${DB_USERNAME_SECRET}
  redis-test:
    image: redis:7.0.8
    ports:
    - ${REDIS_PORT}:6379
networks:
  default:
    name: application
</code></pre>
<p>Now what this pipeline does, it builds the docker containers and then starts them. Then it runs the <code>ls</code> command which prints out all the files in the working-dir of the container. However, this working-dir is empty. This is caused by the volume mount in the <code>docker-compose-test.yml</code> with this line:</p>
<pre><code>volumes:
  - .:/var/www/html
</code></pre>
<p>In the Dockerfile I also have this:</p>
<pre><code>COPY . /var/www/html/
</code></pre>
<p>So when I remove the volume mount in <code>docker-compose-test.yml</code>, all files are there, so the copying does work for the <code>Dockerfile</code>, but not mounting it later on. I saw <a href="https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41227" rel="nofollow noreferrer">this</a> thread and tried some of their solutions and tested it with their test script:</p>
<pre><code>variables:
  SHARED_PATH: /builds/shared/$CI_PROJECT_PATH
script:
  - mkdir -p ${SHARED_PATH}
  - touch ${SHARED_PATH}/test_file
  - docker run -v ${SHARED_PATH}:/mnt ubuntu ls /mnt
</code></pre>
<p>But this still resulted in an empty <code>/mnt</code> dir, while the <code>test_file</code> should have been there. In the GitLab runner I added this line to the config:</p>
<pre><code>volumes = [&quot;/cache&quot;, &quot;/builds:/builds&quot;]
</code></pre>
<p>Unfortunately, this did not change anything. I am not sure, but my guess is that I need to access the /builds from my other docker instance, because I have a feeling that I am mounting the /builds from the host machine, which is not the docker I am using in my pipeline. If this is the case, I am not sure how to configure my Kubernetes cluster to use the other one. The weird thing is that when
I do <code>cd /builds/projects/laravel-api</code> (my repo is named <code>laravel-api</code> and its inside the <code>projects</code> group) and then <code>ls</code> in my pipeline, I do see my repository containing all the files. But when I try to mount that directory in my docker-compose-test.yml I still get an empty dir. So I mean this:</p>
<pre><code>volumes:
  - /builds/projects/laravel-api:/var/www/html
</code></pre>
<p>So every way of mounting volumes after builds are resulting in empty directories...</p>
<h2>Wrap up</h2>
<p>So to summarize the problem. Every form of mounting I do in my pipeline results eventually in an empty directory. When copying files from a Dockerfile only the directory does work, but that is not something I can work with.</p>
<p>I hope this covers the entire problem. Some help is really appreciated! If there are any questions about the setup or something like that, please ask I will respond ASAP!</p>

<p>This is a tricky problem requiring in-depth explanation of GitLab and Kubernetes dynamics.</p>
<p><strong>Summary of your problem:</strong></p>
<blockquote>
<pre><code>  volumes:
    - .:/var/www/html
    # or 
    # - /builds/projects/laravel-api:/var/www/html
</code></pre>
</blockquote>
<p>The Docker service you are using does not share the same filesystem than your job's Pod. When you instruct Docker to mount <code>/builds/projects/laravel-api</code> (or <code>.</code> which resolves to the same), Docker will mount this directory <em>from its own filesystem</em> which is indeed empty.</p>
<p><img src="https://i.stack.imgur.com/JBdvN.png" alt="" /></p>
<p>As you already pointed out, you must somehow share the <code>/builds</code> directory between job's Pod and Docker service.</p>
<h2>Solution 1: share Persistent Volume between Docker service and job's Pod</h2>
<p>Create a <a href="https://docs.gitlab.com/runner/executors/kubernetes.html#persistentvolumeclaim-volume" rel="nofollow noreferrer">Persistent Volume Claim (PVC)</a> so that they share the <code>/builds</code> directory:</p>
<ul>
<li>Create a PVC such as:
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-claim
spec:
  accessModes:
    - ReadWriteOnce
    # or ReadWriteMany
resources:
  requests:
    storage: 8Gi
</code></pre>
</li>
<li>Configure GitLab Runner Kubernetes executor to mount the PVC at <code>/builds</code>. For example:
<pre class="lang-ini prettyprint-override"><code>[[runners]]
  [runners.kubernetes]
      [[runners.kubernetes.volumes.pvc]]
      name = &quot;gitlab-claim&quot;
      mount_path = &quot;/builds&quot;
</code></pre>
</li>
<li>Configure Docker deployment to mount the PVC at <code>/builds</code>. This depends on how you configured Docker service, but you'll probably have to configure a Container spec such as:
<pre class="lang-yaml prettyprint-override"><code>spec:
  volumes:
    - name: job-volume
      persistentVolumeClaim:
        claimName: gitlab-claim
  containers:
    - name: docker
      image: docker:dind
      # ...
      volumeMounts:
        - name: job-volume
          mountPath: /builds
</code></pre>
</li>
</ul>
<p>Your setup will look something like this. Both pods will share the same Volume mounted at <code>/builds</code>.</p>
<p><img src="https://i.stack.imgur.com/4OLW4.png" alt="" /></p>
<p>Important note: choose carefully between <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer"><code>ReadWriteOnce</code> / <code>ReadWriteMany</code> access mode</a>:</p>
<ul>
<li>Use <code>ReadWriteMany</code> if your provider supports it, it will allow the same volume to be shared across multiple nodes.</li>
<li>If not, <code>ReadWriteOnce</code> will require your job's Pods to be running on the same node as Docker service as volume won't be shareable across Kubernetes nodes.</li>
</ul>
<h2>Solution 2: use GitLab service to run Docker-in-Docker (DinD)</h2>
<p>This setup is a bit different as you'll deploy a fresh Docker service every build. However data won't be persisted across jobs as Docker service will be recreated for each job.</p>
<p><a href="https://docs.gitlab.com/runner/executors/kubernetes.html#mount-volumes-on-service-containers" rel="nofollow noreferrer"><em>Volumes defined for the build container are also automatically mounted for all services containers.</em></a>. You can then share an <code>emptyDir</code> between job's pod and a Docker DinD service:</p>
<ul>
<li>Configure a <a href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html" rel="nofollow noreferrer">DinD service on your CI</a> such as:
<pre class="lang-yaml prettyprint-override"><code>image: docker:20.10.16-dind
variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: &quot;&quot;

services:
- docker:20.10.16-dind
</code></pre>
</li>
<li>Configure Kubernetes executor to mount an <a href="https://docs.gitlab.com/runner/executors/kubernetes.html#mount-volumes-on-service-containers" rel="nofollow noreferrer"><code>emptyDir</code> at <code>/builds</code></a> which will be shared between job's and services containers:
<pre class="lang-ini prettyprint-override"><code>[runners.kubernetes]
  [[runners.kubernetes.volumes.empty_dir]]
    name = &quot;builds-data&quot;
    mount_path = &quot;/builds&quot;
</code></pre>
</li>
</ul>
<p>In this case, you'll have a single Pod with both job's container and DinD service container, both sharing the <code>emptyDir</code> volume <code>/builds</code>. Your <code>laravel</code> and other containers will run <em>inside</em> the DinD service container.</p>
<p><img src="https://i.stack.imgur.com/lp94D.png" alt="enter image description here" /></p>
<h2>Which solution is better?</h2>
<p>Depends on context:</p>
<p>Efficiency-wize, Solution 1 is better as your Docker service will remain static and keep build caches, downloaded images, etc. allowing much faster Docker builds and deployments. However, your Docker service may be shared with multiple projects/entities, causing security risk: anyone with access to Docker service may also access containers not related to their project or scope and eventually retrieve sensitive data from within another projects container/volume/...</p>
<p>For example: consider project A and project B using Docker service to deploy testing apps, each configured to access AWS account A &amp; B. Entity with access to Project A may easily grab credentials from a container running for Project B and access AWS account B to start a crypto miner on EC2.</p>
<p>It's possible to mitigate that risk by deploying Docker services dedicated per project or scope depending on your need, and/or securing via SSH/TLS to only allow certain entities to use it.</p>
<p>Solution 2 is much easier to setup and secure, but will be less efficient: a new CI Docker service will be started each build, so you'll have to download Docker images again every pipeline and any Docker build cache will be lost. You may optimize CI config to download cache or such, but it requires a more complex setup (might as well go for Solution 2 if you're ready to add complexity)</p>
<hr />
<p>This is all a bit complex and plays with lots of various concepts. Do not hesitate to ask questions, I'll answer or edit as possible.</p>
