title: Reuse a conda environment within the same gitlab CI pipeline?
body: <h3>Motivation</h3>
  <p>My main goal is this: <em>Within a pipeline, I would like to reuse as much as possible (i.e. not build the conda environment multiple times, if all jobs share the same environment)</em>.</p>
  <p>In my project, I use conda as depency manager and gitlab ci/cd for continuous integration. For the sake of simplicity, let's say I have a <em>build</em> job and a <em>test</em> job. The most straight forward approach would be to create the conda environment from the <code>environment.yml</code> in any job and then do the actual work. This adds an overhead of several minutes to any job. It also seems like overhead to me, since I would like to build the environment once in the <em>build</em> job and then use it in my <em>test</em> job (especially when creating multiple jobs for different tests).</p>
  <h3>Research Results</h3>
  <p>The first thing I need to do is to set the <code>CONDA_ENVS_PATH</code> to somewhere in my project directory.</p>
  <p>I've looked at gitlab's caching mechanism, but found that it only helps for the same job in repeated runs of the same pipeline, but not not for different jobs of the same run within a pipeline.</p>
  <p>I've also looked at gitlab's artifacts mechanism, but found that due the up- and download of those, they don't increase run time significantly (basically I only save time by not downloading many small packages and not having to compile them again, but loose time by compressing and decompressing them).</p>
  <p>I've also tried to make use of the <code>GIT_CLEAN_FLAGS</code> by setting them to <code>none</code> in my <em>test</em> job. That way, the conda environment is not deleted when getting the latest data from git. This does cause a serious speedup in my pipelines, but it does not work all the time. Some jobs fail, not finding the conda environment. A simple rerun does magically work however. Of course, in a CI / CD setting, this nondeterminism is not practical.</p>
  <p>As a workaround to the original question, we've come up with an intermediate solution. We introduce a docker image, holding our custom environment, through a minimal Dockerfile and a couple of changes to our <code>.gitlab-ci.yml</code> (see below for an example). By only executing the job that builds our custom docker image when the dockerfile or environment changed, we save valuable time on each run. At the same time, we keep the full flexibility in our environment definition and can adjust it exactly how we usually would: by changing the <code>environment.yml</code>.</p>
  <h3>Question</h3>
  <p>All solutions tried so far are not really satisfactory. Thus my question is: <strong>How can my <em>test</em> job reuse the same conda environment as my <em>build</em> job in gitlab-ci?</strong></p>
  <hr />
  <p>In case someone else would like to use a similar setup: Here is my current approach:</p>
  <pre class="lang-sh prettyprint-override"><code># Dockerfile
  
  FROM continuumio/miniconda3:latest
  
  COPY environment.yml .
  
  RUN conda env create -f environment.yml
  
  ENTRYPOINT [&quot;&quot;]
  </code></pre>
  <pre class="lang-yaml prettyprint-override"><code># .gitlab-ci.yml
  
  # Use the latest version of this project's docker file
  # This will be the default image for all jobs unless specified otherwise
  image: $CI_REGISTRY_IMAGE:latest
  
  
  # Change cache directories to be inside the project directory since we can
  # only cache local items.
  variables:
    PRE_COMMIT_HOME: &quot;${CI_PROJECT_DIR}/.cache/pre-commit&quot;
  
  
  stages:
    - build
    - test
  
  
  # Make conda environment available to all jobs
  # This expects the conda environment to have the same name as the gitlab project path
  # Avoid dashes and other non-alphabetical characters
  default:
    before_script:
      - source activate &quot;${CI_PROJECT_NAME}&quot;
  
  # Build the docker image including the correct conda environment for subsequent jobs
  # This assumes a docker image registry being configured for your gitlab instance
  dockerimage:
    stage: build
    image:
      name: gcr.io/kaniko-project/executor:debug
      entrypoint: [&quot;&quot;]
    rules:
      - changes:
        - Dockerfile
        - environment.yml
    before_script: [ ]
    script:
      - mkdir -p /kaniko/.docker
      - echo &quot;{\&quot;auths\&quot;:{\&quot;$CI_REGISTRY\&quot;:{\&quot;username\&quot;:\&quot;$CI_REGISTRY_USER\&quot;,\&quot;password\&quot;:\&quot;$CI_REGISTRY_PASSWORD\&quot;}}}&quot; &gt; /kaniko/.docker/config.json
      - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:latest
  
  
  # Run pytest
  pytest:
    stage: test
    script:
      - conda install pytest-cov
      - pytest tests --cov=src
  </code></pre>
  <hr />
  <p>Edit: I replaced the example code for using GIT_CLEAN_FLAGS with our most recent approach: using a custom docker image.</p>
  <hr />
  <p>Disclaimer: I saw <a href="https://stackoverflow.com/questions/31623924/test-after-build-would-run-in-new-environment-on-gitlab-ci">this</a> and <a href="https://stackoverflow.com/questions/38869929/gitlab-ci-preserve-environment-between-build-stages">this</a> question, but they are both dated, don't have a satisfying answer and I only found them after writing this question, so I hope my additional question increases discoverability of the topic.</p>