title: Gitlab CI/CD Pass artifacts/variables between pipelines
body: <h1>tl;dr</h1>
  <p>How do I pass data, e.g. the <code>$BUILD_VERSION</code> variable, between jobs in different pipelines in Gitlab CI?</p>
  <p>So (in my case) this:</p>
  <pre><code>Pipeline 1 on push ect.            Pipeline 2 after merge
  
      `building` job ...                `deploying` job
            │                                ▲
            └─────── $BUILD_VERSION ─────────┘
  </code></pre>
  <hr />
  <h1>Background</h1>
  <p>Consider the following example (full <code>yml</code> below):</p>
  <pre class="lang-yaml prettyprint-override"><code>building:
      stage: staging
      # only on merge requests
      rules:
          # execute when a merge request is open
          - if: $CI_PIPELINE_SOURCE == &quot;merge_request_event&quot;
            when: always
          - when: never
      script:
          - echo &quot;BUILD_VERSION=1.2.3&quot; &gt; build.env
      artifacts:
          reports:
              dotenv: build.env
  
  deploying:
      stage: deploy
      # after merge request is merged
      rules:
          # execute when a branch was merged to staging
          - if: $CI_COMMIT_BRANCH == $STAGING_BRANCH
            when: always
          - when: never
      dependencies: 
          - building
      script:
          - echo $BUILD_VERSION
  </code></pre>
  <p>I have two stages, <em>staging</em> and <em>deploy</em>. The <code>building</code> job in <em>staging</em> builds the app and creates a &quot;Review App&quot; (no separate build stage for simplicity). The <code>deploying</code> job in <em>deploy</em> then uploads the new app.</p>
  <p>The pipeline containing the <code>building</code> job runs whenever a merge request is opened. This way the app is built and the developer can click on the &quot;Review App&quot; icon in the merge request. The <code>deploying</code> job is run right after the merge request is merged. The idea is the following:</p>
  <pre><code>                      *staging* stage (pipeline 1)        *deploy* stage (pipeline 2)
  
  &lt;open merge request&gt; -&gt; `building` job (and show)   ...   &lt;merge&gt; -&gt; `deploying` job
                               │                                            ▲
                               └───────────── $BUILD_VERSION ───────────────┘
  </code></pre>
  <p>The problem for me is, that the <em>staging</em>/<code>building</code> creates some data, e.g. a <code>$BUILD_VERSION</code>. I want to have this <code>$BUILD_VERSION</code> in the <em>deploy</em>/<code>deploying</code>, e.g. for creating a new release via the Gitlab API.</p>
  <p>So my question is: How do I pass the <code>$BUILD_VERSION</code> (and other data) from <em>staging</em>/<code>building</code> to <em>deploy</em>/<code>deploying</code>?</p>
  <hr />
  <h1>What I've tried so far</h1>
  <h2><code>artifacts.reports.dotenv</code></h2>
  <p>The described case is more less handled in the gitlab docs in <a href="https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job" rel="noreferrer">Pass an environment variable to another job</a>. Also the <code>yml</code> file shown below is heavily inspired by this example. Still, it does not work.</p>
  <p>The <code>build.env</code> artifact is created in <code>building</code>, but whenever the <code>deploying</code> job is executed, the <code>build.env</code> file gets removed as shown below in line 15: &quot;Removing build.env&quot;. I tried to add <code>build.env</code> to the <code>.gitignore</code> but it still gets removed.</p>
  <p><a href="https://i.stack.imgur.com/pYibY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pYibY.png" alt="Preparing environment - Running on runner- via gitlab-runner... - Getting source from Git repository - Fetching changes with git depth set to 50... - Reinitialized existing Git repository in  - Checking out as staging... - Removing build.env - Skipping Git submodules setup - Executing &quot;step_script&quot; stage of the job script - Using docker image - echo $BUILD_VERSION - Job succeeded" /></a></p>
  <p>After hours of searching I found in <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/246777#note_417210007" rel="noreferrer">this gitlab issue comment</a> and <a href="https://stackoverflow.com/a/65878853/5934316">this stackoverflow post</a> that the <code>artifacts.reports.dotenv</code> doesn't work with the <code>dependencies</code> or the <code>needs</code> keywords.</p>
  <p>Removing <code>dependencies</code> doesn't work. Using <code>needs</code> only doesn't work either. Using both is not allowed.</p>
  <p>Does anyone know a way how to get this to work? I feel like this is the way it should work.</p>
  <h2>Getting the artifacts as a file</h2>
  <p>This answer of the stackoverflow post <a href="https://stackoverflow.com/a/65878853/5934316">Gitlab ci cd removes artifact for merge requests</a> suggests to use the <code>build.env</code> as a normal file. I also tried this. The (relevant) <code>yml</code> is the following:</p>
  <pre class="lang-yaml prettyprint-override"><code>building:
      # ...
      artifacts:
          paths:
              - build.env
  
  deploying:
      # ...
      before_script:
          - source build.env
  </code></pre>
  <p>The result is the same as above. The <code>build.env</code> gets removed. Then the <code>source build.env</code> command fails because <code>build.env</code> does not exist. (Doesn't matter if <code>build.env</code> is in the <code>.gitignore</code> or not, tested both)</p>
  <h2>Getting the artifacts from the API</h2>
  <p>I also found the answer of the stackoverflow post <a href="https://stackoverflow.com/a/66344557/5934316">Use artifacts from merge request job in GitLab CI</a> which suggests to use the API together with <code>$CI_JOB_TOKEN</code>. But since I need the artifacts in a non-merge-request pipeline, I cannot use the suggested <code>CI_MERGE_REQUEST_REF_PATH</code>.</p>
  <p>I tried to use <code>$CI_COMMIT_REF_NAME</code>. The (important section of the) <code>yml</code> is then:</p>
  <pre class="lang-yaml prettyprint-override"><code>deploying:
      # ...
      script:
          - url=$CI_API_V4_URL/projects/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=building
          - echo &quot;Downloading $url&quot;
          - 'curl --header &quot;JOB-TOKEN: ${CI_JOB_TOKEN}&quot; --output $url'
          # ...
  </code></pre>
  <p>But this the API request gets rejected with &quot;404 Not Found&quot;. Since <a href="https://docs.gitlab.com/ee/api/job_artifacts.html#download-a-single-artifact-file-from-specific-tag-or-branch" rel="noreferrer">commit SHAs are not supported</a>, <code>$CI_COMMIT_BEFORE_SHA</code> or <code>$CI_COMMIT_SHA</code> do not work either.</p>
  <h2>Using <code>needs</code></h2>
  <p><em>Update</em>: I found the section <a href="https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-between-pipelines-in-the-same-project" rel="noreferrer">Artifact downloads between pipelines in the same project</a> in the gitlab docs which is exactly what I want. But: I can't get it to work.</p>
  <p>The <code>yml</code> looks like the following after more less copying from the docs:</p>
  <pre class="lang-yaml prettyprint-override"><code>building:
      # ...
      artifacts:
          paths:
              - version
          expire_in: never
  
  deploying:
      # ...
      needs:
          - project: $CI_PROJECT_PATH
            job: building
            ref: staging # building runs on staging branch, main doesn't work either
            artifacts: true
  </code></pre>
  <p>Now the <code>deploying</code> job instantly fails and I get the following error banner:</p>
  <p><a href="https://i.stack.imgur.com/LCXJr.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/LCXJr.jpg" alt="This job depends on other jobs with expired/erased artifacts:
  Please refer to https://docs.gitlab.com/ee/ci/yaml/README.html#dependencies" /></a></p>
  <p>I tried to set <code>artifacts.expire_in = never</code> (as shown) but I still get the same error. Also in <em>Settings</em> &gt; <em>CI/CD</em> &gt; <em>Artifacts</em> &quot;Keep artifacts from most recent successful jobs&quot; is selected. So the artifact should be present. What did I miss here? This should work according to the docs!</p>
  <hr />
  <p>I hope somebody can help me on getting the <code>$BUILD_VERSION</code> to the <code>deploying</code> job. If there are other ways than the ones I've tried, I'm very happy to hear them. Thanks in advance.</p>
  <hr />
  <p>The example <code>.gitlab-ci.yml</code>:</p>
  <pre class="lang-yaml prettyprint-override"><code>stages:
      - staging
      - deploy
  
  building:
      tags: 
          - docker
      image: bash
      stage: staging
      rules:
          - if: ($CI_PIPELINE_SOURCE == &quot;merge_request_event&quot;) &amp;&amp; $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == &quot;staging&quot;
            when: always
          - when: never
      script:
          - echo &quot;BUILD_VERSION=1.2.3&quot; &gt; build.env
      artifacts:
          reports:
              dotenv: build.env
      environment:
          name: Example
          url: https://example.com
  
  deploying:
      tags: 
          - docker
      image: bash
      stage: deploy
      rules:
          - if: $CI_COMMIT_BRANCH == &quot;staging&quot;
            when: always
          - when: never
      dependencies:
          - building
      script:
          echo $BUILD_VERSION
  </code></pre>