title: How to get Terraform providers on GitLab CI runners in an air-gapped environment
body: <p>I am running a Gitlab CI runner on an Azure Red Hat Linux 7.9 VM which is air-gapped and cannot communicate with the outside world due to our network restrictions. In the .gitlab-ci.yml file of my GitLab pipeline, I run some Terraform commands including <code>Terraform init</code> which subsequently attempts to pull down the Terraform provider plugins from the default Hashicorp location.</p>
  <p>Obviously, with the prevailing network restrictions, the aforementioned Terraform command fails and subsequently the entire pipeline job fails. From what I've now established and bearing in mind our particular network setup and circumstances, I probably have two options:</p>
  <p><strong>A)</strong> Configure my GitLab pipeline with a Proxy to allow access to sites like registry.terraform.io. I have been provided a proxy address by the way, but I'm not entirely sure how to get this configured in GitLab. Will therefore appreciate some assistance with this.</p>
  <p><strong>B)</strong> It also appears I will be able to force my Terraform configuration to reference the required provider plugins locally or on an internal share. Again, any advice on how to set this up would be greatly appreciated, having spent hours poring over endless online material on both options without any success.</p>
  <p><strong>[[ Case Update after recommendations by @sytech ]]</strong></p>
  <p>I discarded the proxy option entirely, due mainly to some resource challenges within my organisation and not at all to do with the proposed solution itself. I therefore went with the option to set up a provider mirror residing on my file system and this is how I went about it.....but still with no success.</p>
  <p><strong>My Implementation Summary</strong></p>
  <p><strong>1.</strong> I uploaded the two required Terraform providers into our Artifactory repo, with each provider content displayed in the image below.
  <a href="https://i.stack.imgur.com/jGFQ1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jGFQ1.png" alt="Terraform providers in our Artifactory repo" /></a></p>
  <p><strong>2.</strong> In my .gitlab-ci.yml file, I create the following directories:</p>
  <pre><code>*/root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread 
  /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm*
  </code></pre>
  <p>I then updated my terraform providers.tf configuration file by amending both provider source locations accordingly (see below screenshot).</p>
  <p><strong>3.</strong> Next, I download each provider (excluding the *.json files) from the Artifactory repo to the GitLab runner local file system and unzip them to their respective directories, created in Step #2 above. Finally, I perform a clean-up by removing the respective provider zip files that were downloaded from Artifactory.</p>
  <p><strong>4.</strong> I then switch to the <code>/root/.terraform.d/plugins/registry.terraform.io/hashicorp</code> directory and run a Linux recursive command to verify both providers are present in the expected directories.</p>
  <p><strong>5.</strong> Finally, I switch to the defined terraform root directory, run the <code>ls -a</code> command to verify that my terraform configuration files are all present. I then run the <code>terraform init</code> and <code>terraform validate</code> commands which result in the subsequent error displayed in the last of the images displayed below.</p>
  <p><strong>[[ Required Providers Block in providers.tf file ]]</strong></p>
  <pre><code>terraform {
    required_providers {
      azurerm = {
        source = &quot;registry.terraform.io/hashicorp/azurerm/terraform-provider-azurerm_v3.19.1_x5&quot;
        version = &quot;3.19.1&quot; 
      }
  
      azuread = {
        source = &quot;registry.terraform.io/hashicorp/azuread/terraform-provider-azuread_v2.27.0_x5&quot;
        version = &quot;2.27.0&quot;
      }
    }
  }
  </code></pre>
  <p><strong>[[ Snippet from my .gitlab-ci.yml file ]]</strong></p>
  <pre><code>default:
    image: 
      name: xxx-local.artifactory.ourdomain:x.x.x
      entrypoint: [&quot;&quot;]
  
  variables:
    TF_ROOT: ${CI_PROJECT_DIR}
    
  stages:
    - validate
    - plan
    - destroy
  
  download-plugins:
    stage: validate 
    script:
      - git config --global url.&quot;https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ourdomain&quot;.insteadOf https://gitlab.ourdomain
  
      # Download and copy azuread provider to the gitlab runner filesystem from our Artifactory store
      - mkdir -p /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread      
      - curl -o /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip https://artifactory.ourdomain/artifactory/local/terraform/providers/registry.terraform.io/hashicorp/azuread/terraform-provider-azuread_2.27.0_linux_amd64.zip
      - unzip /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip -d /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread
      - rm /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azuread/azuread.zip
  
      # Download and copy azurerm provider to the gitlab runner filesystem from our Artifactory store
      - mkdir -p /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm      
      - curl -o /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip https://artifactory.ourdomain/artifactory/local/terraform/providers/registry.terraform.io/hashicorp/azurerm/terraform-provider-azurerm_3.19.1_linux_amd64.zip
      - unzip /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip -d /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm
      - rm /root/.terraform.d/plugins/registry.terraform.io/hashicorp/azurerm/azurerm.zip
  
      - cd /root/.terraform.d/plugins/registry.terraform.io/hashicorp
      - echo &quot;Display contents of the present working directory:&quot; `pwd`
      - ls -a -R
  
      - echo &quot;Switch to the $TF_ROOT directory to verify file content and then run terraform commands&quot;
      - cd $TF_ROOT
      - ls -a
      - terraform init
      - terraform validate
  
    tags:
      - &quot;AzureVMRunner&quot;
     
  </code></pre>
  <p><a href="https://i.stack.imgur.com/F9km6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/F9km6.png" alt="Display contents of the present working directory" /></a></p>
  <p><a href="https://i.stack.imgur.com/caL7r.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/caL7r.png" alt="Switch to the $TF_ROOT directory to verify file content and then run terraform commands" /></a></p>
  <p><a href="https://i.stack.imgur.com/TT5ZY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TT5ZY.png" alt="Error displayed after Terraform command(s) are run" /></a></p>