title: How to use while loop in gitlab-ci script section
body: <p>I'm trying to iterate over url entries in a file and use each file as an input for a crawler tool. It's result should be written to a file.</p>
  
  <p>here is the gitlab-ci.yml file:</p>
  
  <pre><code>stages:
    - test
  test:
    stage: test
  
    tags:
      - shell-docker
    script:
      - wget https://github.com/FaKeller/sireg/releases/download/v0.3.1/sireg-linux
      - chmod 775 sireg-linux
      - mkdir output
      - ls -alF
      - while read line; do
           echo $line;
           ./sireg-linux exec --loader-sitemap-sitemap \"$line\" &gt;&gt; ./output/${line##*/}_out.txt;
        done &lt; sitemap-index
      - ls -alF output
    artifacts:
      paths:
        - output/*
      expire_in: 1 hrs
  
  </code></pre>
  
  <p>and here is the sitemap-index file (only one entry):</p>
  
  <pre><code>http://example.com/sitemap.xml
  </code></pre>
  
  <p>both files are in the same directory. I expect a file sitemap.xml_out.txt to be written into the output folder(also the same directory). I am pretty sure the ./sireg-linux script does not execute because it usually takes few minutes to complete (tested locally).</p>
  
  <p>the output of the stage looks like this:</p>
  
  <pre><code>2020-04-02 18:22:21 (4,26 MB/s) - »sireg-linux« saved [62566347/62566347]
  
  $ chmod 775 sireg-linux
  $ mkdir output
  $ ls -alF
  total 61128
  drwxrwxr-x  4 gitlab-runner gitlab-runner     4096 Apr  2 18:22 ./
  drwxrwxr-x 10 gitlab-runner gitlab-runner     4096 Apr  2 15:46 ../
  drwxrwxr-x  5 gitlab-runner gitlab-runner     4096 Apr  2 18:22 .git/
  -rw-rw-r--  1 gitlab-runner gitlab-runner      512 Apr  2 18:22 .gitlab-ci.yml
  drwxrwxr-x  2 gitlab-runner gitlab-runner     4096 Apr  2 18:22 output/
  -rw-rw-r--  1 gitlab-runner gitlab-runner       30 Apr  2 15:46 README.md
  -rwxrwxr-x  1 gitlab-runner gitlab-runner 62566347 Nov 11  2017 sireg-linux*
  -rw-rw-r--  1 gitlab-runner gitlab-runner       55 Apr  2 18:08 sitemap-index
  $ while read line; do echo $line; ./sireg-linux **exec** --loader-sitemap-sitemap \"$line\" &gt;&gt; 
  ./output/${line##*/}_out.txt; done &lt; sitemap-index
  $ ls -alF output
  total 8
  drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr  2 18:22 ./
  drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr  2 18:22 ../
  Uploading artifacts...
  Runtime platform                                    arch=amd64 os=linux pid=23813 revision=1f513601 version=11.10.1
  WARNING: output/*: no matching files               
  ERROR: No files to upload                          
  Job succeeded
  
  </code></pre>
  
  <p><strong>update</strong></p>
  
  <p>tried to move all steps into a separate script but that did not work either.</p>
  
  <p><strong>update 2</strong></p>
  
  <p>forgot to add exec in the command:</p>
  
  <pre><code>./sireg-linux exec --loader-sitemap-sitemap \"$line\" &gt;&gt; 
  ./output/${line##*/}_out.txt;
  </code></pre>
  
  <p>unfortunately it didn't help.</p>
  
  <p>what can I do to make it working?</p>