title: In Jenkins how to use Docker image with cross compilation environment
body: <p>I'm currently trying to setup a Jenkins build Pipeline and I'm struggling including a docker container. My main problem is the way Jenkins sends build steps to the running container.</p>
  
  <h2>Background</h2>
  
  <p>I'm building a cross compilation project. This happens in a linux, set up for cross compilation. All necessary stuff is provided by a SDK including the appropriate compilers, headers, libs, etc. </p>
  
  <p>To setup everything, it also includes an environment file, which needs to be sourced by the current shell. This script adds necessary info, like adding environment variables (e.g. adjusting $PATH) or aliases.</p>
  
  <p>For the Jenkins builds I'm creating a docker image containing the SDK. The image is working fine, and I'm able to build the project when I'm running the container. But it doesn't work on Jenkins. After some evaluation I found out that the problem lies in the way Jenkins uses the container. </p>
  
  <h2>The problem</h2>
  
  <p>It seems Jenkins starts the container for a docker image in the background via <code>docker run -t -d imagename cat</code>. Then all subsequent build steps are sent using <code>docker exec</code>. This is usually fine. In my case I get errors, as all the executables (e.g. cmake) are not found. The executables are not installed in the regular system root but in the SDK folder.<br>
  I added a <code>source</code> command to the docker image's entrypoint script. The <code>source</code> command sources the SDK environment. As the entrypoint is only executed during <code>run</code> and not with <code>exec</code>, all build steps produce errors because the <code>$PATH</code> variable is not set properly.</p>
  
  <p>This is a stripped down Jenkinsfile:</p>
  
  <pre><code>pipeline {
      agent {
          docker {
              label 'docker'
              image 'test:01'
          }
      }
      stages {
          stage('build') {
              steps {
                  cmakeBuild  installation: "InSearchPath",
                              generator: "Unix Makefile",
                              buildDir: 'build',
                              sourceDir: 'source',
                              steps: [
                                  [args: 'all install']
                              ]
              }
           }
      }
  }
  </code></pre>
  
  <p>Which yields following log output:</p>
  
  <pre><code>...
  [Pipeline] withDockerContainer
  $ docker run -t -d -u 1078:1001 -w /jenkins/workspace/project -v /jenkins/workspace/project:/jenkins/workspace/project:rw,z -v /jenkins/workspace/project@tmp:/jenkins/workspace/project@tmp:rw,z  test:01 cat
  $ docker top ff1b4ee4e929b83e5741ef7db1e57688f7f996ca3a10a00ae4c5426cf108cb2a -eo pid,comm
  ...
  ...
  ...
  [Pipeline] cmakeBuild
  [build] $ docker exec --workdir /jenkins/workspace/project/build ff1b4ee4e929b83e5741ef7db1e57688f7f996ca3a10a00ae4c5426cf108cb2a cmake -G "Unix Makefile" /jenkins/workspace/project/source
  OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"cmake\": executable file not found in $PATH": unknown
  ...
  </code></pre>
  
  <h2>The reproduce image</h2>
  
  <p>To find out more about the behavior I created a small docker image which illustrates the problem. </p>
  
  <p>The Dockerfile:</p>
  
  <pre><code>FROM debian:jessie
  
  # Add entrypoint script which sources envvars file
  RUN echo "#!/bin/bash\n\n\
            . /envvars \n\n\
            exec \"\$@\"\
            " &gt;/entrypoint.sh \
      &amp;&amp; chmod ugo+rx /entrypoint.sh
  
  # Add a script foo.sh which returns the content of $PATH
  RUN mkdir -p /home/test/script \
      &amp;&amp; echo "#!/bin/bash\n\
              echo \$PATH \n\
              " &gt; /home/test/script/foo.sh \
      &amp;&amp; chmod ugo+rx /home/test/script/foo.sh
  
  # Add the directory containing the foo.sh script to $PATH
  RUN echo "export PATH=/home/test/script:$PATH \n" &gt; /envvars
  
  ENTRYPOINT ["/entrypoint.sh"]
  CMD ["/bin/bash"]
  </code></pre>
  
  <p>You can build this image:</p>
  
  <pre><code>$&gt; docker build -t "test" .
  Sending build context to Docker daemon   55.3kB
  Step 1/6 : FROM debian:jessie
   ---&gt; c9d6adb06e4d
  Step 2/6 : RUN echo "#!/bin/bash\n\n          . /envvars \n\n          exec \"\$@\"          " &gt;/entrypoint.sh     &amp;&amp; chmod ugo+rx /entrypoint.sh
   ---&gt; Using cache
   ---&gt; 648affce60a6
  Step 3/6 : RUN mkdir -p /home/test/script     &amp;&amp; echo "#!/bin/bash\n            echo \$PATH \n            " &gt; /home/test/script/foo.sh     &amp;&amp; chmod ugo+rx /home/test/script/foo.sh
   ---&gt; Running in d13391c77668
  Removing intermediate container d13391c77668
   ---&gt; 7d88c20f8673
  Step 4/6 : RUN echo "export PATH=/home/test/script:$PATH \n" &gt; /envvars
   ---&gt; Running in 55ac66323579
  Removing intermediate container 55ac66323579
   ---&gt; 35e081186bfa
  Step 5/6 : ENTRYPOINT ["/entrypoint.sh"]
   ---&gt; Running in 53154fed036f
  Removing intermediate container 53154fed036f
   ---&gt; 4ca347cbe757
  Step 6/6 : CMD ["/bin/bash"]
   ---&gt; Running in a598aefa837b
  Removing intermediate container a598aefa837b
   ---&gt; 78fc760bb9db
  Successfully built 78fc760bb9db
  Successfully tagged test:latest
  </code></pre>
  
  <p>Once the image is built, you can run it interactively. So you are inside the container, in a bash shell, and can call <code>foo.sh</code> directly, because <code>/home/test/script</code> is part of <code>$PATH</code>:</p>
  
  <pre><code>$&gt; docker run --rm -ti test
  root@42529686fe22:/# foo.sh
  /home/test/script:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  root@42529686fe22:/# exit
  </code></pre>
  
  <p>You can even override <code>CMD</code> setting from Dockerfile and call foo.sh from outside the container using <code>docker run</code>:</p>
  
  <pre><code>$&gt; docker run --rm -t test foo.sh
  /home/test/script:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  </code></pre>
  
  <p><strong>But</strong> when you start the container in the background (as it is done in Jenkins), you can't call <code>foo.sh</code> any more:</p>
  
  <pre><code>$&gt; docker run --rm --name test -td test cat
  0ecf3efecc54d33b73aaab6b4a1e191056c55597b3fe558ff7a7a6f93db2b695
  
  $&gt; docker exec test foo.sh
  OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"foo.sh\": executable file not found in $PATH": unknown
  </code></pre>
  
  <p>Though calling <code>foo.sh</code> using the absolute path works. The result shows that <code>$PATH</code> does not contain the script directory <code>/home/test/script</code>:</p>
  
  <pre><code>$&gt; docker exec test /home/test/script/foo.sh
  /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  </code></pre>
  
  <h2>The question</h2>
  
  <p>Do you know of any possibility I could configure either the docker image or the Jenkins pipeline to make my scenario work?</p>
  
  <p>One possible solution I could think of, is wrapping all build steps in shell scripts, which itself source the environment file. But I would prefer to use the plugins provided by Jenkins and write a clean Jenkinsfile (i.e. not everything is hidden in shell scripts)</p>
  
  <p>Another solution could be to install the SDK directly in the system root. But I'm not sure of side effects. Furthermore I still need specific environment variables set for the build step.</p>