title: Running Spark job in dataproc with docker-compose
body: <p>I have created and tested successfully a project which runs locally with docker-compose using sample data. Bash file to run the whole pipeline job:</p>
  
  <pre><code>cp -r ../data . # transfer data used for job
  
  docker-compose up --build -d
  docker exec project_master_1 bin/spark-submit \
                              --py-files /project/utils.py\
                              /project/main.py
  docker cp project_master_1:/project/data/output/. ../project/output #some tranfer of generated files
  docker exec project_master_1 rm -rf /project/data
  
  docker-compose down
  </code></pre>
  
  <p>My docker-compose file is identical to the one from this: <a href="https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml" rel="nofollow noreferrer">github link</a></p>
  
  <p>So what I do is to create a dataproc cluster with initialization action to install docker:</p>
  
  <pre><code>gs://dataproc-initialization-actions/docker/docker.sh
  </code></pre>
  
  <p>Next:</p>
  
  <ul>
  <li>I enter with ssh the master </li>
  <li>git clone my project(this can be done also in initialization action(???))</li>
  <li>install docker-compose </li>
  <li>run the bash file.</li>
  </ul>
  
  <p>It starts running but it seems that it runs only in master and not in the workers. So what do I do wrong here? Any insight is useful.</p>