title: Passing Google service account credentials to Docker
body: <p>My use case is a little different than others with this problem, so a little up-front description:</p>
  
  <p>I am working on Google Cloud and have a "dockerized" Django app.  Part of the app depends on using gsutil for moving files to/from a Google Storage bucket.  For various reasons, we do not want to use Google Container Engine to manage our containers.  Rather, we would like to scale horizontally by starting additional Google Compute VMs which will, in turn, run this Docker container.  Similar to <a href="https://cloud.google.com/python/tutorials/bookshelf-on-compute-engine" rel="noreferrer">https://cloud.google.com/python/tutorials/bookshelf-on-compute-engine</a> except we will use a container rather than pulling a git repository.</p>
  
  <p>The VM's will be built from a basic Debian image, and the startup and installation of dependencies (e.g. Docker itself) will be orchestrated with a startup script (e.g. <code>gcloud compute instances create some-instance --metadata-from-file startup-script=/path/to/startup.sh</code>).</p>
  
  <p>If I manually create a VM, elevate with <code>sudo -s</code>, run <code>gsutil config -f</code> (which creates a credential file at /root/.boto) and then run my docker container (see Dockerfile below) with </p>
  
  <p><code>docker run -v /root/.boto:/root/.boto username/gs gsutil ls gs://my-test-bucket</code> </p>
  
  <p>then it works.  However, that requires my interaction to create the boto file. </p>
  
  <p><strong>My question is</strong>:  How can I pass the default service credentials to the Docker container that will be starting in that new VM?  </p>
  
  <p>gsutil works out of the box on even a "fresh" Debian VM since it is using the default compute engine credentials that all VMs are loaded with.  Is there a way to use those credentials and pass them to the docker container?  After the first call to gsutil on a fresh VM, I've noticed that it creates ~/.gsutil and ~/.config folders.    Unfortunately, mounting both of those in Docker with</p>
  
  <p><code>docker run -v ~/.config/:/root/.config -v ~/.gsutil:/root/.gsutil username/gs gsutil ls gs://my-test-bucket</code> </p>
  
  <p>does not fix my problem.  It tells me:</p>
  
  <p><code>ServiceException: 401 Anonymous users does not have storage.objects.list access to bucket my-test-bucket.</code></p>
  
  <p>A minimal gsutil Dockerfile (not mine):</p>
  
  <pre><code>FROM alpine
  #install deps and install gsutil
  RUN apk add --update \
      python \
      py-pip \
      py-cffi \
      py-cryptography \
    &amp;&amp; pip install --upgrade pip \
    &amp;&amp; apk add --virtual build-deps \
      gcc \
      libffi-dev \
      python-dev \
      linux-headers \
      musl-dev \
      openssl-dev \
    &amp;&amp; pip install gsutil \
    &amp;&amp; apk del build-deps \
    &amp;&amp; rm -rf /var/cache/apk/*
  CMD ["gsutil"]
  </code></pre>
  
  <p><strong>Addition:  a workaround:</strong></p>
  
  <p>I have since solved my issue, but it is quite roundabout so I'm <strong>still interested in a simpler way</strong>, if possible.  All the details are below:</p>
  
  <p>First, a description:
  I first created a service account in the web console.  I then save the JSON keyfile (call it credentials.json) into a storage bucket.  In the startup script for the GCE VM, I copy that keyfile to the local filesystem (<code>gsutil cp gs://&lt;bucket&gt;/credentials.json /gs_credentials/</code>).  I then start my docker container, mounting that local directory.  Then, as the docker container starts, it runs a script that authenticates the credentials.json (which creates a .boto file inside the docker), export BOTO_PATH=, and finally I can perform gsutil operations in the Docker container.</p>
  
  <p>Here are the files for a small working example:</p>
  
  <p><strong>Dockerfile:</strong></p>
  
  <pre><code>FROM alpine
  #install deps and install gsutil
  RUN apk add --update \
      python \
      py-pip \
      py-cffi \
      py-cryptography \
      bash \
      curl \
    &amp;&amp; pip install --upgrade pip \
    &amp;&amp; apk add --virtual build-deps \
      gcc \
      libffi-dev \
      python-dev \
      linux-headers \
      musl-dev \
      openssl-dev \
    &amp;&amp; pip install gsutil \
    &amp;&amp; apk del build-deps \
    &amp;&amp; rm -rf /var/cache/apk/*
  
  # install the gcloud SDK- 
  # this allows us to use gcloud auth inside the container
  RUN curl -sSL https://sdk.cloud.google.com &gt; /tmp/gcl \
        &amp;&amp; bash /tmp/gcl --install-dir=~/gcloud --disable-prompts
  
  RUN mkdir /startup
  ADD gsutil_docker_startup.sh /startup/gsutil_docker_startup.sh
  ADD get_account_name.py /startup/get_account_name.py
  
  ENTRYPOINT ["/startup/gsutil_docker_startup.sh"]
  </code></pre>
  
  <p><strong>gsutil_docker_startup.sh:</strong>  Takes a single argument, which is the path to a JSON-format service account credentials file.  The file exists because the directory on the host machine was mounted in the container.</p>
  
  <pre><code>#!/bin/bash
  
  CRED_FILE_PATH=$1
  
  mkdir /results
  
  # List the bucket, see that it gives a "ServiceException:401"
  gsutil ls gs://&lt;input bucket&gt; &gt; /results/before.txt
  
  # authenticate the credentials- this creates a .boto file:
  /root/gcloud/google-cloud-sdk/bin/gcloud auth activate-service-account --key-file=$CRED_FILE_PATH
  
  # need to extract the service account which is like:
  # &lt;service acct ID&gt;@&lt;google project&gt;.iam.gserviceaccount.com"
  SERVICE_ACCOUNT=$(python /startup/get_account_name.py $CRED_FILE_PATH)
  
  # with that service account, we can locate the .boto file:
  export BOTO_PATH=/root/.config/gcloud/legacy_credentials/$SERVICE_ACCOUNT/.boto
  
  # List the bucket and copy the file to an output bucket for good measure
  gsutil ls gs://&lt;input bucket&gt; &gt; /results/after.txt
  gsutil cp /results/*.txt gs://&lt;output bucket&gt;/
  </code></pre>
  
  <p><strong>get_account_name.py:</strong></p>
  
  <pre><code>import json
  import sys
  j = json.load(open(sys.argv[1]))
  sys.stdout.write(j['client_email'])
  </code></pre>
  
  <p>Then, the GCE startup script (executed automatically as the VM is started) is:</p>
  
  <pre><code>#!/bin/bash
  
  # &lt;SNIP&gt;
  # Install docker, other dependencies
  # &lt;/SNIP&gt;
  
  # pull docker image
  docker pull userName/containerName
  
  # get credential file:
  mkdir /cloud_credentials
  gsutil cp gs://&lt;bucket&gt;/credentials.json /cloud_credentials/creds.json
  
  # run container
  # mount the host machine directory where the credentials were saved.
  # Note that the container expects a single arg, 
  # which is the path to the credential file IN THE CONTAINER
  docker run -v /cloud_credentials:/cloud_credentials \ 
      userName/containerName /cloud_credentials/creds.json
  </code></pre>