title: EC2 instance role gets 'Unknown' error when attempting aws s3 cp KMS encrypted file
body: <p>I've got an <code>ASG</code> that assigns an <code>IAM Role</code> to each of the instances that join it. Therefore, each instance has the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> environment variables baked-in, which will be used upon instantiation to download and decrypt credentials that are stored in an <code>S3</code> bucket and encrypted using <code>KMS</code> keys.
  <br><br></p>
  
  <p>So I'll have the following components:</p>
  
  <ul>
  <li>An <code>S3</code> bucket called <code>top-secret.myapp.com</code></li>
  <li>All objects in this bucket are encrypted using a <code>KMS</code> key called <code>My-KMS-Key</code></li>
  <li>An <code>IAM</code> instance role with inline policies attached granting it the ability to interact with both the bucket and the <code>KMS</code> key used to encrypt/decrypt the contents of the bucket (see below)</li>
  <li>A <code>user data</code> script that installs the <code>aws-cli</code> upon instantiation and then goes about attempting to download and decrypt an object from the <code>top-secret.myapp.com</code> bucket.
  <br><br></li>
  </ul>
  
  <p><strong>The User Data Script</strong></p>
  
  <p>Upon instantiation, any given instance runs the following script:</p>
  
  <pre><code>#!/bin/bash
  
  apt-get update
  apt-get -y install python-pip
  apt-get -y install awscli
  
  cd /home/ubuntu
  aws s3 cp s3://top-secret.myapp.com/secrets.sh . --region us-east-1
  chmod +x secrets.sh
  . secrets.sh
  shred -u -z -n 27 secrets.sh
  </code></pre>
  
  <p><br></p>
  
  <p><strong>IAM Role Policies</strong></p>
  
  <p>The <code>IAM</code> role for my <code>ASG</code> instances has three policies attached inline:</p>
  
  <pre><code>{
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": "*",
              "Resource": "*"
          }
      ]
  }
  
  
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:ListBucket"
              ],
              "Resource": [
                  "arn:aws:s3:::top-secret.myapp.com"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "s3:List*",
                  "s3:Get*"
              ],
              "Resource": [
                  "arn:aws:s3:::top-secret.myapp.com/secrets.sh"
              ]
          }
      ]
  }
  
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "kms:*"
              ],
              "Resource": "arn:aws:kms:us-east-1:UUID-OF-MY-SECRET-KEY-HERE"
          }
      ]
  }
  </code></pre>
  
  <p>The first policy is essentially a full-root-access policy with no restrictions. Or so I thought, but it doesn't work. So I thought it might be that I need to explicitly apply policies that allow interaction with <code>S3</code> encryption and/or <code>KMS</code>, makes sense.</p>
  
  <p>So I added the second policy that allows the <code>IAM</code> instance role to list the <code>top-secret.myapp.com</code> bucket, and <code>LIST</code> and <code>GET</code> the <code>secrets.sh</code> object within the bucket. But this produced the error illustrated below.</p>
  
  <p><strong>The (Unknown) Error I'm Getting</strong></p>
  
  <pre><code>download failed: s3://top-secret.myapp.com/secrets.sh to ./secrets.sh
  A client error (Unknown) occurred when calling the GetObject operation: Unknown
  </code></pre>
  
  <p>Anyone have any idea what could be causing this error?</p>
  
  <blockquote>
    <p><strong>Note:</strong> This method for transferring encrypted secrets from S3 and decrypting them on-instance works fine using the standard <code>Amazon S3 service master key</code></p>
  </blockquote>