title: EC2 instance can't access amazon-linux repos (eg amazon-linux-extras install docker) through s3 gateway endpoint
body: <p>I'm having s3 endpoint grief. When my instances initialize they can not install docker. Details:</p>
  <p>I have ASG instances sitting in a VPC with pub and private subnets. Appropriate routing and EIP/NAT is all stitched up.Instances in private subnets have outbouond 0.0.0.0/0 routed to NAT in respective public subnets. NACLs for public subnet allow internet traffic in and out, the NACLs around private subnets allow traffic from public subnets in and out, traffic out to the internet (and traffic from s3 cidrs in and out). I want it pretty locked down.</p>
  <ul>
  <li>I have DNS and hostnames enabled in my VPC</li>
  <li>I understand NACLs are stateless and have enabled IN and OUTBOUND rules for s3 amazon IP cidr blocks on ephemeral port ranges (yes I have also enabled traffic between pub and private subnets)</li>
  <li>yes I have checked a route was provisioned for my s3 endpoint in my private route tables</li>
  <li>yes I know for sure it is the s3 endpoint causing me grief and not another blunder -&gt; when I delete it and open up my NACLs I can yum update and install docker (as expected) I am not looking for suggestions that require opening up my NACLs, I'm using a VPC gateway endpiont because I want to keep things locked down in the private subnets. I mention this because similar discussions seem to say 'I opened 0.0.0.0/0 on all ports and now x works'</li>
  <li>Should I just bake an AMI with docker installed? That's what I'll do if I can't resolve this. I really wanted to set up my networking so everything is nicely locked down and feel like it should be pretty straight forward utilizing endpoints. Largely this is a networking exercise so I would rather not do this because it avoids solving and understanding the problem.</li>
  <li>I know my other VPC endpoints work perfectly -&gt; Auto-scaling service interface endpoint is performing (I can see it scaling down instances as per the policy), SSM interface endpoint allowing me to use session manager, and ECR endpoint(s) are working in conjunction with s3 gateway endpoint (s3 gateway endpoint is required because image layers are in s3) -&gt; I know this works because if I open up NACLS and delete my s3 endpoint and install docker, then lock everything down again, bring back my s3 gatewayendpoint I can successfully pull my ECR images. SO the s3 gateway endpoint is fine for accessing ecr image layers, but not amazon-linux-extra repos.</li>
  <li>SGs attached to instances are not the problem (instances have default outbound rule)</li>
  <li>I have tried adding increasingly generous policies to my s3 endpoint as I have seen in this 7 year old thread and thought this had to do the trick (yes I subbed my region correctly)</li>
  <li>I strongly feel the solution lies with the s3 gateway policy as discussed in this thread, however have had little luck with my increasingly desperate policies.</li>
  </ul>
  <p><a href="https://stackoverflow.com/questions/20822021/amazon-ec2-instance-cant-update-or-use-yum">Amazon EC2 instance can&#39;t update or use yum</a></p>
  <p>another s3 struggle with resolution:</p>
  <p><a href="https://blog.saieva.com/2020/08/17/aws-s3-endpoint-gateway-access-for-linux-2-amis-resolving-http-403-forbidden-error/" rel="nofollow noreferrer">https://blog.saieva.com/2020/08/17/aws-s3-endpoint-gateway-access-for-linux-2-amis-resolving-http-403-forbidden-error/</a></p>
  <p>I have tried:</p>
  <pre><code>  S3Endpoint:
  Type: 'AWS::EC2::VPCEndpoint'
  Properties:
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal: '*'
          Action:
            - 's3:GetObject'
          Resource: 
            - 'arn:aws:s3:::prod-ap-southeast-2-starport-layer-bucket/*'
            - 'arn:aws:s3:::packages.*.amazonaws.com/*'
            - 'arn:aws:s3:::repo.*.amazonaws.com/*'
            - 'arn:aws:s3:::amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::amazonlinux.*.amazonaws.com/*'
            - 'arn:aws:s3:::*.amazonaws.com'
            - 'arn:aws:s3:::*.amazonaws.com/*'
            - 'arn:aws:s3:::*.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::*.ap-southeast-2.amazonaws.com/'
            - 'arn:aws:s3:::*repos.ap-southeast-2-.amazonaws.com'
            - 'arn:aws:s3:::*repos.ap-southeast-2.amazonaws.com/*'
            - 'arn:aws:s3:::repo.ap-southeast-2-.amazonaws.com'
            - 'arn:aws:s3:::repo.ap-southeast-2.amazonaws.com/*'
    RouteTableIds:
      - !Ref PrivateRouteTableA
      - !Ref PrivateRouteTableB   
    ServiceName: !Sub 'com.amazonaws.${AWS::Region}.s3'
    VpcId: !Ref BasicVpc
    VpcEndpointType: Gateway
  </code></pre>
  <p>(as you can see, very desperate) The first rule is required for the ECR interface endpoints to pull the image layers from s3, all of the others are attempts to reach amazon-linux-extras repos.</p>
  <p>Below is the behavior happening on initialization I have recreated by connecting with session manager using SSM endpoint:</p>
  <p><a href="https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/" rel="nofollow noreferrer">https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/</a></p>
  <p>I can not yum install or update</p>
  <pre><code>root@ip-10-0-3-120 bin]# yum install docker -y
  </code></pre>
  <p>Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
  Could not retrieve mirrorlist <a href="https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/core/latest/x86_64/mirror.list" rel="nofollow noreferrer">https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/core/latest/x86_64/mirror.list</a> error was
  14: HTTPS Error 403 - Forbidden</p>
  <p>One of the configured repositories failed (Unknown),
  and yum doesn't have enough cached data to continue. At this point the only
  safe thing yum can do is fail. There are a few ways to work &quot;fix&quot; this:</p>
  <pre><code> 1. Contact the upstream for the repository and get them to fix the problem.
  
   2. Reconfigure the baseurl/etc. for the repository, to point to a working
      upstream. This is most often useful if you are using a newer
      distribution release than is supported by the repository (and the
      packages for the previous distribution release still work).
  
   3. Run the command with the repository temporarily disabled
          yum --disablerepo=&lt;repoid&gt; ...
  
   4. Disable the repository permanently, so yum won't use it by default. Yum
      will then just ignore the repository until you permanently enable it
      again or use --enablerepo for temporary usage:
  
          yum-config-manager --disable &lt;repoid&gt;
      or
          subscription-manager repos --disable=&lt;repoid&gt;
  
   5. Configure the failing repository to be skipped, if it is unavailable.
      Note that yum will try to contact the repo. when it runs most commands,
      so will have to try and fail each time (and thus. yum will be be much
      slower). If it is a very temporary problem though, this is often a nice
      compromise:
  
          yum-config-manager --save --setopt=&lt;repoid&gt;.skip_if_unavailable=true
  </code></pre>
  <p>Cannot find a valid baseurl for repo: amzn2-core/2/x86_64</p>
  <p>and can not:</p>
  <pre><code>amazon-linux-extras install docker
  
  Catalog is not reachable. Try again later.
  </code></pre>
  <p>catalogs at <a href="https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64-v2.json" rel="nofollow noreferrer">https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64-v2.json</a>, <a href="https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64.json" rel="nofollow noreferrer">https://amazonlinux-2-repos-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/2/extras-catalog-x86_64.json</a>
  Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/site-packages/amazon_linux_extras/software_catalog.py&quot;, line 131, in fetch_new_catalog
  request = urlopen(url)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 154, in urlopen
  return opener.open(url, data, timeout)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 435, in open
  response = meth(req, response)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 548, in http_response
  'http', request, response, code, msg, hdrs)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 473, in error
  return self._call_chain(*args)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 407, in _call_chain
  result = func(*args)
  File &quot;/usr/lib64/python2.7/urllib2.py&quot;, line 556, in http_error_default
  raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  HTTPError: HTTP Error 403: Forbidden</p>
  <p>Any gotchas I've missed? I'm very stuck here. I am familiar with basic VPC networking, NACLs and VPC endpoints (the ones I've used at least), I have followed the trouble-shooting (although I already had everything set-up as outlined).</p>
  <p>I feel the s3 policy is the problem here OR the mirror list.
  Many thanks if you bothered to read all that!
  Thoughts?</p>