Choosing Your VPC Endpoint Strategy for Amazon S3
AWS Architecture BlogChoosing Your VPC Endpoint Strategy for Amazon S3by Jeff Harman and Gilles-Kuessan Satchivi | on23 JUL 2021| inAmazon Simple Storage Service (S3),Amazon VPC,Architecture,AWS Identity and Access Management (IAM),AWS PrivateLink|Permalink|ShareThis post was co-written with Anusha Dharmalingam, former AWS Solutions Architect.Must yourAmazon Web Services (AWS)application connect toAmazon Simple Storage Service (Amazon S3)buckets? Must the connection scale to accommodate bandwidth demands? AWS offers a mechanism calledVPC endpointto meet these requirements. This blog post provides guidance for selecting the right VPC endpoint type to access Amazon S3. A VPC endpoint enables workloads in anAmazon VPCto connect to supported public AWS services or third-party applications over the AWS network. This approach is used for workloads that shouldnotcommunicate over public networks.When a workload architecture uses VPC endpoints, the application benefits from the scalability, resilience, security, and access controls native to AWS services. Amazon S3 can be accessed using an interface VPC endpoint powered byAWS PrivateLinkor agateway VPC endpoint. To determine the right endpoint for your workloads, we’ll discuss selection criteria to consider based on your requirements.VPC endpoint overviewA VPC endpoint is a virtual scalable networking component you create in a VPC and use as a private entry point to supported AWS services and third-party applications. Currently, two types of VPC endpoints can be used to connect to Amazon S3:interface VPC endpointandgateway VPC endpoint.When you configure an interface VPC endpoint, an elastic network interface (ENI) with a private IP address is deployed in your subnet. AnAmazon EC2instance in the VPC can communicate with an Amazon S3 bucket through the ENI and AWS network. Using the interface endpoint, applications in your on-premises data center can easily query S3 buckets overAWS Direct ConnectorSite-to-Site VPN. Interface endpoint supports a growing list of AWS services. Consult our documentation to findAWS services compatible with interface endpointspowered by AWS PrivateLink.Gateway VPC endpointsuse prefix lists as the IP route target in a VPC route table. This routes traffic privately to Amazon S3 orAmazon DynamoDB. An EC2 instance in a VPC without internet access can still directly read from and/or write to an Amazon S3 bucket. Amazon DynamoDB and Amazon S3 are the services currently accessible via gateway endpoints.Your internal security policies may have strict rules against communication between your VPC and the internet. To maintain compliance with these policies, you can use VPC endpoint to connect to AWS public services like Amazon S3. To control user or application access to the VPC endpoint and the resources it supports, you can use anAWS Identity and Access Management (AWS IAM)resource policy. This will separately secure the VPC endpoint and accessible resources.Selecting gateway or interface VPC endpointsWith both interface endpoint and gateway endpoint available for Amazon S3, here are some factors to consider as you choose one strategy over the other.Cost:Gateway endpoints for S3 are offered at no cost and the routes are managed through route tables. Interface endpoints are priced at $0.01/per AZ/per hour. Cost depends on the Region, checkcurrent pricing. Data transferred through the interface endpoint is charged at $0.01/per GB (depending on Region).Access pattern:S3 access through gateway endpoints is supported only for resources in a specific VPC to which the endpoint is associated. S3 gateway endpoints do not currently support access from resources in a different Region, different VPC, or from an on-premises (non-AWS) environment. However, if you’re willing to manage a complex custom architecture, you can use proxies. In all those scenarios, where access is from resources external to VPC, S3 interface endpoints access S3 in a secure way.VPC endpoint architecture:Some customersuse centralized VPC endpoint architecture patterns. This is where the interface endpoints are all managed in a central hub VPC for accessing the service from multiple spoke VPCs. This architecture helps reduce the complexity and maintenance for multiple interface VPC endpoints across different VPCs. When using an S3 interface endpoint, you must consider the amount of network traffic that would flow through your network from spoke VPCs to hub VPC. If the network connectivity between spoke and hub VPCs are set up using transit gateway, or VPC peering, consider thedata processing charges(currently $0.02/GB). If VPC peering is used, there is no charge for data transferred between VPCs in the same Availability Zone. However, data transferred between Availability Zones or between Regions will incurcharges as defined in our documentation.In scenarios where you must access S3 buckets securely from on-premises or from across Regions, we recommend using an interface endpoint. If you chose a gateway endpoint, install a fleet of proxies in the VPC to address transitive routing.Figure 1. VPC endpoint architectureBandwidth considerations:When setting up an interface endpoint, choose multiple subnets across multiple Availability Zones to implement high availability. The number of ENIs should equal to number of subnets chosen. Interface endpoints offer a throughput of 10 Gbps per ENI with a burst capability of 40 Gbps. If your use case requires higher throughput, contact AWS Support.Gateway endpoints are route table entries that route your traffic directly from the subnet where traffic is originating to the S3 service. Traffic does not flow through an intermediate device or instance. Hence, there is no throughput limit for the gateway endpoint itself. The initial setup for gateway endpoints consists in specifying the VPC route tables you would like to use to access the service. Route table entries for the destination (prefix list) and target (endpoint ID) are automatically added to the route tables.The two architectural options for creating and managing endpoints are:Single VPC architectureUsing a single VPC, we can configure:Gateway endpoints for VPC resources to access S3VPC interface endpoint for on-premises resources to access S3The following architecture shows the configuration on how both can be set up in a single VPC for access. This is useful when access from within AWS is limited to a single VPC while still enabling external (non-AWS) access.Figure 2. Single VPC architectureDNS configured on-premises will point to the VPC interface endpoint IP addresses. It will forward all traffic from on-premises to S3 through the VPC interface endpoint. The route table configured in the subnet will ensure that any S3 traffic originating from the VPC will flow to S3 using gateway endpoints.Multi-VPC centralized architectureIn a hub and spoke architecture that centralizes S3 access for multi-Region, cross-VPC, and on-premises workloads, we recommend using an interface endpoint in the hub VPC. The same pattern would also work in multi-account/multi-region design where multiple VPCs require access to centralized buckets.Please note that when using a Transit Gateway and VPC Endpoint combination to route traffic to a service destination, cumulative inbound and outbound processing charges for Transit Gateway and VPC Endpoint may be incurred. Care should be taken to understand this cost implication. SeeAWS PrivateLink pricingandAWS Transit Gateway pricing.In addition, firewall appliances that monitor east-west traffic will experience increased load with the Multi-VPC centralized architecture. It may be necessary to use the single VPC endpoint design to reduce impact to firewall appliances.Figure 3. Multi-VPC centralized architectureConclusionBased on preceding considerations, you can choose to use a combination of gateway and interface endpoints to meet your specific needs. Depending on the account structure and VPC setup, you can support both types of VPC endpoints in a single VPC by using a shared VPC architecture.With AWS, you can choose between two VPC endpoint types (gateway endpoint or interface endpoint) to securely access your S3 buckets using a private network. In this blog, we showed you how to select the right VPC endpoint using criteria like VPC architecture, access pattern, and cost. To learn more about VPC endpoints and improve the security of your architecture, readSecurely Access Services Over AWS PrivateLink.TAGS:VPC EndpointsJeff HarmanJeff Harman is a Senior Solution Architect at AWS based out of Boston. He has a passion for designing and implementing scalable, modern platforms on the cloud, for financial services. He worked in financial services for 20 years before joining AWS.Gilles-Kuessan SatchiviGilles-Kuessan Satchivi is an AWS Enterprise Solutions Architect with a background in Networking, Infrastructure, Security, and IT Operations. He is passionate about helping customers build Well-Architected systems on AWS. Before joining AWS, he worked in e-commerce for 17 years. Outside of work, he likes to spend time with his family, and cheer on his children’s soccer team.
This post was co-written with Anusha Dharmalingam, former AWS Solutions Architect. Must your Amazon Web Services (AWS) application connect to Amazon Simple Storage Service (Amazon S3) buckets? Must the connection scale to accommodate bandwidth demands? AWS offers a mechanism called VPC endpoint to meet these requirements. This blog post provides guidance for selecting the right […]
Jeff Harman
Amazon Simple Storage Service (S3) Amazon VPC Architecture AWS Identity and Access Management (IAM) AWS PrivateLink