How Wego secured developer connectivity to Amazon Relational Database Service instances
AWS Architecture BlogHow Wego secured developer connectivity to Amazon Relational Database Service instancesby Adriaan de Jonge and Madan Kumar | on01 NOV 2022| inAmazon EC2,Amazon ElastiCache,Amazon RDS,Architecture,AWS Systems Manager|Permalink|ShareHow do you securely accessAmazon Relational Database Service (Amazon RDS)instances from a developer’s laptop? Online travel marketplace,Wego, shares their journey from bastion hosts in the public subnet to lightweight VPN tunnels on top of Session Manager, a capability ofAWS Systems Manager, using temporary access keys.In this post, we explore how developers get access to allow-listed resources in their virtual private cloud (VPC) directly from their workstation, by tunnelling VPN over secure shell (SSH), which, in turn, is tunneled over Session Manager.Note:This blog post is not intended as a step-by-step, how-to guide. Commands stated here are for illustrative purposes and may need customization.Wego’s architecture before starting this journeyIn 2021, Wego’s developer connectivity architecture was based on jump hosts in a public subnet, as illustrated in Figure 1.Figure 1. Original Wego architectureFigure 1 demonstrates a network architecture with both public and private subnets. The public subnet contains anAmazon Elastic Compute Cloud (Amazon EC2)instance that serves as jump host. The diagram illustrates a VPN tunnel between the developer’s desktop and the VPC.In Wego’s previous architecture, the jump host was connected to the internet for terminal access through the secure shell (SSH) protocol, which accepts traffic at Port 22. Despite restrictions to the allowed source IP addresses, exposing Port 22 to the internet can increase the likeliness of a security breach; it is possible to spoof (mimic) an allowed IP address and attempt a denial of service attack.Moving the jump host to a private subnet with Session ManagerSession Manager helps minimize the likeliness of a security breach. Figure 2 demonstrates how Wego moved the jump host from a public subnet to a private subnet. In this architecture, Session Manager serves as the main entry point for incoming network traffic.Figure 2. Wego’s new architecture using Session ManagerWe will explore how developers connect to Amazon RDS directly from their workstation in this architecture.Tunnel TCP traffic through Session ManagerSession Manager is best known for its terminal access capability, but it can also tunnel TCP connections. This is helpful if you want to access EC2 instances from your local workstation (Figure 3).Figure 3. Tunneling TCP traffic over Session ManagerHere’s an example command to forward traffic from local host Port 8888 to an EC2 instance:$ aws ssm start-session --target <instance-id> \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8888"], "localPortNumber":["8888"]}'This assumes the target EC2 instance is configured withAWS Systems Manager connectivity.Tunnel SSH traffic over Session ManagerSSH is a protocol built on top of TCP; therefore, you can tunnel SSH traffic similarly (Figure 4).Figure 4. Tunneling SSH traffic over Session ManagerTo allow a short-hand notation for SSH over SSM, add the following configuration to the~/.ssh/configconfiguration file:host i-* mi-*ProxyCommand sh -c "aws ssm start-session --target %h \--document-name AWS-StartSSHSession \--parameters 'portNumber=%p'"You can now connect to the EC2 instance over SSH with the following command:ssh -i <key-file> <username>@<ec2-instance-id>For example:ssh -i my_key ec2-user@i-1234567890abcdef0Ideally, your key-file is ashort-livedcredential, as recommended by theAWS Well-Architected Framework, as it narrows the window of opportunity for a security breach. However, it can be tedious to manage short-lived credentials. This is where EC2 Instance Connect comes to the rescue!Replace SSH keys with EC2 Instance ConnectEC2 Instance Connectis available both on the AWS console and the command line. It makes it easier to work with short-lived keys. On the command line, it allows us to install our own temporary access credentials into a private EC2 instance for the duration of 60 seconds (Figure 5).Figure 5. Connecting to SSH with temporary keysEnsure theEC2 instance connect pluginis installed on your workstation:pip3 install ec2instanceconnectcliThis blog post assumes you are usingAmazon Linuxon the EC2 instance with all pre-requisites installed. Make sure your IAM role or user has therequired permissions.To generate a temporary SSH key pair, insert:$ ssh-keygen -t rsa -f my_key
$ ssh-add my_keyTo install the public key into the EC2 instance, insert:$ aws ec2-instance-connect send-ssh-public-key \
  --instance-id<instance-id>\
  --instance-os-user<username>\
  --ssh-public-key<location ssh key public key>\
  --availability-zone<availabilityzone>\
  --region<region>For example:$ aws ec2-instance-connect send-ssh-public-key \
  --instance-id i-1234567890abcdef0 \
  --instance-os-user ec2-user \
  --ssh-public-key file://my_key.pub \
  --availability-zone ap-southeast-1b \
  --region ap-southeast-1Connect to the EC2 instance within 60 seconds and delete the key after use.Tunneling VPN over SSH, then over Session ManagerIn this section, we adopt a third-party, open-source tool that is not supported by AWS, calledsshuttle. sshuttle is a transparent proxy server that works as a VPN over SSH. It is based on Python and released under the LGPL 2.1 license. It runs across a wide range of Linux distributions and on macOS (Figure 6).Figure 6. Tunneling VPN over SSH over Session ManagerWhy do we need to tunnel VPN over SSH, rather than using the earlier TCP over Session Manager? After all, you could useport forwarding to remote hosts using Session Manageras explained in the AWS blog postUse port forwarding in AWS Systems Manager Session Manager to connect to remote hosts.A lightweight VPN solution, like sshuttle, simplifies the connectivity by allowing you to forward traffic from Amazon EC2 to Amazon RDS. From the developer’s perspective, this works transparently, as if it is regular network traffic.To install sshuttle, use one of thedocumented commands:$ pip3 install sshuttleTo start sshuttle, use the following command pattern:$ sshuttle -r<username>@<instance-id> <private CIDR range>For example:$ sshuttle -r ec2-user@i-1234567890abcdef0 10.0.0.0/16Make sure the security group for the RDS DB instance allows network access from the jump host. You can now connect directly from the developer’s workstation to the RDS DB instance based on its IP address.Advantages of this architectureIn this blog post, we layered a VPN over SSH that, in turn, is layered over Session Manager, plus we used temporary SSH keys.Wego designed this architecture, and it was practical and stable for day-to-day use. They found that this solution runs at lower cost thanAWS Client VPNand is sufficient for the use case of developers accessing online development environments.Wego’s new architecture has a number of advantages, including:More easily connecting to workloads in private and isolated subnetsInbound security group rules are not required for the jump host, as Session Manager is an outbound connectionAccess attempts are logged inAWS CloudTrailAccess control uses standard IAM policies, including tag-based resource accessSecurity groups and network access control lists still apply to “allow” or “deny” traffic to specific destinationsSSH keys are installed only temporarily for 60 seconds through EC2 Instance ConnectConclusionIn this blog post, we explored Wego’s access patterns that can help you reduce your exposure to potential security attacks. Whether you adopt Wego’s full architecture or only adopt intermediary steps (like SSH over Session Manager and EC2 Instance Connect), reducing exposure to the public subnet and shortening the lifetime of access credentials can improve your security posture!Further readingAWS Systems Manager Session ManagerAllow and controlling permissions for SSH connections through Session ManagerConnect using EC2 Instance Connectsshuttle Github repository(third-party, open-source software)sshuttle: where transparent proxy meets VPN meets sshAdriaan de JongeAdriaan de Jonge is Sr Solutions Architect at AWS in Singapore, supporting the Digital Native Business segment. Adriaan is particularly interested in serverless, cloud-native development, and DevOps. In his spare time, he likes to bake cakes that are suitable for people with allergies.Madan KumarMadan Kumar is a DevOps Tech Lead at Wego, responsible for designing solutions on IaC, security, automation, SRE and cloud journey. He has deep dive experience of over 9 years in IT Software.
How do you securely access Amazon Relational Database Service (Amazon RDS) instances from a developer’s laptop? Online travel marketplace, Wego, shares their journey from bastion hosts in the public subnet to lightweight VPN tunnels on top of Session Manager, a capability of AWS Systems Manager, using temporary access keys. In this post, we explore how […]
Adriaan de Jonge
Amazon EC2 Amazon ElastiCache Amazon RDS Architecture AWS Systems Manager